include_next "loophole"

[yd23]

I tried to build deepvariant on a local ubuntu server.
With GCP support turned off, so far I am stuck with an error after build_and_test.sh

(13:58:12) ERROR: /root/deepvariant/deepvariant/core/python/BUILD:174:1: CLIF wrapping deepvariant/core/python/hts_verbose.clif failed (Exit 4): pyclif failed: error execut ing command (cd /root/.cache/bazel/_bazel_root/8422bf851bfac3671a35809acde131a7/execroot/genomics && \ exec env - \ bazel-out/host/bin/external/clif/pyclif --modname deepvariant.core.python.hts_verbose -c bazel-out/k8-opt/genfiles/deepvariant/core/python/hts_verbose.cc -g bazel-out/k8- opt/genfiles/deepvariant/core/python/hts_verbose.h -i bazel-out/k8-opt/genfiles/deepvariant/core/python/hts_verbose_init.cc --prepend /root/opt/clif/python/types.h -Iextern al/protobuf_archive -Ibazel-out/k8-opt/genfiles -Ibazel-out/k8-opt/genfiles/external/local_config_python -Iexternal/htslib -Ibazel-out/k8-opt/genfiles/external/htslib -I. - Iexternal/bazel_tools -Ibazel-out/k8-opt/genfiles/external/bazel_tools -Iexternal/htslib/htslib/htslib_1_6 -Ibazel-out/k8-opt/genfiles/external/htslib/htslib/htslib_1_6 -Ie xternal/bazel_tools/tools/cpp/gcc3 -Iexternal/clif -Ibazel-out/k8-opt/genfiles/external/clif -Iexternal/local_config_python -Ibazel-out/k8-opt/genfiles/external/protobuf_ar chive -Iexternal/local_config_python/python_include -Ibazel-out/k8-opt/genfiles/external/local_config_python/python_include -Iexternal/protobuf_archive/src -Ibazel-out/k8-o pt/genfiles/external/protobuf_archive/src '-f-Iexternal/protobuf_archive -Ibazel-out/k8-opt/genfiles -Ibazel-out/k8-opt/genfiles/external/local_config_python -Iexternal/hts lib -Ibazel-out/k8-opt/genfiles/external/htslib -I. -Iexternal/bazel_tools -Ibazel-out/k8-opt/genfiles/external/bazel_tools -Iexternal/htslib/htslib/htslib_1_6 -Ibazel-out/ k8-opt/genfiles/external/htslib/htslib/htslib_1_6 -Iexternal/bazel_tools/tools/cpp/gcc3 -Iexternal/clif -Ibazel-out/k8-opt/genfiles/external/clif -Iexternal/local_config_py thon -Ibazel-out/k8-opt/genfiles/external/protobuf_archive -Iexternal/local_config_python/python_include -Ibazel-out/k8-opt/genfiles/external/local_config_python/python_inc lude -Iexternal/protobuf_archive/src -Ibazel-out/k8-opt/genfiles/external/protobuf_archive/src -std=c++11' deepvariant/core/python/hts_verbose.clif) _BackendError: Matcher failed with status 1 In file included from /dev/stdin:1: In file included from /root/opt/clif/python/types.h:27: In file included from bazel-out/k8-opt/genfiles/external/local_config_python/python_include/Python.h:19: /usr/include/limits.h:123:16: fatal error: 'limits.h' file not found # include_next <limits.h>`

Before that, I adjust the "/deepvariant/third_party/clif.bzl" file to include a clif header which the same build command complained about.

clif.bzl: "--prepend", "/root/opt/clif/python/types.h"

The system is a fresh ubuntu16.4
Linux AnnoSpark 4.4.0-103-generic #126-Ubuntu SMP Mon Dec 4 16:23:28 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

My gcc include appears to be so
$gcc -xc -E -v -

/usr/lib/gcc/x86_64-linux-gnu/4.8/include
/usr/local/include
/usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed
/usr/include/x86_64-linux-gnu
/usr/include

clif build
commit 6c6d894a112d978bd5abfcab1052c60c5ee365a9

Any help or direction is deeply appreciated.
Dan

[pgrosu]

limits.h should be in /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed

Try the following command:

ls -l /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed/limits.h

[yd23]

@pgrosu I am actually reckoning some sort of include order issue related to clif, which I am not quite familiar with. To my understanding, it is quite normal when defining new types, one would like to extend the defs in limits.h (really bad...). In my case I think it is telling me at the end of my include path list there is a include_next which leads to no where, I am not sure if what I need is a modified derivative for headers.

/root/opt/clif/clang/lib/clang/5.0.0/include/limits.h
/root/clif_backend/llvm/tools/clang/lib/Headers/limits.h
/root/clif_backend/build_matcher/lib/clang/5.0.0/include/limits.h
/usr/include/c++/v1/support/ibm/limits.h
/usr/include/c++/4.8/tr1/limits.h
/usr/include/c++/5/tr1/limits.h
/usr/include/limits.h
/usr/include/linux/limits.h
/usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed/limits.h
/usr/lib/gcc/x86_64-linux-gnu/5/include-fixed/limits.h
/usr/src/linux-headers-4.4.0-103/arch/arm/include/asm/limits.h
/usr/src/linux-headers-4.4.0-103/include/uapi/linux/limits.h

includes = [
    include_htslib,
    ".",
   "/usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed/",
]

I actually modified the chunk in deepvariant/third_party/htslib.BUILD, to specifically include a particular location which contains the type python needs and without the include_next loophole, but still doesnot work, maybe the syntax was wrong, the build_and_test actually issued a warning saying

WARNING: /root/.cache/bazel/_bazel_root/8422bf851bfac3671a35809acde131a7/external/htslib/BUILD.bazel:216:16: in includes attribute of cc_library rule @htslib//:htslib: ignoring invalid absolute path '/usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed'

[depristo]

Having personally fought through all sorts of similar errors when we were preparing the OSS release, I know how painful this is. Before diving into this, maybe you can tell me what you are trying to do here. Are you saying that you can't run build_and_tesh.sh without modification, and you are trying to overcome some issue that's not itemized here? Or are you trying to do something else?

[depristo]

I do have one recommendation for you though; try building CLIF on your machine before running ./build-prereqs.sh. That script only uses our prebuilt CLIF binary if CLIF isn't already installed on your machine. It's possible that your system has local upgrades to ubuntu 16 that are confusing our pre-built binary (we are waiting for an official CLIF binary distribution, so in the meantime we are stuck with this less-than-ideal solution). The installation is pretty painless (https://github.com/google/clif#installation)

[yd23]

@depristo Thanks a lot for the clarification.
In my case, I am trying to have a standalone version to test with, without GCP at this stage.

In fact, I did manually pull clif from github (commit 6c6d894a112d978bd5abfcab1052c60c5ee365a9) before building, bypassing the chunk note_build_stage Install CLIF binary in build-prereq.sh,
since I encountered the complaint of that OSS specific part with gsutil and clif, with this modified piece instead:

  export DV_PLATFORM="ubuntu-16"
  cd ..
  git clone https://github.com/google/clif 
  cd clif
  ./INSTALL.sh
  python setup.py install
  sudo ldconfig  # Reload shared libraries.

To my understanding, the manually built clif should be of the same structure as the setup script intended. However I didnot check if this pre-built binary is actually functionally similar to my manually built one. Any chance to put this oss-prebuilt in the git repo, at least for testing ?

OSS_CLIF_PKG="oss_clif.${DV_PLATFORM}.latest.tgz"

[depristo]

Does build_and_test.sh run properly without this modification? There's really nothing special about our pre-built CLIF binary. Here's our exact build commands:

# Install prereqs.
sudo -H apt-get -y install ninja-build subversion
sudo -H apt-get -y install virtualenv python-pip pkg-config
sudo -H pip install 'pyparsing>=2.2.0'
sudo -H pip install 'protobuf>=3.4'

echo === building protobufs

sudo -H apt-get install -y autoconf automake libtool curl make g++ unzip
wget https://github.com/google/protobuf/releases/download/v3.4.1/protobuf-cpp-3.4.1.tar.gz
tar xvzf protobuf-cpp-3.4.1.tar.gz
(cd protobuf-3.4.1 &&
  ./autogen.sh &&
  ./configure &&
  make -j 32 &&
  make -j 32 check &&
  sudo make -j 32 install &&
  sudo ldconfig)

echo === building CLIF

git clone https://github.com/google/clif.git
sed -i 's/\$HOME\/opt/\/usr\/local/g' clif/INSTALL.sh
sed -i 's/-j 2//g' clif/INSTALL.sh
(cd clif && sudo ./INSTALL.sh)

echo === creating package tgz

sudo find ${CLIF_DIR} -type d -exec chmod a+rx {} \;
sudo find ${CLIF_DIR} -type f -exec chmod a+r {} \;
tar czf "${CLIF_PACKAGE}" /usr/local/lib/libproto* "${CLIF_DIR}"

echo === SUCCESS: package is "${CLIF_PACKAGE}"

which is similar to your script but not identical. In fact you may be getting burned by sed -i 's/\$HOME\/opt/\/usr\/local/g' clif/INSTALL.sh which changes the install dir from ~/opt to /usr/local/.

Is there a reason not to use this pre-built binary? If you are able to do git clone you should be able to reach GCS to get the binary. If you don't want to do that inside the script, you can always do it outside the script once, install it manually, and then build_and_test.sh won't try to refetch. Or you can follow the exact instructions above and it should create the actual clif binaries we distribute. We are looking forward to an official binary version of CLIF from that team...

[yd23]

@depristo In fact, without touching any setup sh scripts, that same build_and_test attempt failed with a missing header of this prepend:
--prepend clif/python/types.h
That is why I ended up modifying clif.bzl in third_party to include the absolute path of this include, not sure if this is the culprit.

I will try to move the clif installation to /usr/local to give it a shot, also with this new 0.4.1 release.

[depristo]

Yes, you can't do that with bazel - it doesn't allow you to do absolute path operations like that in general due to their approach to sandboxing / hermetic builds. I suspect moving CLIF to the expected location may fix it.

[yd23]

@depristo The porting to /usr/local works, thanks a lot.

With 0.4.0 build_and_test did complain about the htslib_gcp_oauth.init issue, which is now fixed in 0.4.1

(18:20:33) INFO: Elapsed time: 2.092s, Critical Path: 0.03s
(18:20:33) INFO: Build completed successfully, 2 total actions

Case closed, cheers.

oops, just to append a comment, by exporting an include variable with a non /usr/local installation of clif doesnot help, pyclif still cant find it. This was tested before touching clif.bzl. This clif thing must have its own include logic.