Compile with conda-forge boost

[Anthchirp]

We should now be in a position where we can ditch our own boost source repository.

However if we do that the build fails in annlib_ext with the following error:

/dev/shm/wra62962/conda_base/bin/x86_64-conda_cos6-linux-gnu-c++ -o annlib_adaptbx/ann/boost_python/annlib_ext.o -c -std=c++11 -pthread -DBOOST_PYTHON_MAX_BASES=2 -DUSE_CONDA -fPIC -fno-strict-aliasing -Wall -Wno-sign-compare -Wno-unknown-pragmas -Wno-parentheses -DNDEBUG -O3 -funroll-loops -DBOOST_ALL_NO_LIB -ffast-math -I/dev/shm/wra62962/build/../conda_base/include -I/dev/shm/wra62962/build/include -I/dev/shm/wra62962/modules/cctbx_project -I/dev/shm/wra62962/build/../conda_base/include -I/dev/shm/wra62962/modules/annlib -I/dev/shm/wra62962/modules/annlib/src -I/dev/shm/wra62962/modules/annlib/include -I/dev/shm/wra62962/modules/annlib_adaptbx/include -I/dev/shm/wra62962/build/annlib_adaptbx/include -I/dev/shm/wra62962/conda_base/include/python3.6m /dev/shm/wra62962/modules/annlib_adaptbx/ann/boost_python/annlib_ext.cpp
/dev/shm/wra62962/conda_base/bin/x86_64-conda_cos6-linux-gnu-c++ -o annlib_adaptbx/self_include/ANN.o -c -std=c++11 -pthread -fPIC -fno-strict-aliasing -Wall -Wno-sign-compare -Wno-unknown-pragmas -Wno-parentheses -DNDEBUG -O3 -funroll-loops -DBOOST_ALL_NO_LIB -ffast-math -I/dev/shm/wra62962/build/include -I/dev/shm/wra62962/modules/cctbx_project -I/dev/shm/wra62962/modules/cctbx_project -I/dev/shm/wra62962/build/../conda_base/include -I/dev/shm/wra62962/modules/annlib -I/dev/shm/wra62962/modules/annlib/src -I/dev/shm/wra62962/modules/annlib/include -I/dev/shm/wra62962/modules/annlib_adaptbx/include -I/dev/shm/wra62962/build/annlib_adaptbx/include annlib_adaptbx/self_include/ANN.cpp
In file included from /dev/shm/wra62962/modules/annlib_adaptbx/ann/boost_python/annlib_ext.cpp:1:0:
/dev/shm/wra62962/build/include/scitbx/array_family/boost_python/flex_fwd.h:12:10: fatal error: boost/python/detail/prefix.hpp: No such file or directory
 #include <boost/python/detail/prefix.hpp>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
scons: *** [annlib_adaptbx/ann/boost_python/annlib_ext.o] Error 1
scons: building terminated because of errors.

We are probably not including boost correctly. I thought this worked for cctbx builds though, so maybe annlib_ext is not built there?

[bkpoon]

@nksauter mentioned this to me.

You need to add the boost package in addition to boost-cpp. The boost-cpp package explicitly checks that the Boost.Python stuff is removed (https://github.com/conda-forge/boost-cpp-feedstock/blob/master/recipe/meta.yaml). The boost package checks that Boost.Python is available (https://github.com/conda-forge/boost-feedstock/blob/master/recipe/meta.yaml).

The Boost conda packages should work since when I was working on cctbx/cctbx_project#409 with the Boost packages from conda, I build rstbx and spotfinder, which should build annlib_adaptbx. The Azure pipeline also runs the test for annlib_adaptbx.

[Anthchirp]

Thanks @bkpoon and @nksauter. Adding the boost package means the compilation commences. It then dies during the linking phase with

/home/vsts/work/1/conda_base/bin/x86_64-conda_cos6-linux-gnu-c++ -o lib/dials_algorithms_background_ext.so -shared -std=c++11 -std=c++11 dials/algorithms/background/boost_python/helpers.o dials/algorithms/background/boost_python/background_ext.o -L/home/vsts/work/1/conda_base/lib -Llib -L/home/vsts/work/1/modules/lib -L/home/vsts/work/1/modules/cctbx_project/lib -lboost_thread -lboost_system -lboost_python36 -lm -lm -lscitbx_boost_python -lboost_python -lboost_thread -lcctbx
/home/vsts/work/1/conda_base/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.5.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: cannot find -lboost_python
collect2: error: ld returned 1 exit status
scons: *** [lib/dials_algorithms_background_ext.so] Error 1
scons: building terminated because of errors.

I note that the -lboost_python parameter is not present in the lines above, and that the failing line is the first one containing the string dials, so maybe something we're doing wrong inside the DIALS SConscript file?

[bkpoon]

The Boost.Python libraries have Python versions in their names. You'll have to do something similar to cctbx/dxtbx#134 to get the versioned names.

[Anthchirp]

Thanks. This fixes the linux build - something is still off on macos.

[bkpoon]

It looks like macOS is pulling Boost 1.73 and linux is pulling Boost 1.70. You'll want to set an upper bound. I think I tried Boost 1.72, but I don't remember if everything worked. The conda-forge packages are being migrated to Boost 1.72, but it's not complete.

[Anthchirp]

Well spotted!

[Anthchirp]

That did the trick. Thanks @bkpoon!

[Anthchirp]

Merged.

New development environments will pick up boost sources
from conda-forge, currently limited to version up to 1.70.
We should look at updating this at some point in the future,
probably once conda-forge has moved to 1.72.

Existing development environments do not need to be updated,
however if you choose to install the conda boost sources manually
or re-bootstrap your conda_base directory then you should remove
the boost directory from modules.

Nothing will change for cctbx-boostrap users with Python<3.7. When
cctbx-bootstrap is used with the DIALS builder and Python>=3.7 the
bootstrap option --no-boost-src should likely be specified.