Update conda environments (Python 3.7 and 3.8)

[bkpoon]

Test updated dependencies and add Python 3.7 (and maybe 3.8) support

List of changes

1. Add cctbxlite environments (not yet enabled)
2. Add pyside2, pyzmq and websocket-server for @Oeffner to cctbx environments
3. Versioned conda environments for linux, macOS, and Windows for Python 2.7, 3.6, 3.7, and 3.8
4. Updated CI build to build all versions of Python (12 builds)
   a) Windows
   b) CentOS 6
   c) macOS 10.14 (Xcode 11.3.1)
5. Update Full build to build all versions of Python (40 builds)
   a) Windows
   b) CentOS 6, 7, 8
   c) Ubuntu 14.04, 16.04, 18.04 (will replace 14.04 with 20.04 when it's released later in April)
   d) macOS 10.14 Xcode 11.3.1 and 10.3, and macOS 10.15 Xcode 11.4
6. Update Boost to 1.70 (source build), closes Boost 1.65: boost::python::numeric removed #70
   a) Match current version for conda
   b) Enables Python 3.7 and later support
   c) Enables support for latest versions of CUDA
7. Bump macOS minimum deployment version from 10.7 to 10.9
   a) Match version for conda packages
8. Add --python flag to bootstrap.py for selecting Python version
   a) Just passes argument to libtbx.install_conda
9. scons moved into conda environment
10. Add --no-boost-src flag to bootstrap.py to prevent downloading of Boost into modules during the update step.
11. Add weekly builds of where -ffast-math (/fp:fast for Windows) is removed and -O0, -O1, -O2, and -O3 (/O1 and /O2 for Windows) are tested on the Full build (160 builds).

@Oeffner Can you double check the Windows builds are working for you?
@luc-j-bourhis @pcxod @Oeffner Are there any changes that you would like to the cctbxlite environments? Currently, these are the dependencies in cctbxlite

biopython
cython
docutils
future
ipython
libsvm
libsvm_py
matplotlib-base
mrcfile
nomkl
numpy
numpydoc
openblas
pillow
pip
psutil
pyopengl
pytest
reportlab
six
sphinx
wxpython
python.app # [osx]
py2app  # [osx]

We can probably cut down the number of dependencies for cctbxlite.

[Oeffner]

Hi @bkpoon I just build this branch on my Windows10 workstation using VS2015 with the commandline
python3 bootstrap.py --builder=cctbx --nproc=10 --python3 --use_conda and it builds fine. But the cctbx.HKLviewer doesn't work as no stub file is created. So I added a change to bootstrap.py so that cctbx.HKLviewer works with the plain cctbx build. Committed in e520f19
The regression tests also seems to run fine:

  Tests run                    : 467
  Failures                     : 0
  Warnings (possible failures) : 0
  Known Failures (  8)         : 8
  Known Unstable (  2)         : 0
  Stderr output (discouraged)  : 10

See run_tests_parallel_zlog for full output.

Bootstrap success: tests

One detail that confused me was how to achieve a successful build of this branch. I ended up using the above command line and hitting the pause button on the keyboard as soon as the download of cctbx_project had completed so that I could manually switch branch from master to newpy. Then hit the return key to resume the boostrap build.

[bkpoon]

@Oeffner I don't normally download a completely new set of repositories. For this branch, since it changes dependencies, I'll make a symbolic link to an existing modules directory, check out the newpy branch in cctbx_project and the 1.70 branch in boost,and then run bootstrap.py with the base and build options.

python bootstrap.py base build --use-conda <extra options>

[pcxod]

Hi @bkpoon , @luc-j-bourhis. I do not use conda for building cctbx for Olex2 - I rely on the old fashioned way that seems to have been working for a while bow. After getting the bundle (which is apparently missing on this page https://cctbx.github.io/installation.html#id3, but as I remember last time I got it with bootstrap), I typically use 'configure iotbx smtbx fast_linalg' and have to leave with the dependencies that come along them. But these are as close as a minimal set.

[Oeffner]

In porting python modules in our phaser repo to python3 I have encountered problems with the scitbx.suffixtree module. The regression test, cctbx_project\scitbx\suffixtree\test\tst_single.py, fails in a python3 build with lots of errors like OverflowError: Python int too large to convert to C long . The test passes fine in a python2 build.
@bkpoon, would you have any idea. I had a look at the output from the full cctbx regression test on windows with VS2015 of this branch and where it exercises this particular test it passes according to the log file. However, running the test manually from the command line with the environment from that build it still fails.

[bkpoon]

@Oeffner It's been failing on Windows, but shows up as a warning because of the stderr output. I looked briefly into it a while ago, but wasn't able find a fix. Also, the test is marked as an expected failure for newer versions of GCC. There is a segmentation fault. @gbunkoczi would know more about that code.

[Anthchirp]

I raised a question in our group meeting today: The next DIALS release is coming up in May. For planning purposes only - do you have a date in mind when you plan to merge this?

[bkpoon]

@Anthchirp How about next Friday, May 8th at 3 pm Pacific? I'll make the changes to just use the master branch of boost before merging.

I made the changes to remove the --python3 and --wxpython4 command-line options for bootstrap.py, but install_base_packages.py should still be able to build Python 3.7 and install wxPython 4. And I added a flag to not download the Boost source package. It's --no-boost-src, but the flag name can be changed. This flag will be useful for building conda packages where boost will come from conda-forge.

@pcxod @luc-j-bourhis I'll try to construct a minimal set of dependencies for the iotbx, smtbx, and fast_linalg modules. It may just be boost, numpy, six, and future. Maybe matplotlib-base for plotting. It'll be useful for building a cctbx-base conda package that is barebones and then the regular cctbx package with all the other useful options for general users. And for Olex2, since boost is in the modules directory, it may be possible to use any Python virtual environment and just use pip to install the other dependencies.

[Anthchirp]

The suggested timestamp works well and leaves us plenty of time to iron out any potential issues before the release. Thank you!