Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
conda: update linux environments
- Add gtk2 for more consistent behavior across linux distributions
- Update openssl to 1.0.2q
  • Loading branch information
bkpoon committed Feb 26, 2019
1 parent 8d20ffe commit 6a8503a
Show file tree
Hide file tree
Showing 2 changed files with 58 additions and 4 deletions.
33 changes: 31 additions & 2 deletions libtbx/auto_build/conda_envs/cctbx_py27_linux-64.txt
Expand Up @@ -10,32 +10,55 @@ https://conda.anaconda.org/conda-forge/linux-64/libsvm-323-0.tar.bz2#f5849cbafea
https://conda.anaconda.org/conda-forge/linux-64/lz4-c-1.8.1.2-0.tar.bz2#61aaf0dbcc58eb0a3030d78f6d2c6b15
https://conda.anaconda.org/conda-forge/linux-64/blosc-1.15.1-hf484d3e_1002.tar.bz2#2cd91766d181359ecc700b3fc66e6c4d
https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.6-h14c3975_1002.tar.bz2#0ab8427d81b55fcd2212a84a8f6f9728
https://conda.anaconda.org/conda-forge/linux-64/graphite2-1.3.13-hf484d3e_1000.tar.bz2#160e65b3904ab30a4a6b0e418b09255f
https://conda.anaconda.org/conda-forge/linux-64/icu-58.2-hf484d3e_1000.tar.bz2#3749214646d41293022b45ce51386141
https://conda.anaconda.org/conda-forge/linux-64/jpeg-9c-h14c3975_1001.tar.bz2#861fd0d02beba22de24ff9d27439b721
https://conda.anaconda.org/conda-forge/linux-64/libffi-3.2.1-hf484d3e_1005.tar.bz2#d9d62449359cb60c9cd97b6ddff89ae0
https://conda.anaconda.org/conda-forge/linux-64/libiconv-1.15-h14c3975_1004.tar.bz2#8f4beea4de351ccaddbf488f57bb6fa0
https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.32.1-h14c3975_1000.tar.bz2#39c6326f6ee5297632c47db6520546fe
https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.1-hf484d3e_1002.tar.bz2#17f2190cdcbc753893307b4c81aff478
https://conda.anaconda.org/conda-forge/linux-64/openblas-0.3.3-h9ac9557_1001.tar.bz2#87ac95d75862ec2f815e3fe427b50359
https://conda.anaconda.org/conda-forge/linux-64/openssl-1.0.2p-h14c3975_1002.tar.bz2#632630425c804a5909b5980f4124682f
https://conda.anaconda.org/conda-forge/linux-64/openssl-1.0.2q-h14c3975_0.tar.bz2#d2ef3d8169d22c3a573fb685781e8a66
https://conda.anaconda.org/conda-forge/linux-64/pcre-8.41-hf484d3e_1003.tar.bz2#738ba678930cf08359ea0d6a454ab22a
https://conda.anaconda.org/conda-forge/linux-64/pixman-0.34.0-h14c3975_1003.tar.bz2#916e51668f4a001f3da98ab6786f4224
https://conda.anaconda.org/conda-forge/linux-64/pthread-stubs-0.4-h14c3975_1001.tar.bz2#e0b9987b65bba8016d21f5bdbe0a7923
https://conda.anaconda.org/conda-forge/linux-64/xorg-kbproto-1.0.7-h14c3975_1002.tar.bz2#6dfe5dbe10d55266e4a5e89287eed578
https://conda.anaconda.org/conda-forge/linux-64/xorg-libice-1.0.9-h14c3975_1004.tar.bz2#a289da2ee5b365338042a3fdba3078a8
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxau-1.0.9-h14c3975_0.tar.bz2#ffa7c2b7a2c7dc779ed9e38b10a93c3c
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxdmcp-1.1.2-h14c3975_1007.tar.bz2#bdd613997411c71a2830f0d9b391cc49
https://conda.anaconda.org/conda-forge/linux-64/xorg-renderproto-0.11.1-h14c3975_1002.tar.bz2#fbcb7fa11dee1a5d3df4371cc55bb229
https://conda.anaconda.org/conda-forge/linux-64/xorg-xextproto-7.3.0-h14c3975_1002.tar.bz2#f08999859c405bad87c4bf9b6cdc7bbb
https://conda.anaconda.org/conda-forge/linux-64/xorg-xproto-7.0.31-h14c3975_1007.tar.bz2#a45d8cd411bdf8f08ced463f68986b62
https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.4-h14c3975_1001.tar.bz2#a617e36d37a44931f273712a129afed4
https://conda.anaconda.org/conda-forge/linux-64/zlib-1.2.11-h14c3975_1004.tar.bz2#1784c46ef493804cef0739301e166ff6
https://conda.anaconda.org/conda-forge/linux-64/blas-1.1-openblas.tar.bz2#c259e7a3ea775c6dcba191072cab3adb
https://conda.anaconda.org/conda-forge/linux-64/gettext-0.19.8.1-h9745a5d_1001.tar.bz2#cd738f4cc0e0f73163c2bbaa5d141107
https://conda.anaconda.org/conda-forge/linux-64/hdf5-1.10.4-nompi_h11e915b_1105.tar.bz2#08a81bbe21d0ee91a725ca29a0df6f96
https://conda.anaconda.org/conda-forge/linux-64/libpng-1.6.36-h84994c4_1000.tar.bz2#a690035c54280c8dba66bbdcd8ee7125
https://conda.anaconda.org/conda-forge/linux-64/libtiff-4.0.10-h648cc4a_1001.tar.bz2#3a75111bd0b59b554a9aed14c60d1183
https://conda.anaconda.org/conda-forge/linux-64/libxcb-1.13-h14c3975_1002.tar.bz2#ca4eb860b5528d5c6de8d97021d9ef78
https://conda.anaconda.org/conda-forge/linux-64/libxml2-2.9.8-h143f9aa_1005.tar.bz2#b8a7f1bcd9575550e4721c83a530aaee
https://conda.anaconda.org/conda-forge/linux-64/msgpack-c-3.1.1-h49c8d1a_1000.tar.bz2#498c33443a24d975a02a0ceb1f72c126
https://conda.anaconda.org/conda-forge/linux-64/readline-7.0-hf8c457e_1001.tar.bz2#ba9f8093574f89e5efc3e775ecc0d7d8
https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.9-h84994c4_1000.tar.bz2#1550cb71a2ad9c340598bb33bb953d91
https://conda.anaconda.org/conda-forge/linux-64/xorg-libsm-1.2.3-h4937e3b_1000.tar.bz2#09d73f21da15c99a8a888150b4fa0835
https://conda.anaconda.org/conda-forge/linux-64/freetype-2.9.1-h94bbf69_1005.tar.bz2#96ef524cb3dfa9a75d7424219093ff04
https://conda.anaconda.org/conda-forge/linux-64/glib-2.58.2-hf63aee3_1001.tar.bz2#6a9d6f49cc81996be71e5c093b720b8e
https://repo.anaconda.com/pkgs/main/linux-64/nomkl-3.0-0.tar.bz2#dd8bdca858ba54245790a3cab33cec32
https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.26.0-h67949de_1000.tar.bz2#6889ef537eb38d03eb2f52f3c04fe921
https://conda.anaconda.org/conda-forge/linux-64/xorg-libx11-1.6.7-h14c3975_1000.tar.bz2#f642071b707442f227137a7b8bf45c5b
https://conda.anaconda.org/conda-forge/linux-64/fontconfig-2.13.1-h2176d3f_1000.tar.bz2#23b59d5fa887d97c90154ae65e3ce20c
https://conda.anaconda.org/conda-forge/linux-64/python-2.7.15-h938d71a_1006.tar.bz2#31948b3fc9fe1d4c3db32e14fedbe9ac
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxext-1.3.3-h14c3975_1004.tar.bz2#da4fac0217a604475641451464d1b3ec
https://conda.anaconda.org/conda-forge/linux-64/xorg-libxrender-0.9.10-h14c3975_1002.tar.bz2#0e3f3ceab5ffb3750ac64f9c80d97be5
https://conda.anaconda.org/conda-forge/noarch/alabaster-0.7.12-py_0.tar.bz2#2489a97287f90176ecdc3ca982b4b0a0
https://conda.anaconda.org/conda-forge/noarch/apipkg-1.5-py_0.tar.bz2#4e79696efff4eae6cbcbeddbb5bbb5ee
https://conda.anaconda.org/conda-forge/linux-64/asn1crypto-0.24.0-py27_1003.tar.bz2#ae0236c983265cdf9edcc9696def09b2
https://conda.anaconda.org/conda-forge/noarch/atomicwrites-1.3.0-py_0.tar.bz2#300ca2021ef2213151407e23ed4bfd9d
https://conda.anaconda.org/conda-forge/noarch/attrs-18.2.0-py_0.tar.bz2#79b7cc62e465cac42e737eb4180f1281
https://conda.anaconda.org/conda-forge/noarch/backports-1.0-py_2.tar.bz2#0da16b293affa6ac31812376f8eb79dd
https://conda.anaconda.org/conda-forge/noarch/backports_abc-0.5-py_1.tar.bz2#6ac8cc52b55791b066dc43469bbaf8f2
https://conda.anaconda.org/conda-forge/linux-64/cairo-1.16.0-ha4e643d_1000.tar.bz2#a3d37962e2ccbb65a5ae74bf1731aa13
https://conda.anaconda.org/conda-forge/linux-64/certifi-2018.11.29-py27_1000.tar.bz2#64648e56dd50038e51fab23f29ab84b3
https://conda.anaconda.org/conda-forge/linux-64/chardet-3.0.4-py27_1003.tar.bz2#fc224746b71438dd2e24337d2af0e7c2
https://conda.anaconda.org/conda-forge/linux-64/cython-0.29.5-py27hf484d3e_0.tar.bz2#e70ee102c6feac0aff0761c33eeca0ed
Expand Down Expand Up @@ -86,6 +109,8 @@ https://conda.anaconda.org/conda-forge/linux-64/biopython-1.73-py27h14c3975_0.ta
https://conda.anaconda.org/conda-forge/linux-64/cffi-1.12.1-py27h9745a5d_0.tar.bz2#17b01ff64bf39c06aa204b635758711b
https://conda.anaconda.org/conda-forge/noarch/cycler-0.10.0-py_1.tar.bz2#b2a29b5f20c4d94552ad8f48884ca89c
https://conda.anaconda.org/conda-forge/noarch/execnet-1.5.0-py_0.tar.bz2#0e27b5afe58ff8d4fe6efa8986a9c449
https://conda.anaconda.org/conda-forge/linux-64/gobject-introspection-1.56.1-py27h2da5eee_1002.tar.bz2#95ee237605ce29f99d60e931d6ef8364
https://conda.anaconda.org/conda-forge/linux-64/harfbuzz-2.3.1-h6824563_0.tar.bz2#581a65d27733750b40bee9d33e6dd11b
https://conda.anaconda.org/cctbx/linux-64/hdf5_plugins-0.1.0-hc3b63c6_2.tar.bz2#88235c677aa94660237f2a658912cc17
https://conda.anaconda.org/conda-forge/linux-64/more-itertools-4.3.0-py27_1000.tar.bz2#7a5ad773b085df21a747fa0164f3670d
https://conda.anaconda.org/conda-forge/linux-64/mrcfile-1.1.2-py27_0.tar.bz2#be27e957ac73e0308b3f5ff135aa758a
Expand All @@ -102,9 +127,12 @@ https://conda.anaconda.org/conda-forge/linux-64/setuptools-40.8.0-py27_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/singledispatch-3.4.0.3-py27_1000.tar.bz2#38b56e21a2ef3acd2bd5a86a6ed683bd
https://conda.anaconda.org/conda-forge/linux-64/traceback2-1.4.0-py27_0.tar.bz2#2c0d651756ed617dea2c5905600ae087
https://conda.anaconda.org/conda-forge/linux-64/traitlets-4.3.2-py27_1000.tar.bz2#a6d524e96a01fc74165353aade760b1c
https://conda.anaconda.org/conda-forge/linux-64/atk-2.25.90-hb9dd440_1002.tar.bz2#3420d29895fdc69c35f307e76a300aa8
https://conda.anaconda.org/conda-forge/noarch/backports.functools_lru_cache-1.5-py_1.tar.bz2#9be0c7ebab662ae3eb5950e093638e04
https://conda.anaconda.org/conda-forge/linux-64/cryptography-2.5-py27hb7f436b_1.tar.bz2#d6950e1ac9d1c28a48de487f29b090df
https://conda.anaconda.org/conda-forge/linux-64/gdk-pixbuf-2.36.12-h49783d7_1002.tar.bz2#f9b565007ee9868e1ac2cffdcfeefa67
https://conda.anaconda.org/conda-forge/noarch/jinja2-2.10-py_1.tar.bz2#ccf0dff646a09d42380fa618260ac2a1
https://conda.anaconda.org/conda-forge/linux-64/pango-1.40.14-h4ea9474_1004.tar.bz2#0e7f36cb390171bf6f10c8f32610de58
https://conda.anaconda.org/conda-forge/linux-64/pickleshare-0.7.5-py27_1000.tar.bz2#f169b28ae020a0471305f0e0c57f2d54
https://conda.anaconda.org/conda-forge/noarch/pygments-2.3.1-py_0.tar.bz2#73da381e8365587b6c62bd14fa2d45a5
https://conda.anaconda.org/conda-forge/linux-64/pytest-3.10.1-py27_1000.tar.bz2#651f68a161d6b8b77614cad70a74df26
Expand All @@ -113,6 +141,7 @@ https://conda.anaconda.org/conda-forge/linux-64/scikit-learn-0.20.2-py27_blas_op
https://conda.anaconda.org/conda-forge/linux-64/tornado-5.1.1-py27h14c3975_1000.tar.bz2#adf3de0f4bc56b631b63e5b5dc0a8886
https://conda.anaconda.org/conda-forge/noarch/unittest2-1.1.0-py_0.tar.bz2#504cc35ff8221d810cdeb49f50c5accc
https://conda.anaconda.org/conda-forge/linux-64/wheel-0.33.0-py27_0.tar.bz2#ac9fa424c0235c90d556cdfc4574cb8a
https://conda.anaconda.org/conda-forge/linux-64/gtk2-2.24.31-hb68c50a_1001.tar.bz2#be1b3666cb212378886a978667b98a44
https://conda.anaconda.org/conda-forge/linux-64/h5py-2.9.0-nompi_py27hf008753_1102.tar.bz2#4d000f447a05e9398e18b5e667dcc1fe
https://conda.anaconda.org/conda-forge/linux-64/matplotlib-base-2.2.3-py27h60b886d_1.tar.bz2#921006bfa239b2dda7d08910ee3dce7b
https://conda.anaconda.org/conda-forge/linux-64/pip-19.0.2-py27_0.tar.bz2#241320071f04f4b1771ac27557b22987
Expand All @@ -127,4 +156,4 @@ https://conda.anaconda.org/conda-forge/linux-64/mock-2.0.0-py27_1000.tar.bz2#dc5
https://conda.anaconda.org/conda-forge/linux-64/requests-2.21.0-py27_1000.tar.bz2#2b614f219f91828ee2406068ce006b5b
https://conda.anaconda.org/conda-forge/linux-64/sphinx-1.8.4-py27_0.tar.bz2#23d63a9b3d16c6ae1a6b01c510748148
https://conda.anaconda.org/conda-forge/noarch/numpydoc-0.8.0-py_1.tar.bz2#5e71b7baaecd06f5c2dfbb1055cb0de3
https://conda.anaconda.org/cctbx/linux-64/cctbx_dependencies-2019.02.13-0.tar.bz2#82e1d08d6c871a7546e0c2d56775f6a8
https://conda.anaconda.org/cctbx/linux-64/cctbx_dependencies-2019.02.13-1.tar.bz2#5f61edf7672f8f7449d0558066207168

7 comments on commit 6a8503a

@graeme-winter
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I right in thinking a bunch of this is X-window libraries? This not a bit out of scope for installing crystallographic tools (i.e. the OS should be responsible for something)

@bkpoon
Copy link
Member Author

@bkpoon bkpoon commented on 6a8503a Feb 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not too far out of scope since we are already compiling libXrender in the "base" directory. The issue that prompted this is that the version of gtk2 specific to Ubuntu 16.04 broke some widgets in wxPython. So even though I originally wanted to move some dependencies to the system when building these environments, gtk2 had to be added back, which pulls in the X11 libraries. The reasoning is explained here, http://conda-forge.org/docs/recipe.html#avoid-dependencies-outside-of-conda-forge, but basically, by providing the full stack of dependencies, we're less likely to encounter ABI incompatibilities across different linux distributions. The minimum requirement for conda on linux is glibc 2.7.

@ndevenish
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ABI glibc compatibilities is mainly a theoretical problem, yes? I seem to remember the main problems with glibc before were that the dependencies were too old to be compatible with newer versions e.g. #85, but that caused problems because any new combination of packages needed to be exhaustively tested.

How big is the distribution going to be now?

@bkpoon
Copy link
Member Author

@bkpoon bkpoon commented on 6a8503a Mar 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I wouldn't say glibc compatibility issues are theoretical. You're work on #85 shows that they are not. If you're asking if an end-user will experience an issue, the answer depends on what the user is doing. For Phenix, some users will build and link external packages to our binary distributions, so ABI compatibility is an issue.

As for size, the conda_base directory in a recent nightly build is about 800 MB and from the last base directory from a nightly build is about 840 MB on linux, so they are about the same. The conda_base directories for macOS and Windows will be bigger by a few hundred megabytes because of more standard dependencies (e.g. openblas, clang libraries on macOS, MKL on Windows, and a bunch of common conda Python dependencies, like requests). For Phenix installers, the dependencies are still less than 20% of the total size of the installation.

The nice thing about conda is that you can manually tweak the DIALS environment to remove anything you do not want, but you just have to make sure that everything still works. Or you can fork the package recipe and make the changes there. You can have a DIALS channel with those packages and conda can use them.

@ndevenish
Copy link
Contributor

@ndevenish ndevenish commented on 6a8503a Mar 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#85 was less an ABI compatibility problem and more just using very old software with very new versions of the source. Luckily whatever was preventing update seems to have been resolved given how often these dependency lists are updated. (Resolved for the future, because IIRC the current recommended way still doesn't work)

Just seeing these lists get longer and longer made me concerned about distribution size. It's grown a couple of hundred mb on mac, but I can't check linux currently because when I tried the conda bootstrap failed with a traceback.

@bkpoon
Copy link
Member Author

@bkpoon bkpoon commented on 6a8503a Mar 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you provide the traceback? And what linux distribution and version are you using?
Also, if you have conda installed, can you provide,

  1. The output from conda info
  2. The contents of your ~/.conda/environments.txt file, if available

@ndevenish
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posted into an issue #298

Please sign in to comment.