Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update poppunk to 2.1.1 #23350

Merged
merged 20 commits into from
Aug 3, 2020
Merged

Update poppunk to 2.1.1 #23350

merged 20 commits into from
Aug 3, 2020

Conversation

BiocondaBot
Copy link
Collaborator

Update poppunk: 2.0.2 → 2.1.1

install with bioconda Conda

Info Link
Recipe recipes/poppunk (click to view/edit other files)
Releases https://pypi.org/pypi/poppunk/json

This pull request was automatically generated (see docs).

@BiocondaBot BiocondaBot added autobump Automatic Version Update new version labels Jul 24, 2020
@johnlees
Copy link
Contributor

@bioconda/core My build succeeds, but then I am getting the pasted error in the run/test stage. It seems like the installation of gtk3 is erroring, which I guess is pulled in my matplotlib-base. Have you seen this before, or have clues as to why this might be happening?

16:58:39 BIOCONDA INFO (OUT) TEST START: /opt/conda/conda-bld/noarch/poppunk-2.1.1-py_0.tar.bz2
16:58:39 BIOCONDA INFO (OUT) Adding in variants from /tmp/tmp3kmps9u1/info/recipe/conda_build_config.yaml
16:58:39 BIOCONDA INFO (OUT) INFO:conda_build.variants:Adding in variants from /tmp/tmp3kmps9u1/info/recipe/conda_build_config.yaml
16:58:39 BIOCONDA INFO (OUT) INFO conda_build.variants:_combine_spec_dictionaries(188): Adding in variants from /tmp/tmp3kmps9u1/info/recipe/conda_build_config.yaml
16:58:39 BIOCONDA INFO (OUT) Renaming work directory,  /opt/conda/conda-bld/poppunk_1595609780416/work  to  /opt/conda/conda-bld/poppunk_1595609780416/work_moved_poppunk-2.1.1-py_0_noarch
16:58:48 BIOCONDA INFO (OUT) Collecting package metadata (repodata.json): ...working... done
16:58:55 BIOCONDA INFO (OUT) Solving environment: ...working... done
16:58:57 BIOCONDA INFO (OUT) 
16:58:57 BIOCONDA INFO (OUT) ## Package Plan ##
16:58:57 BIOCONDA INFO (OUT) 
16:58:57 BIOCONDA INFO (OUT)   environment location: /opt/conda/conda-bld/poppunk_1595609780416/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_
16:58:57 BIOCONDA INFO (OUT) 
16:58:57 BIOCONDA INFO (OUT) 
16:58:57 BIOCONDA INFO (OUT) The following NEW packages will be INSTALLED:
16:58:57 BIOCONDA INFO (OUT) 
16:58:57 BIOCONDA INFO (OUT)     _libgcc_mutex:         0.1-conda_forge               conda-forge
16:58:57 BIOCONDA INFO (OUT)     _openmp_mutex:         4.5-0_gnu                     conda-forge
16:58:57 BIOCONDA INFO (OUT)     at-spi2-atk:           2.34.2-h1369247_0             conda-forge
16:58:57 BIOCONDA INFO (OUT)     at-spi2-core:          2.36.0-h1369247_1             conda-forge
16:58:57 BIOCONDA INFO (OUT)     atk-1.0:               2.36.0-haf93ef1_1             conda-forge
16:58:57 BIOCONDA INFO (OUT)     boost:                 1.72.0-py38h9de70de_0         conda-forge
16:58:57 BIOCONDA INFO (OUT)     boost-cpp:             1.72.0-h7b93d67_2             conda-forge
16:58:57 BIOCONDA INFO (OUT)     bzip2:                 1.0.8-h516909a_2              conda-forge
16:58:57 BIOCONDA INFO (OUT)     ca-certificates:       2020.6.20-hecda079_0          conda-forge
16:58:57 BIOCONDA INFO (OUT)     cairo:                 1.16.0-h3fc0475_1005          conda-forge
16:58:57 BIOCONDA INFO (OUT)     cairomm:               1.12.2-2                      conda-forge
16:58:57 BIOCONDA INFO (OUT)     cairomm-1.0:           1.12.2-h0069156_2             conda-forge
16:58:57 BIOCONDA INFO (OUT)     capnproto:             0.6.1-hfc679d8_1              conda-forge
16:58:57 BIOCONDA INFO (OUT)     certifi:               2020.6.20-py38h32f6830_0      conda-forge
16:58:57 BIOCONDA INFO (OUT)     cycler:                0.10.0-py_2                   conda-forge
16:58:57 BIOCONDA INFO (OUT)     dbus:                  1.13.6-he372182_0             conda-forge
16:58:57 BIOCONDA INFO (OUT)     dendropy:              4.4.0-py_1                    bioconda
16:58:57 BIOCONDA INFO (OUT)     epoxy:                 1.5.4-h516909a_2              conda-forge
16:58:57 BIOCONDA INFO (OUT)     expat:                 2.2.9-he1b5a44_2              conda-forge
16:58:57 BIOCONDA INFO (OUT)     fontconfig:            2.13.1-h1056068_1002          conda-forge
16:58:57 BIOCONDA INFO (OUT)     freetype:              2.10.2-he06d7ca_0             conda-forge
16:58:57 BIOCONDA INFO (OUT)     fribidi:               1.0.10-h516909a_0             conda-forge
16:58:57 BIOCONDA INFO (OUT)     gdk-pixbuf:            2.38.2-h3f25603_4             conda-forge
16:58:57 BIOCONDA INFO (OUT)     gettext:               0.19.8.1-hc5be6a0_1002        conda-forge
16:58:57 BIOCONDA INFO (OUT)     glib:                  2.65.0-h6f030ca_0             conda-forge
16:58:57 BIOCONDA INFO (OUT)     gmp:                   6.2.0-he1b5a44_2              conda-forge
16:58:57 BIOCONDA INFO (OUT)     gobject-introspection: 1.64.1-py38h03d966d_1         conda-forge
16:58:57 BIOCONDA INFO (OUT)     graph-tool:            2.33-py38hed06623_0           conda-forge
16:58:57 BIOCONDA INFO (OUT)     graphite2:             1.3.13-he1b5a44_1001          conda-forge
16:58:57 BIOCONDA INFO (OUT)     gsl:                   2.6-h294904e_0                conda-forge
16:58:57 BIOCONDA INFO (OUT)     gtk3:                  3.24.21-h45fd312_0            conda-forge
16:58:57 BIOCONDA INFO (OUT)     h5py:                  2.10.0-nompi_py38hfb01d0b_104 conda-forge
16:58:57 BIOCONDA INFO (OUT)     harfbuzz:              2.4.0-hee91db6_5              conda-forge
16:58:57 BIOCONDA INFO (OUT)     hdbscan:               0.8.26-py38h8790de6_1         conda-forge
16:58:57 BIOCONDA INFO (OUT)     hdf5:                  1.10.6-nompi_h3c11f04_100     conda-forge
16:58:57 BIOCONDA INFO (OUT)     hicolor-icon-theme:    0.17-1                        conda-forge
16:58:57 BIOCONDA INFO (OUT)     icu:                   67.1-he1b5a44_0               conda-forge
16:58:57 BIOCONDA INFO (OUT)     joblib:                0.16.0-py_0                   conda-forge
16:58:57 BIOCONDA INFO (OUT)     jpeg:                  9d-h516909a_0                 conda-forge
16:58:57 BIOCONDA INFO (OUT)     kiwisolver:            1.2.0-py38hbf85e49_0          conda-forge
16:58:57 BIOCONDA INFO (OUT)     krb5:                  1.17.1-hfafb76e_1             conda-forge
16:58:57 BIOCONDA INFO (OUT)     lcms2:                 2.11-hbd6801e_0               conda-forge
16:58:57 BIOCONDA INFO (OUT)     ld_impl_linux-64:      2.34-h53a641e_7               conda-forge
16:58:57 BIOCONDA INFO (OUT)     libblas:               3.8.0-11_openblas             conda-forge
16:58:57 BIOCONDA INFO (OUT)     libcblas:              3.8.0-11_openblas             conda-forge
16:58:57 BIOCONDA INFO (OUT)     libcroco:              0.6.13-h8d621e5_1             conda-forge
16:58:57 BIOCONDA INFO (OUT)     libcups:               2.2.12-hf10b501_1             conda-forge
16:58:57 BIOCONDA INFO (OUT)     libedit:               3.1.20191231-h46ee950_1       conda-forge
16:58:57 BIOCONDA INFO (OUT)     libffi:                3.2.1-he1b5a44_1007           conda-forge
16:58:57 BIOCONDA INFO (OUT)     libgcc-ng:             9.2.0-h24d8f2e_2              conda-forge
16:58:57 BIOCONDA INFO (OUT)     libgfortran-ng:        7.5.0-hdf63c60_6              conda-forge
16:58:57 BIOCONDA INFO (OUT)     libgomp:               9.2.0-h24d8f2e_2              conda-forge
16:58:57 BIOCONDA INFO (OUT)     libiconv:              1.15-h516909a_1006            conda-forge
16:58:57 BIOCONDA INFO (OUT)     liblapack:             3.8.0-11_openblas             conda-forge
16:58:57 BIOCONDA INFO (OUT)     libopenblas:           0.3.6-h6e990d7_6              conda-forge
16:58:57 BIOCONDA INFO (OUT)     libpng:                1.6.37-hed695b0_1             conda-forge
16:58:57 BIOCONDA INFO (OUT)     librsvg:               2.49.3-h33a7fed_0             conda-forge
16:58:57 BIOCONDA INFO (OUT)     libstdcxx-ng:          9.2.0-hdf63c60_2              conda-forge
16:58:57 BIOCONDA INFO (OUT)     libtiff:               4.1.0-hc7e4089_6              conda-forge
16:58:57 BIOCONDA INFO (OUT)     libuuid:               2.32.1-h14c3975_1000          conda-forge
16:58:57 BIOCONDA INFO (OUT)     libwebp-base:          1.1.0-h516909a_3              conda-forge
16:58:57 BIOCONDA INFO (OUT)     libxcb:                1.13-h14c3975_1002            conda-forge
16:58:57 BIOCONDA INFO (OUT)     libxml2:               2.9.10-h72b56ed_1             conda-forge
16:58:57 BIOCONDA INFO (OUT)     lz4-c:                 1.9.2-he1b5a44_1              conda-forge
16:58:57 BIOCONDA INFO (OUT)     mash:                  2.2.2-ha61e061_2              bioconda
16:58:57 BIOCONDA INFO (OUT)     matplotlib:            3.3.0-1                       conda-forge
16:58:57 BIOCONDA INFO (OUT)     matplotlib-base:       3.3.0-py38h91b0d89_1          conda-forge
16:58:57 BIOCONDA INFO (OUT)     ncurses:               6.2-he1b5a44_1                conda-forge
16:58:57 BIOCONDA INFO (OUT)     numpy:                 1.19.1-py38h8854b6b_0         conda-forge
16:58:57 BIOCONDA INFO (OUT)     olefile:               0.46-py_0                     conda-forge
16:58:57 BIOCONDA INFO (OUT)     openblas:              0.3.6-h6e990d7_6              conda-forge
16:58:57 BIOCONDA INFO (OUT)     openssl:               1.1.1g-h516909a_0             conda-forge
16:58:57 BIOCONDA INFO (OUT)     pandas:                1.0.5-py38hcb8c335_0          conda-forge
16:58:57 BIOCONDA INFO (OUT)     pango:                 1.42.4-h7062337_4             conda-forge
16:58:57 BIOCONDA INFO (OUT)     pcre:                  8.44-he1b5a44_0               conda-forge
16:58:57 BIOCONDA INFO (OUT)     pillow:                7.2.0-py38h9776b28_1          conda-forge
16:58:57 BIOCONDA INFO (OUT)     pip:                   20.1.1-py_1                   conda-forge
16:58:57 BIOCONDA INFO (OUT)     pixman:                0.38.0-h516909a_1003          conda-forge
16:58:57 BIOCONDA INFO (OUT)     poppunk:               2.1.1-py_0                    local
16:58:57 BIOCONDA INFO (OUT)     pp-sketchlib:          1.4.0-py38h212a85b_0          conda-forge
16:58:57 BIOCONDA INFO (OUT)     pthread-stubs:         0.4-h14c3975_1001             conda-forge
16:58:57 BIOCONDA INFO (OUT)     pycairo:               1.19.1-py38h323dad1_3         conda-forge
16:58:57 BIOCONDA INFO (OUT)     pygobject:             3.36.1-py38h2faf24e_1         conda-forge
16:58:57 BIOCONDA INFO (OUT)     pyparsing:             2.4.7-pyh9f0ad1d_0            conda-forge
16:58:57 BIOCONDA INFO (OUT)     python:                3.8.5-h425cb1d_1_cpython      conda-forge
16:58:57 BIOCONDA INFO (OUT)     python-dateutil:       2.8.1-py_0                    conda-forge
16:58:57 BIOCONDA INFO (OUT)     python_abi:            3.8-1_cp38                    conda-forge
16:58:57 BIOCONDA INFO (OUT)     pytz:                  2020.1-pyh9f0ad1d_0           conda-forge
16:58:57 BIOCONDA INFO (OUT)     rapidnj:               v2.3.2-hc9558a2_2             bioconda
16:58:57 BIOCONDA INFO (OUT)     readline:              8.0-he28a2e2_2                conda-forge
16:58:57 BIOCONDA INFO (OUT)     scikit-learn:          0.23.1-py38h3a94b23_0         conda-forge
16:58:57 BIOCONDA INFO (OUT)     scipy:                 1.5.2-py38h8c5af15_0          conda-forge
16:58:57 BIOCONDA INFO (OUT)     setuptools:            49.2.0-py38h32f6830_0         conda-forge
16:58:57 BIOCONDA INFO (OUT)     sigcpp-2.0:            2.10.3-he1b5a44_0             conda-forge
16:58:57 BIOCONDA INFO (OUT)     six:                   1.15.0-pyh9f0ad1d_0           conda-forge
16:58:57 BIOCONDA INFO (OUT)     sparsehash:            2.0.3-hf484d3e_1000           conda-forge
16:58:57 BIOCONDA INFO (OUT)     sqlite:                3.32.3-hcee41ef_1             conda-forge
16:58:57 BIOCONDA INFO (OUT)     threadpoolctl:         2.1.0-pyh5ca1d4c_0            conda-forge
16:58:57 BIOCONDA INFO (OUT)     tk:                    8.6.10-hed695b0_0             conda-forge
16:58:57 BIOCONDA INFO (OUT)     tornado:               6.0.4-py38h1e0a361_1          conda-forge
16:58:57 BIOCONDA INFO (OUT)     wheel:                 0.34.2-py_1                   conda-forge
16:58:57 BIOCONDA INFO (OUT)     xorg-kbproto:          1.0.7-h14c3975_1002           conda-forge
16:58:57 BIOCONDA INFO (OUT)     xorg-libice:           1.0.10-h516909a_0             conda-forge
16:58:57 BIOCONDA INFO (OUT)     xorg-libsm:            1.2.3-h84519dc_1000           conda-forge
16:58:57 BIOCONDA INFO (OUT)     xorg-libx11:           1.6.9-h516909a_0              conda-forge
16:58:57 BIOCONDA INFO (OUT)     xorg-libxau:           1.0.9-h14c3975_0              conda-forge
16:58:57 BIOCONDA INFO (OUT)     xorg-libxdmcp:         1.1.3-h516909a_0              conda-forge
16:58:57 BIOCONDA INFO (OUT)     xorg-libxext:          1.3.4-h516909a_0              conda-forge
16:58:57 BIOCONDA INFO (OUT)     xorg-libxrender:       0.9.10-h516909a_1002          conda-forge
16:58:57 BIOCONDA INFO (OUT)     xorg-renderproto:      0.11.1-h14c3975_1002          conda-forge
16:58:57 BIOCONDA INFO (OUT)     xorg-xextproto:        7.3.0-h14c3975_1002           conda-forge
16:58:57 BIOCONDA INFO (OUT)     xorg-xproto:           7.0.31-h14c3975_1007          conda-forge
16:58:57 BIOCONDA INFO (OUT)     xz:                    5.2.5-h516909a_1              conda-forge
16:58:57 BIOCONDA INFO (OUT)     zlib:                  1.2.11-h516909a_1006          conda-forge
16:58:57 BIOCONDA INFO (OUT)     zstd:                  1.4.5-h6597ccf_1              conda-forge
16:58:57 BIOCONDA INFO (OUT) 
.16:59:54 BIOCONDA INFO (OUT) Preparing transaction: ...working... done
17:00:00 BIOCONDA INFO (OUT) Verifying transaction: ...working... done
17:00:15 BIOCONDA INFO (OUT) Executing transaction: ...working... done
17:00:15 BIOCONDA INFO (OUT) ERROR:conda.core.link:An error occurred while installing package 'conda-forge::gtk3-3.24.21-h45fd312_0'.
17:00:15 BIOCONDA INFO (OUT) ERROR conda.core.link:_execute(700): An error occurred while installing package 'conda-forge::gtk3-3.24.21-h45fd312_0'.
17:00:19 BIOCONDA INFO (OUT) Rolling back transaction: ...working... done
17:00:19 BIOCONDA INFO (OUT) Traceback (most recent call last):
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/bin/conda-build", line 11, in <module>
17:00:19 BIOCONDA INFO (OUT)     sys.exit(main())
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 474, in main
17:00:19 BIOCONDA INFO (OUT)     execute(sys.argv[1:])
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 465, in execute
17:00:19 BIOCONDA INFO (OUT)     verify=args.verify, variants=args.variants)
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/api.py", line 209, in build
17:00:19 BIOCONDA INFO (OUT)     notest=notest, need_source_download=need_source_download, variants=variants)
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/build.py", line 2870, in build_tree
17:00:19 BIOCONDA INFO (OUT)     test(pkg, config=metadata.config.copy(), stats=stats)
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/build.py", line 2668, in test
17:00:19 BIOCONDA INFO (OUT)     is_conda=metadata.name() == 'conda')
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/environ.py", line 875, in create_env
17:00:19 BIOCONDA INFO (OUT)     execute_actions(actions, index)
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda/common/io.py", line 88, in decorated
17:00:19 BIOCONDA INFO (OUT)     return f(*args, **kwds)
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda/plan.py", line 321, in execute_actions
17:00:19 BIOCONDA INFO (OUT)     execute_instructions(plan, index, verbose)
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda/plan.py", line 533, in execute_instructions
17:00:19 BIOCONDA INFO (OUT)     cmd(state, arg)
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda/instructions.py", line 73, in UNLINKLINKTRANSACTION_CMD
17:00:19 BIOCONDA INFO (OUT)     unlink_link_transaction.execute()
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda/core/link.py", line 249, in execute
17:00:19 BIOCONDA INFO (OUT)     self._execute(tuple(concat(interleave(itervalues(self.prefix_action_groups)))))
17:00:19 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda/core/link.py", line 716, in _execute
17:00:19 BIOCONDA INFO (OUT)     rollback_excs,
17:00:19 BIOCONDA INFO (OUT) conda.CondaMultiError: post-link script failed for package conda-forge::gtk3-3.24.21-h45fd312_0
17:00:19 BIOCONDA INFO (OUT) location of failed script: /opt/conda/conda-bld/poppunk_1595609780416/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/.gtk3-post-link.sh
17:00:19 BIOCONDA INFO (OUT) ==> script messages <==
17:00:19 BIOCONDA INFO (OUT) <None>
17:00:19 BIOCONDA INFO (OUT) ==> script output <==
17:00:19 BIOCONDA INFO (OUT) stdout:
17:00:19 BIOCONDA INFO (OUT) stderr: /opt/conda/conda-bld/poppunk_1595609780416/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/gtk-query-immodules-3.0: error while loading shared libraries: libXi.so.6: cannot open shared object file: No such file or directory
17:00:19 BIOCONDA INFO (OUT) 
17:00:19 BIOCONDA INFO (OUT) return code: 127
17:00:19 BIOCONDA INFO (OUT) 
17:00:19 BIOCONDA INFO (OUT) ()
17:00:19 BIOCONDA INFO (OUT)

@johnlees
Copy link
Contributor

Key line I think is: error while loading shared libraries: libXi.so.6: cannot open shared object file: No such file or directory
Does X11 need to be listed manually, or is there a possible problem with the version on the build machine?

@johnlees
Copy link
Contributor

@bioconda/core To get this to pass I needed to add all of the xorg recipes to get gtk3 (a dependency of matplotlib-base) to find X11 when it runs. Surely this cannot be correct? Or have you seen this in other recipes?
There is some discussion here and here which suggests a required setup of X11 in yum_requirements.txt

@johnlees johnlees added the please review & merge set to ask for merge label Jul 28, 2020
@johnlees johnlees mentioned this pull request Jul 31, 2020
Closed
@mbargull
Copy link
Member

(busy atm. Will comment later, sorry, in the meantime relevant references below)
refs:

@johnlees
Copy link
Contributor

Also: conda-forge/gtk3-feedstock#16

@johnlees
Copy link
Contributor

I guess we can't have a yum_requirements.txt on bioconda?

@mbargull
Copy link
Member

Hi, I am sorry haven't explained in detail/at all.
But you are correct, unlike conda-forge we don't support installing RPM packages during build/build setup on Bioconda.
I'll have to think about what the best approach is going to be here. I'll get back to you tomorrow.

@ryanvolz
Copy link

Hello from the conda-forge gtk3 package! That bioconda does not allow/use yum_requirements.txt is a very persuasive data point in favor of switching the gtk3 package to depend on the conda-forge Xorg packages instead of the CDTs.

Probably no one here knows, but how does bioconda deal with packages that depend on Qt? The Qt package has been around a lot longer uses CDTs for X11, so I'd have thought that this problem would have come up in the test phase of some Qt-depending package. Maybe there are none or they are not run in the test phase.

For this specific issue, assuming that a user's system will have X11 installed, you can probably make this work in the test phase by adding all of those xorg- packages as test requirements (in the test section) instead of run requirements. That at least doesn't force them to be installed by end users. In the long term, we'll want to do something with the gtk3 package to make this work better.

@epruesse
Copy link
Member

epruesse commented Aug 2, 2020

I've dealt with the "something needs X11" issue with SINA by breaking the arb-bio (ARB) package into sub-packages. Normally, sina -> arb-bio -> openmotif -> xorg.... With arb-bio split into X11 and non-X11 subpackages, I can have sina depend only on the non-X11 parts, so installing the latter won't drag in all the X11 stuff.

What's the story with matplot-base? If it functions without X11, it should be split to break dependency tree.

Not sure how many packages use this. openmotif was the first conda-forge package to use it and arb-bio the first bioconda one. But it's the way that almost all distros handle this issue (all excepting those that rebuild custom packages).

@mbargull
Copy link
Member

mbargull commented Aug 3, 2020

Hi, wasn't able to get to this on the weekend.
So, before conda-forge/docker-images@f598f6e#diff-eaf90fc2058811bb1d22f2204a4775a8 we had the necessary system libraries installed in our build environment. Thus, for Qt packages etc., we had only to tell people to use extra: container: extended-base: True such that those libraries were also available on our additional container-based tests (and through that, also for container images published via BioContainers).

A special case then was matplotlib/matplotlib-base because you can tell it on runtime via MPLBACKEND=agg python ... to not use those libraries. Hence the choice for the packager is either use extra: container: extended-base: True or tell the (container) users to use MPLBACKEND=agg to get smaller dependencies/containers.

Now, what is new with the change in the build container image, is that the builds may actually fail because either the build itself or the test in that container encounter the missing libraries. To fix that, we'd have to re-introduce those libraries in the container.
Regarding failing tests, I'd like us to skip the tests in the build container all together at some point. (Meaning this is a Bioconda build system change that might happen some time in the future.)
For the build itself things can get more complicated. If we only need the libraries to link against, we could use CDT packages. If binaries from packages in requirements: host are actually run during the build, we'd have to supply the conda-forge::xorg-* packages. (Ideally those packages could declare some dependency that expresses "install the conda-forge::xorg-* package only if the system does not supply the needed libraries -- we don't have such thing currently, I think, and have to work on the conda-forge side on this (IDK what the previous discussion about this entailed).)

In sum: To really fix the issue and make things reasonably comfortable for the packager, we have to adjust Bioconda's build system and look into a conda-forge::xorg-*-or-system-lib-choice mechanism at conda-forge. Until then, we have to work around it.
I'll adjust this PR with a workaround.

@mbargull
Copy link
Member

mbargull commented Aug 3, 2020

@epruesse

What's the story with matplot-base? If it functions without X11, it should be split to break dependency tree.

We have matplotlib and matplotlib-base. The former depends on Qt (to provide a Qt-based backend) and to get fewer dependencies the latter *-base was created. If Qt isn't available, Matplotlib will use the Tk backend. This will fail because python -c 'import tkinter' fails if the libraries are not available.
Now, what I did not know (and just now noticed while fact-checking so I don't write non-sense ;P) is that Maplotlib >=3.0 uses Agg backend as a default that is not dependent on the X11 libs. Which is neat, because that means we don't have to explicitly MPLBACKEND=agg ... anymore!

$ python -c 'import matplotlib as m; print(m.__version__, m.get_backend())'
2.2.5 TkAgg

$ python -c 'import matplotlib as m; print(m.__version__, m.get_backend())'
3.3.0 agg

@mbargull
Copy link
Member

mbargull commented Aug 3, 2020 

stderr: /usr/local/bin/gtk-query-immodules-3.0: error while loading shared libraries: libXi.so.6: cannot open shared object file: No such file or directory

So, this means this uses libraries that are not even available in the bioconda/extended-base-image.
Which means at least some of the xorg-* dependencies are needed to create a functioning BioContainers image.

@johnlees
Copy link
Contributor

johnlees commented Aug 3, 2020

Thanks for all the detailed comments from everyone.

One thing I figured out, and hope I didn't confuse by, is that the error here was caused by adding the graph-tool conda-forge package (not matplotlib[-base]) which depends on gtk3 from some drawing functions.

I did get this to pass in 33c2202 by adding the xorg dependencies. Shall I re-add these, or do you have a better solution (I'm sorry, I wasn't able to follow all of the discussion above)

@mbargull
Copy link
Member

mbargull commented Aug 3, 2020

@johnlees, I didn't check which Xorg packages were needed before. Hence I assumed those were just the once that are provided by bioconda/extended-base-image, which are just

# dpkg-query -f '${Package}\n' -W libx\* 
libx11-6
libx11-data
libx11-xcb1
libxau6
libxcb-dri2-0
libxcb-dri3-0
libxcb-glx0
libxcb-present0
libxcb-sync1
libxcb-xlib0
libxcb1
libxdamage1
libxdmcp6
libxext6
libxfixes3
libxml2
libxshmfence1
libxsloader-perl
libxxf86vm1

But since we need other xorg-* packages anyway, I now moved them back to requirements: run.

@johnlees
Copy link
Contributor

johnlees commented Aug 3, 2020

@BiocondaBot please merge

@BiocondaBot
Copy link
Collaborator Author

I will attempt to upload artifacts and merge this PR. This may take some time, please have patience.

@BiocondaBot BiocondaBot merged commit f0ed9ef into master Aug 3, 2020
@BiocondaBot BiocondaBot deleted the bump/poppunk branch August 3, 2020 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@johnlees johnlees johnlees approved these changes

Assignees
No one assigned
Labels
autobump Automatic Version Update new version please review & merge set to ask for merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@BiocondaBot @johnlees @mbargull @ryanvolz @epruesse