New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clarification on Bioconductor packages #8947

Open
daler opened this Issue May 17, 2018 · 18 comments

Comments

Projects
None yet
6 participants
@daler
Member

daler commented May 17, 2018

Hi @bioconda/all -

Bioconductor 3.7 was released earlier this month. We all want to use the new packages for our analyses, and PRs for updating Bioconductor packages have been starting to come in. I wanted to provide some explanation in one place why we unfortunately cannot merge these updates yet.

The major block is that BioC 3.7 needs R 3.5. This means to build updated BioC 3.7 conda packages we need to have a corresponding R 3.5 conda package. This will not be available for a while (late June? xref conda-forge/r-base-feedstock#43 (comment)), so we can't start updating Bioconductor packages at least until then.

Some explanation: The conda-forge folks build and maintain the conda R package, and do an amazing job with all the complexities of simultaneously building in on Linux, macOS, and Windows. Doing this takes substantial effort. Given the limited resources available to do this work and update all dependent R packages, it is only possible to support a single R conda package per minor version. The logical choice is to build the x.x.1 version (3.5.1) which incorporates the first round of bugfixes after a minor release. R 3.5.1 comes out in June, and the idea is to build it when it comes out.

@daler

This comment has been minimized.

Member

daler commented May 17, 2018

Updates: some PRs with BioC 3.7 updates were merged the past couple of days. Those packages were built and uploaded to the channel, causing other Bioconductor 3.6 packages to break.

Example error message:

Error: Functions found when exporting methods from the namespace ‘IRanges’
which are not S4 generic: ‘endoapply’, ‘mendoapply’

How to reproduce the error (prior to the resolution described below):

conda create -y -n s4 bioconductor-iranges
# included these:
#   bioconductor-biocgenerics: 0.26.0-r3.4.1_0 bioconda
#   bioconductor-iranges:      2.12.0-r3.4.1_0 bioconda
#   bioconductor-s4vectors:    0.18.1-r3.4.1_0 bioconda

source activate s4
Rscript -e "library(IRanges)"

To resolve this, on the bioconda channel I have just marked the following packages with the broken label so they will not install by default. You can still install them if you specify the additional argument --channel bioconda/labels/broken to conda.

Linux packages marked as broken:

  • linux-64/bioconductor-genomicranges-1.32.3-r3.4.1_0.tar.bz2
  • linux-64/bioconductor-genomeinfodb-1.16.0-r3.4.1_0.tar.bz2
  • linux-64/bioconductor-xvector-0.20.0-r3.4.1_0.tar.bz2
  • linux-64/bioconductor-iranges-2.14.10-r3.4.1_0.tar.bz2
  • linux-64/bioconductor-s4vectors-0.18.1-r3.4.1_0.tar.bz2
  • linux-64/bioconductor-s4vectors-0.18.2-r3.4.1_0.tar.bz2
  • linux-64/bioconductor-biocgenerics-0.26.0-r3.4.1_0.tar.bz2

OSX packages marked as broken:

  • osx-64/bioconductor-genomicranges-1.32.3-r3.4.1_0.tar.bz2
  • osx-64/bioconductor-genomeinfodb-1.16.0-r3.4.1_0.tar.bz2
  • osx-64/bioconductor-xvector-0.20.0-r3.4.1_0.tar.bz2
  • osx-64/bioconductor-iranges-2.14.10-r3.4.1_0.tar.bz2
  • osx-64/bioconductor-s4vectors-0.18.1-r3.4.1_0.tar.bz2
  • osx-64/bioconductor-s4vectors-0.18.2-r3.4.1_0.tar.bz2
  • osx-64/bioconductor-biocgenerics-0.26.0-r3.4.1_0.tar.bz2
@jdblischak

This comment has been minimized.

Member

jdblischak commented Jun 6, 2018

R 3.5.1 will be released on July 2

https://stat.ethz.ch/pipermail/r-announce/2018/000629.html

@federicomarini

This comment has been minimized.

Contributor

federicomarini commented Jul 2, 2018

Indeed released according to the timeline!

https://twitter.com/pdalgd/status/1013697234091864069

Once it is there on conda forge, I can also proceed with #8803. Let's hope it goes down smooth on that side, there's a lot of Bioc packages which could get benefit of this ;)

@dpryan79

This comment has been minimized.

Contributor

dpryan79 commented Sep 25, 2018

Is there any update on this? R 3.5.1 is in conda-forge and has been added to conda-forge-pinning in release 2018.08.28.

@federicomarini

This comment has been minimized.

Contributor

federicomarini commented Sep 25, 2018

I would also be extremely interested in knowing what the next steps could be, the iSEE package (#8803) can find another nice spot as conda recipe ;)

@bgruening

This comment has been minimized.

Contributor

bgruening commented Sep 25, 2018

We have been rebuilding r-* packages on conda-forge since weeks now. If someone wants to update our pinning, I think we can start the migration ... the only Q is if this is needed urgently, we also could wait for the GCC migration and the new bioc release ... both finsished/released in a month (?).

@federicomarini

This comment has been minimized.

Contributor

federicomarini commented Sep 25, 2018

We have been rebuilding r-* packages on conda-forge since weeks now. If someone wants to update our pinning, I think we can start the migration ... the only Q is if this is needed urgently, we also could wait for the GCC migration and the new bioc release ... both finsished/released in a month (?).

Can't say how urgent this is, it was only to have the package readily installable also via conda.
The new bioc release would anyway not change the R major version.
All in all, we all want things getting done good rather than in hurry... I guess it is a tough call what gets the priority here 🤷‍♂️- I'm of course fine with any decision.

@dpryan79

This comment has been minimized.

Contributor

dpryan79 commented Sep 25, 2018

The question becomes at what point the GCC migration can be declared "finished". Even the CB3 migration here is only "finished" at best, given the 1000+ packages still blacklisted. My concern is that we're getting into a "just one more thing!" cycle with some of these transitions, when there will always be something that could be waited on.

@bgruening

This comment has been minimized.

Contributor

bgruening commented Sep 25, 2018

I was mailing with @daler to get a clear picture of what we should do ... We don't know as it seems. So if the community is pushing for a rebuild, let's do this!

  1. update all Bioconductor packages with the Bioconductor skeleton to the latest version
  2. create PR against bulk and get a rough review
  3. merge PR into bulk and babysit it for a while ...
  • I guess we need to rebuild still some r-* packages in conda-forge
@dpryan79

This comment has been minimized.

Contributor

dpryan79 commented Sep 25, 2018

I imagine something in bioconda-utils will need to be changed too, or does bioconda-recipes use a particular version that could be changed in a PR to bulk?

@bgruening

This comment has been minimized.

Contributor

bgruening commented Sep 25, 2018

We need to update the dependency on conda-forge-pinning afaik. The rest we hopefully have done in bioconda/bioconda-utils#338 already.

Maybe we need to fix/adjust the skeleton generator. That I can not forsee.

@jdblischak

This comment has been minimized.

Member

jdblischak commented Sep 25, 2018

Is there any update on this? R 3.5.1 is in conda-forge and has been added to conda-forge-pinning in release 2018.08.28.

@dpryan79 Of the 1,239 R packages on conda-forge, 359 of them have been built for R 3.5.1.

@dpryan79

This comment has been minimized.

Contributor

dpryan79 commented Sep 25, 2018

25% finished on conda-forge isn't the worst place to start.

@bgruening

This comment has been minimized.

Contributor

bgruening commented Sep 25, 2018

@jdblischak That sounds depressing ;)

@jdblischak

This comment has been minimized.

Member

jdblischak commented Sep 25, 2018

25% finished on conda-forge isn't the worst place to start.
That sounds depressing ;)

As they say, the glass is a fourth full 😀

@scottgigante

This comment has been minimized.

Contributor

scottgigante commented Oct 14, 2018

Any advice on how to install Bioconductor packages with conda in the meantime? My build is breaking because of GenomeInfoDb and I'm not sure how to make it work.

Loading required package: GenomeInfoDb
Error: package or namespace load failed for "GenomeInfoDb" in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/gigante.s/.conda/envs/my_bioconda_env/lib/R/library/RCurl/libs/RCurl.so':
  /home/gigante.s/.conda/envs/my_bioconda_env/lib/R/library/RCurl/libs/../../../.././libcom_err.so.3: symbol k5_strerr
or_r, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference
Error: package "GenomeInfoDb" could not be loaded

My environment.yml below:
conda_env.txt

@dpryan79

This comment has been minimized.

Contributor

dpryan79 commented Oct 15, 2018

@scottgigante Are all of the packages you need available in bioconductor 3.7? I haven't checked everything in that conda env, but at least GenomeInfoDb has been updated for both R 3.5 and 3.4. Since it's passed mulled testing I assume it doesn't have this issue.

@scottgigante

This comment has been minimized.

Contributor

scottgigante commented Oct 15, 2018

Hmm, yes they are. I'm not doing anything intentionally to get the dev version so maybe this is an unrelated issue. Apologies for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment