Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

osx perl-json-xs (and other packages) install perl-threaded #10100

Open
nathanweeks opened this issue Jul 26, 2018 · 6 comments

Comments

Projects
None yet
3 participants
@nathanweeks
Copy link
Contributor

commented Jul 26, 2018

On macOS, perl-json-xs (at least) depends on the deprecated perl-threaded, which is Perl 5.22.0. When perl-threaded is installed, it causes $CONDA_PREFIX/bin/perl to refer to its version of perl, causing problems when other perl modules that depend on the conda-forge 5.26.2 are installed (e.g., perl-perlio-gzip).

Pristine Miniconda3 installation w/ bioconda channels configured per http://bioconda.github.io/:

$ conda install perl-json
...
The following NEW packages will be INSTALLED:

    perl:              5.26.2-h16c6ff1_0 conda-forge
    perl-common-sense: 3.74-pl526_1      bioconda
    perl-json:         2.90-pl526_2      bioconda
    perl-json-xs:      2.34-0            bioconda
    perl-threaded:     5.22.0-10         bioconda
...
$ conda install perl-perlio-gzip
...
The following NEW packages will be INSTALLED:

    perl:             5.26.2-h16c6ff1_0    conda-forge
    perl-perlio-gzip: 0.19-pl526h1341992_3 bioconda

Installing both:

$ conda install perl-json perl-perlio-gzip
The following NEW packages will be INSTALLED:

    perl:              5.26.2-h16c6ff1_0    conda-forge
    perl-common-sense: 3.74-pl526_1         bioconda
    perl-json:         2.90-pl526_2         bioconda
    perl-json-xs:      2.34-0               bioconda
    perl-perlio-gzip:  0.19-pl526h1341992_3 bioconda
    perl-threaded:     5.22.0-10            bioconda

$CONDA_PREFIX/bin/perl (perl-threaded) can use the installed perl-json module, but not perl-perlio-gzip. Conversely, $CONDA_PREFIX/bin/perl5.26.2 can use perl-perlio-gzip, but not perl-json:

$ perl -v | head -n 2

This is perl 5, version 22, subversion 0 (v5.22.0) built for darwin-thread-multi-2level
$ perl5.26.2 -MPerlIO::gzip -e ''
$ perl -MPerlIO::gzip -e ''
Can't locate PerlIO/gzip.pm in @INC (you may need to install the PerlIO::gzip module) (@INC contains: /Volumes/scratch/miniconda3/lib/perl5/site_perl/5.22.0/darwin-thread-multi-2level /Volumes/scratch/miniconda3/lib/perl5/site_perl/5.22.0 /Volumes/scratch/miniconda3/lib/perl5/5.22.0/darwin-thread-multi-2level /Volumes/scratch/miniconda3/lib/perl5/5.22.0 .).
$ perl -MJSON -e ''
$ perl5.26.2 -MJSON -e ''
Can't locate JSON.pm in @INC (you may need to install the JSON module) (@INC contains: /Volumes/scratch/miniconda3/lib/site_perl/5.26.2/darwin-thread-multi-2level /Volumes/scratch/miniconda3/lib/site_perl/5.26.2 /Volumes/scratch/miniconda3/lib/5.26.2/darwin-thread-multi-2level /Volumes/scratch/miniconda3/lib/5.26.2 .).
BEGIN failed--compilation aborted.
@nathanweeks

This comment has been minimized.

Copy link
Contributor Author

commented Jul 26, 2018

This affects at least some Linux packages as well; e.g., perl-perlio-gzip and perl-bioperl (the latter of which has a dependency on perl-threaded somewhere).

In this case, $CONDA_PREFIX/bin/perl is perl5.26.2, and there isn't an executable for Perl 5.22.0. So PerlIO::gzip can be used, but bioperl cannot...

$ conda install perl-perlio-gzip perl-bioperl
...
$ perl --version | head -n 2

This is perl 5, version 26, subversion 2 (v5.26.2) built for x86_64-linux-thread-multi
$ perl -MPerlIO::gzip -e ''
$ perl -MBio::DB::SeqFeature::Store -e ''
Can't locate Bio/DB/SeqFeature/Store.pm in @INC (you may need to install the Bio::DB::SeqFeature::Store module) (@INC contains: /project/test/miniconda/lib/site_perl/5.26.2/x86_64-linux-thread-multi /project/test/miniconda/lib/site_perl/5.26.2 /project/test/miniconda/lib/5.26.2/x86_64-linux-thread-multi /project/test/miniconda/lib/5.26.2 .).
BEGIN failed--compilation aborted.
$ find $CONDA_PREFIX -name Store.pm
/project/test/miniconda/lib/perl5/site_perl/5.22.0/Bio/DB/SeqFeature/Store.pm

@nathanweeks nathanweeks referenced this issue Jul 27, 2018

Merged

Bump JBrowse version to 1.15.0 #10116

0 of 3 tasks complete
@acaprez

This comment has been minimized.

Copy link
Contributor

commented Aug 1, 2018

Yeah, I noticed this awhile back as well. I've been slowly working through small batches of packages trying to fix things.

@acaprez

This comment has been minimized.

Copy link
Contributor

commented Aug 1, 2018

It's even worse, as some packages were rebuilt after the switch to 5.26.2 and perl-threaded was pulled in. So they are tagged as 5.26 builds, but the package contents are installed in the wrong (5.22.0) location. Example of that here.

@nathanweeks

This comment has been minimized.

Copy link
Contributor Author

commented Aug 2, 2018

Thanks @acaprez , I appreciate your efforts to go clean this up!

@bgruening

This comment has been minimized.

Copy link
Member

commented Aug 2, 2018

Sorry, to be late here. @acaprez I thought I removed all occurrences of perl-threaded a while back. Are there still packages depending on it? Should we release a new/last version of perl-threaded pointing to the new perl release? Will this help?

@acaprez

This comment has been minimized.

Copy link
Contributor

commented Aug 3, 2018

No worries @bgruening. Yes, there are still perl packages that use it - for example, perl-xml-twig and perl-xml-path. I suspect what happened is some packages were added to the blacklist and so were never rebuilt with perl 5.26.2, but were still pulled in as dependencies for other packages, thus pulling in perl-threaded.

I'd be hesitant to try anything too clever with the perl-threaded package. In hindsight, using that name wasn't the best idea, as conda build will happily install perl-threaded and perl in the same environment. My suggestion I suppose would be to remove the perl-threaded package completely from the bioconda channel (or at least the OS X packages), and then do a force rebuilding of all the perl packages on OS X. Anything that still tries to pull in perl-threaded will fail immediately then with a dependency error, and be easy to spot. This will also make creation of any environment still using perl-threaded fail, but IMHO it's better to have a hard failure immediately than successfully create an env that is half-broken. Ideally we could create a dependency DAG of all the perl packages and rebuild them in batches according to their level in the graph, but I haven't looked into how difficult it would be to generate that graph.

alexeigurevich added a commit that referenced this issue Aug 21, 2018

QUAST 5.0.0: circos dependency added
Circos dependency added to meta.yaml to prevent bioconda revert back to QUAST 4.6.3 when installing Circos.
The issue is caused by Perl packages required by Circos and built with deprecated Perl 5.22.0 on macOS. See #10100 for more details. 

Note that QUAST 4.6.3 has pretty the same dependencies as 5.0.0 (at least it also requires Perl) but it was built using the previous version of bioconda and Perl version is not specified there explicitly. 

Also note: if Circos is installed, QUAST can draw Circos plots automatically (--circos option). This functionality was added since version 5.0

@alexeigurevich alexeigurevich referenced this issue Aug 21, 2018

Merged

QUAST 5.0.0: circos dependency added #10582

3 of 5 tasks complete

alexeigurevich added a commit that referenced this issue Aug 21, 2018

QUAST 5.0.0: circos dependency added (#10582)
Circos dependency added to meta.yaml to prevent bioconda revert back to QUAST 4.6.3 when installing Circos.
The issue is caused by Perl packages required by Circos and built with deprecated Perl 5.22.0 on macOS. See #10100 for more details. 

Note that QUAST 4.6.3 has pretty the same dependencies as 5.0.0 (at least it also requires Perl) but it was built using the previous version of bioconda and Perl version is not specified there explicitly. 

Also note: if Circos is installed, QUAST can draw Circos plots automatically (--circos option). This functionality was added since version 5.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.