Pigeon and Isoseq3 no longer work

[jamestwebber]

Operating system
Tested this on linux (Linux 4.19.0-22-cloud-amd64 x86_64) and Mac OS in fresh conda environments: conda create -n pbtest -c bioconda pbpigeon isoseq3

Package name
The latest releases of pigeon and isoseq3 have this issue. The output of tool --version is Segmentation fault. 🙃 But I assume it's supposed to be v1.0.0 and v3.8.2 respectively.

Conda environment

# packages in environment at /home/jupyter/mambaforge/envs/pbtest:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
isoseq3                   3.8.2                h9ee0642_0    bioconda
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
pbpigeon                  1.0.0                h9f5acd7_1    bioconda

Describe the bug
The tools do not execute whatsoever, even --help or --version results in an immediate Segmentation fault.

In macOS the shell complains that they can't be executed: exec format error: pigeon in zsh or annot execute binary file in bash.

Error message
See above. No stacktrace, just failure.

To Reproduce

conda create -n pbtest -c bioconda pbpigeon isoseq3
conda activate pbtest
pigeon --version  # "Segmentation fault"
isoseq3 --version  # ditto

Expected behavior
The tools should work. 😂

[armintoepfer]

They won't work on MacOS, as the pbbioconda main README tables states. But they work on linux. Following your exact commands:

(base) conda create -n pbtest -c bioconda pbpigeon isoseq3
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 22.9.0
  latest version: 22.11.1

Please update conda by running

    $ conda update -n base -c conda-forge conda



## Package Plan ##

  environment location: /pbi/flash/atoepfer/miniconda3-mamba/envs/pbtest

  added / updated specs:
    - isoseq3
    - pbpigeon


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _openmp_mutex-4.5          |            2_gnu          23 KB  conda-forge
    isoseq3-3.8.2              |       h9ee0642_0         1.6 MB  bioconda
    libgcc-ng-12.2.0           |      h65d4601_19         931 KB  conda-forge
    libgomp-12.2.0             |      h65d4601_19         455 KB  conda-forge
    libstdcxx-ng-12.2.0        |      h46fd767_19         4.3 MB  conda-forge
    pbpigeon-1.0.0             |       h9f5acd7_1         1.1 MB  bioconda
    ------------------------------------------------------------
                                           Total:         8.3 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge None
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu None
  isoseq3            bioconda/linux-64::isoseq3-3.8.2-h9ee0642_0 None
  libgcc-ng          conda-forge/linux-64::libgcc-ng-12.2.0-h65d4601_19 None
  libgomp            conda-forge/linux-64::libgomp-12.2.0-h65d4601_19 None
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-12.2.0-h46fd767_19 None
  pbpigeon           bioconda/linux-64::pbpigeon-1.0.0-h9f5acd7_1 None


Proceed ([y]/n)? y


Downloading and Extracting Packages
libgomp-12.2.0       | 455 KB    | ################################################################################################################################################################################## | 100%
libstdcxx-ng-12.2.0  | 4.3 MB    | ################################################################################################################################################################################## | 100%
_openmp_mutex-4.5    | 23 KB     | ################################################################################################################################################################################## | 100%
libgcc-ng-12.2.0     | 931 KB    | ################################################################################################################################################################################## | 100%
pbpigeon-1.0.0       | 1.1 MB    | ################################################################################################################################################################################## | 100%
isoseq3-3.8.2        | 1.6 MB    | ################################################################################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: |
##############################################################################
#                                                                            #
# PacBio(R) tools distributed via Bioconda are: pre-release versions, not    #
# necessarily ISO compliant, intended for Research Use Only and not for use  #
# in diagnostic procedures, intended only for command-line users, and        #
# possibly newer than the currently available SMRT(R) Analysis builds. While #
# efforts have been made to ensure that releases on Bioconda live up to the  #
# quality that PacBio strives for, we make no warranty regarding any         #
# Bioconda release.                                                          #
#                                                                            #
# As PacBio tools distributed via Bioconda are not covered by any service    #
# level agreement or the like, please *do not* contact a PacBio Field        #
# Applications Scientist or PacBio Customer Service for assistance with any  #
# Bioconda release. We instead provide an issue tracker for you to report    #
# issues to us at:                                                           #
#                                                                            #
#   https://github.com/PacificBiosciences/pbbioconda                         #
#                                                                            #
# We make no warranty that any such issue will be addressed,                 #
# to any extent or within any time frame.                                    #
#                                                                            #
# BSD 3-Clause Clear License                                                 #
#                                                                            #
# Please see https://github.com/PacificBiosciences/pbbioconda for            #
# information on License, Copyright and Disclaimer                           #
#                                                                            #
##############################################################################

done
#
# To activate this environment, use
#
#     $ conda activate pbtest
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Retrieving notices: ...working... done
(base) conda activate pbtest
(pbtest) which isoseq3
/pbi/flash/atoepfer/miniconda3-mamba/envs/pbtest/bin/isoseq3
(pbtest) isoseq3 --version
isoseq3 3.8.2 (commit v3.8.2)

Using:
  pbbam     : 2.2.0 (commit v2.2.0-1-g8c081f6)
  pbcopper  : 2.1.0 (commit v2.1.0)
  pbmm2     : 1.9.0 (commit v1.9.0-2-gbec8e28)
  minimap2  : 2.15
  parasail  : 2.1.3
  boost     : 1.77
  htslib    : 1.14
  zlib      : 1.2.11
(pbtest) pigeon --version
pigeon 1.0.0 (commit -v1.0.0)

Using:
  pbbam     : 2.2.0 (commit v2.2.0-1-g8c081f6)
  pbcopper  : 2.1.0 (commit v2.1.0)
  boost     : 1.77
  htslib    : 1.14
  zlib      : 1.2.11

[jamestwebber]

Nevertheless, it still isn't working on a GCP VM (running debian 10). What version of Linux is that?

It seems like it must be a configuration issue or some other requirement that isn't being specified. The previous versions worked, so presumably the issue stems from a recent change.

[jamestwebber]

They won't work on MacOS, as the pbbioconda main README tables states.

Yes, I don't actually care about MacOS but I tried it there to verify the issue. As an aside, you could prevent conda from even installing these packages on an unsupported OS by changing the recipe to specify linux only.

[armintoepfer]

We'll look into what changed. Thank you for reporting

[jamestwebber]

Update: I created a fresh VM with debian 11, installed micromamba and created a new environment:

micromamba create -n pbtest -c bioconda pbpigeon isoseq3

...

  + isoseq3     3.8.2  h9ee0642_0  bioconda/linux-64      2MB
  + pbpigeon    1.0.0  hdfd78af_0  bioconda/noarch        1MB

Interestingly it only installed these two packages, and nothing else. Not sure why that is (edit: I guess all the requirements are stemming from the new pigeon release). However, notice that it is the previous release of pbpigeon (hdfd78af_0 rather than h9f5acd7_1 above). This version doesn't crash! It seems like maybe the second release had an issue?

isoseq3 is the same version but also works now, which is confusing...

[armintoepfer]

Let's cut out conda. Can you try using the binaries from GitHub directly?

https://github.com/PacificBiosciences/IsoSeq/releases/download/v3.8.2/isoseq3

https://github.com/PacificBiosciences/pigeon/releases/download/v1.0.0/pigeon

[jamestwebber]

These work! So maybe it is an issue with the conda recipe...

[armintoepfer]

At least we got you unblocked for now!

[philoel]

Hey! I am having what looks to be the same or very similar experience, and I wanted to try Armin's suggestion to use the binaries. Both my isoseq3 and pigeon are giving the same issue as James described. I'm working on a linux compute cluster, and the initial post by James describes what I have exactly.

I'm hoping my issue is quick to fix; this is the first time I've tried to install binaries directly from Github into a conda environment, perhaps it's just me picking the wrong stackoverflow comment to trust.

When I run
pip install git+https://github.com/PacificBiosciences/pigeon/releases/download/v1.0.0/pigeon
or
pip install git+https://github.com/PacificBiosciences/IsoSeq/releases/download/v3.8.2/isoseq3

in both cases I get the error:

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting git+https://github.com/PacificBiosciences/IsoSeq/releases/download/v3.8.2/isoseq3
Cloning https://github.com/PacificBiosciences/IsoSeq/releases/download/v3.8.2/isoseq3 to /scratch/jobs/64742796/pip-req-build-ArAuFr
Running command git clone -q https://github.com/PacificBiosciences/IsoSeq/releases/download/v3.8.2/isoseq3 /scratch/jobs/64742796/pip-req-build-ArAuFr
remote: Not Found
fatal: repository 'https://github.com/PacificBiosciences/IsoSeq/releases/download/v3.8.2/isoseq3/' not found
ERROR: Command errored out with exit status 128: git clone -q https://github.com/PacificBiosciences/IsoSeq/releases/download/v3.8.2/isoseq3 /scratch/jobs/64742796/pip-req-build-ArAuFr Check the logs for full command output.

Any tips?

[jamestwebber]

I'm hoping my issue is quick to fix; this is the first time I've tried to install binaries directly from Github into a conda environment, perhaps it's just me picking the wrong stackoverflow comment to trust.

Those instructions aren't to install a binary directly, they're for a git repository.

You just need to download the isoseq3 binary from that link (if you click on it, it'll prompt you to save it) and put the file under your $PATH (a good choice is the bin directory of your current environment, that's where it would be normally).

[philoel]

Many thanks! Worked like a charm!