Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Installation of taiyaki #12

Closed
pwh124 opened this issue Mar 17, 2022 · 7 comments
Closed

Installation of taiyaki #12

pwh124 opened this issue Mar 17, 2022 · 7 comments

Comments

@pwh124
Copy link

pwh124 commented Mar 17, 2022

Hello!

I am attempting to get everything prepared for training remora models.

I have megalodon installed and I am attempting to install taiyaki. I followed what was suggested in the README, namely:
Remora data preparation begins from Taiyaki mapped signal files generally produced from Megalodon containing modified base annotations. This requires installation of Taiyaki via pip install git+https://github.com/nanoporetech/taiyaki.

However, when I run this command in a clean python virtual environment, I encounter the following error:
Screen Shot 2022-03-17 at 4 11 17 PM

Any thoughts on why this is happening? Are there any specific requirements needed to install taiyaki in this way?

Thanks,
Paul
@marcus1487
Copy link
Collaborator

It looks as though there is more error text above the screenshot. It could be that some of the above text is more informative, but from this snippet it appears to be an issue with the compiler having access to the OMP library. I've seen issues like this with conda installations. Could you try using a vanilla python virtual environment? (e.g. python3.8 -m venv venv; source venv/bin/activate; pip install git+https://github.com/nanoporetech/taiyaki)

@pwh124
Copy link
Author

pwh124 commented Mar 17, 2022

Thanks Marcus!

Similar error to before when running python3.8 -m venv venv; source venv/bin/activate; pip install git+https://github.com/nanoporetech/taiyaki

    Running setup.py install for taiyaki ... error
    ERROR: Command errored out with exit status 1:
     command: /home/phook2/software/venv/bin/python3.8 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-rpkzkcow/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-rpkzkcow/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-k30tu_5i/install-record.txt --single-version-externally-managed --compile --install-headers /home/phook2/software/venv/include/site/python3.8/taiyaki
         cwd: /tmp/pip-req-build-rpkzkcow/
    Complete output (87 lines):
    Compiling taiyaki/squiggle_match/squiggle_match.pyx because it changed.
    Compiling taiyaki/ctc/ctc.pyx because it changed.
    Compiling taiyaki/decodeutil/decodeutil.pyx because it changed.
    [1/3] Cythonizing taiyaki/ctc/ctc.pyx
    /home/phook2/software/venv/lib/python3.8/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-req-build-rpkzkcow/taiyaki/ctc/ctc.pyx
      tree = Parsing.p_module(s, pxd, full_module_name)
    [2/3] Cythonizing taiyaki/decodeutil/decodeutil.pyx
    /home/phook2/software/venv/lib/python3.8/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-req-build-rpkzkcow/taiyaki/decodeutil/decodeutil.pyx
      tree = Parsing.p_module(s, pxd, full_module_name)
    [3/3] Cythonizing taiyaki/squiggle_match/squiggle_match.pyx
    /home/phook2/software/venv/lib/python3.8/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-req-build-rpkzkcow/taiyaki/squiggle_match/squiggle_match.pyx
      tree = Parsing.p_module(s, pxd, full_module_name)
    WARNING: The wheel package is not available.
    WARNING: The wheel package is not available.
    WARNING: The wheel package is not available.
    WARNING: The wheel package is not available.
    WARNING: The wheel package is not available.
    WARNING: The wheel package is not available.
    WARNING: The wheel package is not available.
    WARNING: The wheel package is not available.
    WARNING: The wheel package is not available.
    WARNING: The wheel package is not available.
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/fileio.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/json.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/prepare_mapping_funcs.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/iterators.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/fast5utils.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/signal.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/activation.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/chunk_selection.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/common_cmdargs.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/mapped_signal_files.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/bio.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/alphabet.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/loss.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/decode.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/basecall_helpers.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/signal_mapping.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/qscores.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/__init__.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/constants.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/cmdargs.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/maths.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/flipflopfings.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/helpers.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/flipflop_remap.py -> build/lib.linux-x86_64-3.8/taiyaki
    copying taiyaki/layers.py -> build/lib.linux-x86_64-3.8/taiyaki
    creating build/lib.linux-x86_64-3.8/taiyaki/cupy_extensions
    copying taiyaki/cupy_extensions/flipflop.py -> build/lib.linux-x86_64-3.8/taiyaki/cupy_extensions
    copying taiyaki/cupy_extensions/__init__.py -> build/lib.linux-x86_64-3.8/taiyaki/cupy_extensions
    creating build/lib.linux-x86_64-3.8/taiyaki/squiggle_match
    copying taiyaki/squiggle_match/__init__.py -> build/lib.linux-x86_64-3.8/taiyaki/squiggle_match
    creating build/lib.linux-x86_64-3.8/taiyaki/decodeutil
    copying taiyaki/decodeutil/__init__.py -> build/lib.linux-x86_64-3.8/taiyaki/decodeutil
    creating build/lib.linux-x86_64-3.8/taiyaki/ctc
    copying taiyaki/ctc/__init__.py -> build/lib.linux-x86_64-3.8/taiyaki/ctc
    running build_ext
    building 'taiyaki.squiggle_match.squiggle_match' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/taiyaki
    creating build/temp.linux-x86_64-3.8/taiyaki/squiggle_match
    /home/phook2/miniconda3/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/phook2/miniconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/phook2/miniconda3/include -fPIC -I./taiyaki/squiggle_match -I/home/phook2/software/venv/lib/python3.8/site-packages/numpy/core/include -I/home/phook2/software/venv/include -I/home/phook2/miniconda3/include/python3.8 -c taiyaki/squiggle_match/squiggle_match.c -o build/temp.linux-x86_64-3.8/taiyaki/squiggle_match/squiggle_match.o -O3 -fopenmp -std=c99
    In file included from /home/phook2/software/venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1960,
                     from /home/phook2/software/venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                     from /home/phook2/software/venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:5,
                     from taiyaki/squiggle_match/squiggle_match.c:722:
    /home/phook2/software/venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
       17 | #warning "Using deprecated NumPy API, disable it with " \
          |  ^~~~~~~
    /home/phook2/miniconda3/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/phook2/miniconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/phook2/miniconda3/include -fPIC -I./taiyaki/squiggle_match -I/home/phook2/software/venv/lib/python3.8/site-packages/numpy/core/include -I/home/phook2/software/venv/include -I/home/phook2/miniconda3/include/python3.8 -c taiyaki/squiggle_match/c_squiggle_match.c -o build/temp.linux-x86_64-3.8/taiyaki/squiggle_match/c_squiggle_match.o -O3 -fopenmp -std=c99
    taiyaki/squiggle_match/c_squiggle_match.c: In function 'squiggle_match_viterbi':
    taiyaki/squiggle_match/c_squiggle_match.c:435:29: warning: comparison of integer expressions of different signedness: 'long unsigned int' and 'int32_t' {aka 'int'} [-Wsign-compare]
      435 |             if (nfstate - 1 != path[sample_max - 1]) {
          |                             ^~
    taiyaki/squiggle_match/c_squiggle_match.c:442:30: warning: comparison of integer expressions of different signedness: 'int32_t' {aka 'int'} and 'size_t' {aka 'const long unsigned int'} [-Wsign-compare]
      442 |             if (path[sample] >= nfstate) {
          |                              ^~
    x86_64-conda_cos6-linux-gnu-gcc -pthread -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/phook2/miniconda3/lib -L/home/phook2/miniconda3/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/phook2/miniconda3/lib -L/home/phook2/miniconda3/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/phook2/miniconda3/lib -Wl,-rpath-link,/home/phook2/miniconda3/lib -L/home/phook2/miniconda3/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/phook2/miniconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/phook2/miniconda3/include build/temp.linux-x86_64-3.8/taiyaki/squiggle_match/squiggle_match.o build/temp.linux-x86_64-3.8/taiyaki/squiggle_match/c_squiggle_match.o -o build/lib.linux-x86_64-3.8/taiyaki/squiggle_match/squiggle_match.cpython-38-x86_64-linux-gnu.so -fopenmp
    /home/phook2/miniconda3/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgomp
    collect2: error: ld returned 1 exit status
    error: command 'x86_64-conda_cos6-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/phook2/software/venv/bin/python3.8 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-rpkzkcow/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-rpkzkcow/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-k30tu_5i/install-record.txt --single-version-externally-managed --compile --install-headers /home/phook2/software/venv/include/site/python3.8/taiyaki Check the logs for full command output.

@marcus1487
Copy link
Collaborator

It looks as though the conda compiler has still found its way into this command. Note the /home/phook2/miniconda3/bin/x86_64-conda-linux-gnu-cc and /home/phook2/miniconda3/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld paths in the error text. I'm not an expert at conda, but my guess would be that this is the root cause of this issue.

@pwh124
Copy link
Author

pwh124 commented Mar 17, 2022

Hmmmm yep that makes sense. I will get off my base conda environment and attempt to install.

@pwh124
Copy link
Author

pwh124 commented Mar 18, 2022

For anyone else who runs into this issue... this worked to get taiyaki installing in a conda environment. The key is first installing pip in the environment so that it is used for other installs:

conda create --name taiyaki
conda activate taiyaki
conda install pip
pip install git+https://github.com/nanoporetech/taiyaki

@pwh124 pwh124 closed this as completed Mar 18, 2022
@pwh124 pwh124 reopened this Mar 18, 2022
@pwh124
Copy link
Author

pwh124 commented Mar 18, 2022 •
edited

Update: I found the best way to get taiyaki installed through a conda env

# Create a python3.8 env - this is needed for the dependencies. 3.10 wont work
conda create --name taiyaki python==3.8
conda activate taiyaki
# installing dependency in advance so the wheel will work
conda install numpy 
# installing dependency in advance so the wheel will work
conda install cython 
# use "python -m pip install" so the conda env is used
python -m pip install git+https://github.com/nanoporetech/taiyaki 
# /misc/ scripts are not initially installed. Download them to use alongside the installed taiyaki
git clone https://github.com/nanoporetech/taiyaki

In addition, I discovered that in installing taiyaki from git, the key script for preprocessing the data for remora did not seem to come along with the install. I've included the last line so that the misc dir with those scripts are also downloaded from taiyaki.

@marcus1487
Copy link
Collaborator

The primary use case for Remora is to supply the API calling interface for megalodon and bonito. Model training is an advanced use case and thus special installation requirements are required. I've updated the README to indicate the download of the taiyaki repo for the misc scripts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pwh124 @marcus1487