run issue

[MartinPippel]

I have some issues to run deepconsensus on our Centos 7 linux cluster. Any advice would be highly appreciated.

python3.6 -m deepconsensus.scripts.run_deepconsensus --input_subreads_aligned=aligned.pbmm2_to_ccs.bam --input_subreads_unaligned=m64046_210915_224507.subreads.bam --input_ccs_fasta=m64046_210915_224507.ccs.fasta --output_directory=out --checkpoint=${CHECKPOINT_PATH}
Traceback (most recent call last):
  File "/projects/dazzler/pippel/prog/pkgs/deepconsensus/lib/python3.6/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/projects/dazzler/pippel/prog/pkgs/deepconsensus/lib/python3.6/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/projects/dazzler/pippel/prog/pkgs/deepconsensus/lib/python3.6/site-packages/deepconsensus/__init__.py", line 35, in <module>
    from nucleus.io import bed
  File "/projects/dazzler/pippel/prog/pkgs/deepconsensus/lib/python3.6/site-packages/nucleus/io/bed.py", line 62, in <module>
    from nucleus.io import genomics_reader
  File "/projects/dazzler/pippel/prog/pkgs/deepconsensus/lib/python3.6/site-packages/nucleus/io/genomics_reader.py", line 68, in <module>
    from nucleus.io.python import tfrecord_reader
ImportError: /lib64/libm.so.6: version 'GLIBC_2.23' not found (required by /projects/dazzler/pippel/prog/pkgs/deepconsensus/lib/python3.6/site-packages/nucleus/io/python/tfrecord_reader.so)

I also got the following two errors during the installation:

  ERROR: Command errored out with exit status 1:
   command: /projects/dazzler/pippel/prog/pkgs/deepconsensus/bin/python3.6 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-pwwbu26e/google-nucleus_2aee0a6f3e0749a692932a2d8a48e902/setup.py'"'"'; __file__='"'"'/tmp/pip-install-pwwbu26e/google-nucleus_2aee0a6f3e0749a692932a2d8a48e902/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open
)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-s20xid5d
       cwd: /tmp/pip-install-pwwbu26e/google-nucleus_2aee0a6f3e0749a692932a2d8a48e902/
  Complete output (3 lines):
  /projects/dazzler/pippel/prog/pkgs/deepconsensus/lib/python3.6/site-packages/setuptools/version.py:1: UserWarning: Module google was already imported from None, but /tmp/pip-install-pwwbu26e/google-nucleus_2aee0a6f3e0749a692932a2d8a48e902 is being added to sys.path
    import pkg_resources
  This package does not support wheel creation.
  ----------------------------------------
  ERROR: Failed building wheel for google-nucleus
  Running setup.py clean for google-nucleus
  Building wheel for avro-python3 (setup.py) ... done
  Created wheel for avro-python3: filename=avro_python3-1.9.2.1-py3-none-any.whl size=43512 sha256=ca414c03c5335f98f1cff074ad5f8399c4ad5781fa3d6aac92b2107fd5e138cb
  Stored in directory: /lustre/projects/dazzler/pippel/prog/pkgs/deepconsensus/cache/pip/wheels/4e/08/0c/727bff8f20fedbdeb8a2c5214e460b214d41c10dc879cf6dac
  Building wheel for crcmod (setup.py) ... done
  Created wheel for crcmod: filename=crcmod-1.7-cp36-cp36m-linux_x86_64.whl size=36011 sha256=8bf271e9143d8486dfc10aad8fd01ff27c61328616118bf24fb4dd872f3016dd
  Stored in directory: /lustre/projects/dazzler/pippel/prog/pkgs/deepconsensus/cache/pip/wheels/ac/bb/07/adfb4ffd0aaace2022ea25c082a7cdc688b10d30e86d6d2fde
  Building wheel for kaggle (setup.py) ... done
  Created wheel for kaggle: filename=kaggle-1.5.12-py3-none-any.whl size=73051 sha256=aab46bf6b1503207873dc81c1e09f5e1febf195e534636f4d833c2be1004384f
  Stored in directory: /lustre/projects/dazzler/pippel/prog/pkgs/deepconsensus/cache/pip/wheels/77/47/e4/44a4ba1b7dfd53faaa35f59f1175e123b213ff401a8a56876b
  Building wheel for oauth2client (setup.py) ... done
  Created wheel for oauth2client: filename=oauth2client-3.0.0-py3-none-any.whl size=106375 sha256=006538ef14effae42f9742b82416925bd713eaa35098036891603963ae88a86e
  Stored in directory: /lustre/projects/dazzler/pippel/prog/pkgs/deepconsensus/cache/pip/wheels/85/84/41/0db9b5f02fab88d266e64a52c5a468a3a70f6d331e75ec0e49
  Building wheel for py-cpuinfo (setup.py) ... done
  Created wheel for py-cpuinfo: filename=py_cpuinfo-8.0.0-py3-none-any.whl size=22258 sha256=c2de11c9b239c87ac3f1fd52183e4ec7a541cc0d84108408ce2723d2de89c6b2
  Stored in directory: /lustre/projects/dazzler/pippel/prog/pkgs/deepconsensus/cache/pip/wheels/3e/e1/d9/9b782b170e5272d6500cee4d29dd6c724598b22dc399d81d01
Successfully built avro-python3 crcmod kaggle oauth2client py-cpuinfo
Failed to build google-nucleus
Installing collected packages: tensorflow-model-optimization, tensorflow-hub, tensorflow-datasets, tensorflow-addons, tensorflow, sentencepiece, scipy, pyyaml, pymongo, pydot, pyarrow, py-cpuinfo, psutil, pandas, opencv-python-headless, oauth2client, mock, matplotlib, kaggle, ipython, hdfs, google-cloud-bigquery, google-api-python-client, gin-config, fastavro, Cython, crcmo
d, contextlib2, avro-python3, tf-models-official, ml-collections, google-nucleus, apache-beam, deepconsensus
    Running setup.py install for google-nucleus ... done
  DEPRECATION: google-nucleus was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
crossmap 0.4.2 requires bx-python, which is not installed.
crossmap 0.4.2 requires pyBigWig, which is not installed.

Thanks,
Martin

[pichuan]

Thank you for reporting this.
To be honest we have not tested on CentOS 7. I just tested and also wasn't able to get it to work, but the error I'm getting is different:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib64/python3.6/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/home/pichuan/.local/lib/python3.6/site-packages/deepconsensus/__init__.py", line 35, in <module>
    from nucleus.io import bed
  File "/home/pichuan/.local/lib/python3.6/site-packages/nucleus/io/bed.py", line 62, in <module>
    from nucleus.io import genomics_reader
  File "/home/pichuan/.local/lib/python3.6/site-packages/nucleus/io/genomics_reader.py", line 68, in <module>
    from nucleus.io.python import tfrecord_reader
ImportError: libbz2.so.1.0: cannot open shared object file: No such file or directory

I also just try testing nucleus 0.5.8, which I suspect also didn't work on CentOS7.

We'll have to think about what to do with this. I'll keep this open for now.

[MartinPippel]

Thanks for your reply. I also got your problem. And after installing bzip2 in my conda environment this error was gone.

[pichuan]

Hi @MartinPippel ,
I can confirm that I was able to get around the libbz2 issue, by doing something a bit hacky like:

sudo ln -sf /usr/lib64/libbz2.so.1.0.6 /usr/lib64/libbz2.so.1.0

After that, I'm also seeing: "`GLIBC_2.23' not found".

Actually, this reminded me that this is probably part of Nucleus - I tested Nucleus 0.5.8 (which DeepConsensus builds on), and we have the same issue. CentOS7 has an older GLIBC, and that's why we're having the issue.

I think it is possible for us to build Nucleus and DeepConsensus in a way that is manylinux2010 compliant, and hopefully resolve this issue. I will add this to our list of tasks, but I don't know for sure when we can prioritize this.

Another approach we could try to take is to use Docker (more like what we did for DeepVariant) instead of pip wheels.

Either approach will take some time for us. @MartinPippel , given that I've filed an internal issue to track this, I'll close this for now. But please feel free to add more here.

[maximilianpress]

I am experiencing the same issue on CentOS7, Python 3.6.9, conda etc. All of the notable details appear to be the same (including bzip2 sub-issue).

[hzhang-2021]

I think this is because the google-nucleus binary by pip was built under /lib64/libm.so.6: version 'GLIBC_2.23'.
the default centos7 only have /lib64/libm.so.6: version 'GLIBC_2.17'.
you have to recompile all that package from source.... that is very difficult job.