Issue with running docker image

[Kwondo87]

Hello,

I'm trying to run Deepvariant using singularity. I just followed the "Notes on Singularity" section in quick start test (https://github.com/google/deepvariant/blob/r0.10/docs/deepvariant-quick-start.md), and I got an error regarding numpy as below.

Could you help me resolve this issue? I used deepvariant_1.6.0 image

2023-12-02 23:23:35.126320: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
I1202 23:23:41.449015 46912500266816 run_deepvariant.py:519] Re-using the directory for intermediate results in /flashscratch/kimkw/tmp/tmppin2lwy5

***** Intermediate results will be written to /flashscratch/kimkw/tmp/tmppin2lwy5 in docker. ****


***** Running the command:*****
time seq 0 0 | parallel -q --halt 2 --line-buffer /opt/deepvariant/bin/make_examples --mode calling --ref "./quickstart-testdata/ucsc.hg19.chr20.unittest.fasta" --reads "./quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam" --examples "/flashscratch/kimkw/tmp/tmppin2lwy5/make_examples.tfrecord@1.gz" --channels "insert_size" --gvcf "/flashscratch/kimkw/tmp/tmppin2lwy5/gvcf.tfrecord@1.gz" --regions "chr20:10,000,000-10,010,000" --task {}

I1202 23:23:46.123890 46912500266816 genomics_reader.py:222] Reading ./quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam with NativeSamReader
I1202 23:23:46.133658 46912500266816 make_examples_core.py:301] Preparing inputs
I1202 23:23:46.139615 46912500266816 genomics_reader.py:222] Reading ./quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam with NativeSamReader
I1202 23:23:46.140348 46912500266816 make_examples_core.py:301] Common contigs are ['chr20']
I1202 23:23:46.141555 46912500266816 make_examples_core.py:301] Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
I1202 23:23:46.150200 46912500266816 genomics_reader.py:222] Reading ./quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam with NativeSamReader
I1202 23:23:46.240882 46912500266816 genomics_reader.py:222] Reading ./quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam with NativeSamReader
I1202 23:23:46.241135 46912500266816 make_examples_core.py:301] Writing gvcf records to /flashscratch/kimkw/tmp/tmppin2lwy5/gvcf.tfrecord-00000-of-00001.gz
I1202 23:23:46.248160 46912500266816 make_examples_core.py:301] Writing examples to /flashscratch/kimkw/tmp/tmppin2lwy5/make_examples.tfrecord-00000-of-00001.gz
I1202 23:23:46.248263 46912500266816 make_examples_core.py:301] Overhead for preparing inputs: 0 seconds
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf . Check the section C-API incompatibility at the Troubleshooting ImportError section at https://numpy.org/devdocs/user/troubleshooting-importerror.html#c-api-incompatibility for indications on how to solve this problem .
Traceback (most recent call last):
  File "/flashscratch/kimkw/tmp/Bazel.runfiles_hubgarxp/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 234, in <module>
    app.run(main)
  File "/flashscratch/kimkw/tmp/Bazel.runfiles_hubgarxp/runfiles/absl_py/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/flashscratch/kimkw/tmp/Bazel.runfiles_hubgarxp/runfiles/absl_py/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "/flashscratch/kimkw/tmp/Bazel.runfiles_hubgarxp/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 224, in main
    make_examples_core.make_examples_runner(options)
  File "/flashscratch/kimkw/tmp/Bazel.runfiles_hubgarxp/runfiles/com_google_deepvariant/deepvariant/make_examples_core.py", line 2847, in make_examples_runner
    region_example_shape = region_processor.writes_examples_in_region(
  File "/flashscratch/kimkw/tmp/Bazel.runfiles_hubgarxp/runfiles/com_google_deepvariant/deepvariant/make_examples_core.py", line 1566, in writes_examples_in_region
    for example in self.create_pileup_examples(
  File "/flashscratch/kimkw/tmp/Bazel.runfiles_hubgarxp/runfiles/com_google_deepvariant/deepvariant/make_examples_core.py", line 2333, in create_pileup_examples
    pileup_images = self.pic.create_pileup_images(
  File "/flashscratch/kimkw/tmp/Bazel.runfiles_hubgarxp/runfiles/com_google_deepvariant/deepvariant/pileup_image.py", line 602, in create_pileup_images
    pileup = _pileup_for_pair_of_alts(alts)
  File "/flashscratch/kimkw/tmp/Bazel.runfiles_hubgarxp/runfiles/com_google_deepvariant/deepvariant/pileup_image.py", line 544, in _pileup_for_pair_of_alts
    ref_image = self.build_pileup(
  File "/flashscratch/kimkw/tmp/Bazel.runfiles_hubgarxp/runfiles/com_google_deepvariant/deepvariant/pileup_image.py", line 490, in build_pileup
    build_pileup_for_one_sample(reads_for_samples[i], sample)
  File "/flashscratch/kimkw/tmp/Bazel.runfiles_hubgarxp/runfiles/com_google_deepvariant/deepvariant/pileup_image.py", line 400, in build_pileup_for_one_sample
    self._encoder.encode_reference(refbases)
ImportError: numpy.core.multiarray failed to import
parallel: This job failed:
/opt/deepvariant/bin/make_examples --mode calling --ref ./quickstart-testdata/ucsc.hg19.chr20.unittest.fasta --reads ./quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam --examples /flashscratch/kimkw/tmp/tmppin2lwy5/make_examples.tfrecord@1.gz --channels insert_size --gvcf /flashscratch/kimkw/tmp/tmppin2lwy5/gvcf.tfrecord@1.gz --regions chr20:10,000,000-10,010,000 --task 0

real    0m6.183s
user    0m3.271s
sys     0m1.140s

[pichuan]

Hi @Kwondo87 ,
I think this might be similar to #640

If I remember correctly, the issue was that Singularity uses the Numpy from your computer (instead of the docker image), and in your setting it's incompatible.

Please look at the discussion in #640 and see if it's helpful for you.

[Kwondo87]

Thank you for the reply, @pichuan

I took a look at the discussion. It looks like the issue was not resolved for deepvariant 1.6.0. Also, the numpy version on my computer doesn't matter, as the numpy version inside the singularity image is being used as you've shown in discussion #640.

1.4.0 image worked on setting but I would like to use the latest version of singularity image if possible.

Thanks!

[pichuan]

@Kwondo87 I wasn't able to reproduce the Singularity issue on my side, so I'll need more information from you.

I also noticed that your original post said you used https://github.com/google/deepvariant/blob/r0.10/docs/deepvariant-quick-start.md . Can you use https://github.com/google/deepvariant/blob/r1.6/docs/deepvariant-quick-start.md instead in case there's differences? (Although, I just quickly checked and they don't look different on the Singularity section)

@Kwondo87 given that I'm not able to reproduce the error, can you give me your step-by-step, including how you get the image, how you run singularity, etc. What your OS is, etc. Maybe check the numpy versions in anywhere that might be relevant and just print them?

I'll be able to help more when I can reproduce on my side. (Something similar to what I did in #745 (comment) will be really helpful. Although, in that case I also wasn't able to reproduce the issue)

[pichuan]

Btw, I have confirmed on a CPU machine , I was able to run the Quick Start with Singularity without any issues.

pichuan@pichuan-cpu:~$ singularity --version
singularity version 3.7.0
pichuan@pichuan-cpu:~$ uname -a
Linux pichuan-cpu 5.15.0-1047-gcp #55~20.04.1-Ubuntu SMP Wed Nov 15 11:38:25 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

And a few different version checks:

pichuan@pichuan-cpu:~$ singularity exec deepvariant_1.6.0.sif pip freeze | grep numpy
numpy==1.24.4
pichuan@pichuan-cpu:~$ singularity shell deepvariant_1.6.0.sif
Singularity> python -c "import numpy; print(numpy.__version__)"
1.24.4
Singularity> python3 -c "import numpy; print(numpy.__version__)"
1.24.4
pichuan@pichuan-cpu:~$ singularity exec --bind "${INPUT_DIR}":"/input","${OUTPUT_DIR}":"/output",/usr/lib/locale/:/usr/lib/locale/ deepvariant_1.6.0.sif python -c "import numpy; print(numpy.__version__)"
1.24.4

[Kwondo87]

Thanks for the quick reply, @pichuan

First of all, I followed the instructions in https://github.com/google/deepvariant/blob/r1.6/docs/deepvariant-quick-start.md, but it gives the same error.

1. I got the image from "docker://google/deepvariant:1.6.0" I tried both: locally download the image and use the image without downloading. But they gave the same error.
2. Follow the instructions in the link above to run the program.

Here is the script that I used

#!/bin/bash

BIN_VERSION="1.6.0"

INPUT_DIR="${PWD}/quickstart-testdata"
DATA_HTTP_DIR="https://storage.googleapis.com/deepvariant/quickstart-testdata"

mkdir -p ${INPUT_DIR}
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/NA12878_S1.chr20.10_10p1mb.bam
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/NA12878_S1.chr20.10_10p1mb.bam.bai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.bed
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.vcf.gz
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.vcf.gz.tbi
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.fai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz.fai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz.gzi

OUTPUT_DIR="${PWD}/quickstart-output"
mkdir -p "${OUTPUT_DIR}"

# Pull the image.
singularity pull docker://google/deepvariant:"${BIN_VERSION}"

# Run DeepVariant.
singularity run -B /usr/lib/locale/:/usr/lib/locale/ \
  docker://google/deepvariant:"${BIN_VERSION}" \
  /opt/deepvariant/bin/run_deepvariant \
  --model_type=WGS \
  --ref="${INPUT_DIR}"/ucsc.hg19.chr20.unittest.fasta \
  --reads="${INPUT_DIR}"/NA12878_S1.chr20.10_10p1mb.bam \
  --regions "chr20:10,000,000-10,010,000" \
  --output_vcf="${OUTPUT_DIR}"/output.vcf.gz \
  --output_gvcf="${OUTPUT_DIR}"/output.g.vcf.gz \
  --intermediate_results_dir "${OUTPUT_DIR}/intermediate_results_dir" \
  --num_shards=1

And I did same things as you to check the versions.

singularity --version
singularity version 3.8.5-2.el7

uname -a
Linux sumner098 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 30 14:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

singularity exec deepvariant_1.6.0.sif pip freeze | grep numpy
numpy==1.22.1

singularity shell -B /usr/lib/locale/:/usr/lib/locale/ docker://google/deepvariant:1.6.0
Singularity> python -c "import numpy; print(numpy.__version__)"
1.22.1
Singularity> python3 -c "import numpy; print(numpy.__version__)"
1.22.1

singularity exec --bind ${PWD}/quickstart-testdata/,${PWD}/quickstart-output/,/usr/lib/locale/:/usr/lib/locale/ deepvariant_1.6.0.sif python -c "import numpy; print(numpy.__version__)"
1.22.1

Thanks a lot!

[pichuan]

Hi @Kwondo87 ,
Can you try updating your numpy to 1.24.4 and see if it changes anything?

[Kwondo87]

Hi, @pichuan

I upgraded numpy from 1.22.1 to 1.24.4, and it worked!!
But I'm a bit confused as I upgraded the numpy in one of the containers using "pip install numpy --upgrade", and it worked in the other container. Does the container image use numpy from outside of the container?

Thanks a million anyway!

Singularity deepvariant_1.6.0.sif:> pip freeze | grep numpy
numpy==1.24.4

[pichuan]

Glad to hear that it worked.

I might have mentioned this before (maybe in the previous thread) - I'm not super familiar with Singularity myself, so I won't be able to help more on that. Given that your use case works, I'll close this issue!