No function of Docker container for DeepVariant

[windwang1]

Hi，
When I execute the commands nothing happens.

So, i tried unzip this *.zip, then i found this question.



I used this dockerfile https://github.com/google/deepvariant/tree/r0.4/deepvariant/docker on my cloud platform(Not GCP).
Is there any problem with my environment?

[scott7z]

If you could describe your environment in more detail (which OS, CPU), and the full sequence of operations you performed (copy and paste the text from the terminal), it will help us to diagnose the problem.

[windwang1]

Thank you for your reply @scott7z

CPU and OS infomation:
uname -a
Linux 6562232b4f47 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
head -n 1 /etc/issue
Ubuntu 16.04.1 LTS \n \l

First, when I ran a container for deepvariant and executed the make_examples command
./make_examples
Nothing happened
python make_examples.zip
Still nothing happened

unzip make_examples.zip
cd runfiles/genomics/deepvariant
#Add deepvariant to PYTHONPATH
echo "export PYTHONPATH=\$PYTHONPATH:/opt/deepvariant/bin/runfiles/genomics" >> /root/.bashrc source /root/.bashrc
python make_examples.py
Then I got
"Illegal instruction (core dumped)"

This are dockerfile, run-prereq.sh and setting.sh, for reference.
Dockerfile.zip

[scott7z]

OK, still trying to follow what is happening. Can you give us the full sequence of commands you typed to get the docker image and then run it?

You shouldn't have to unzip any files; python knows how to execute them.

[windwang1]

This directory here contains dockerfile and other files that need to be ADD to the image

docker build -f dockerfile -t deepvariant_1214 .
Successfully built b829b45a9401

This directory contains the deepvariant models and quickstart-testdata files downloaded from https://console.cloud.google.com/storage/browser/deepvariant

docker run --name deepvariant_test_1214 -v /***/bioinfo/wangwd/workflow/deepvariant/data:/data -it b829b45a9401 /bin/bash
Then I cd /opt/deepvariant/bin/

./make_examples \ --mode calling \ --ref /data/quickstart-testdata/ucsc.hg19.chr20.unittest.fasta \ --reads /data/quickstart-testdata/NA12878_S1.chr20.10_10p1mb.bam \ --regions "chr20:10,000,000-10,010,000" \ --examples /data/quickstart-output/examples.tfrecord.gz
When I finished executing this command, I did not get any result.

[arostamianfar]

Thanks for the updates. It looks like you are building a customized docker image. Before we can debug that, could you please confirm whether you can:

• Run quickstart on your machine (without using docker)? https://github.com/google/deepvariant/blob/r0.4/docs/deepvariant-quick-start.md
• Run DeepVariant using our provided docker image from gcr.io/deepvariant-docker/deepvariant:0.4.1? https://github.com/google/deepvariant/blob/r0.4/docs/deepvariant-docker.md

[scott7z]

Also, if you can show us the output of
cat /proc/cpuinfo
that will help. It's possible the pre-compiled binaries require a newer CPU than you are using,
which could explain the illegal instruction message.

[windwang1]

Cloud Platform CPU Information：

I installed a Ubuntu 16.04.3 virtual machine on my laptop and it works with or without docker @arostamianfar
Laptop CPU information：

[scott7z]

It looks to me like the first CPU (the cloud platform one) doesn't support AVX,
while the second (your laptop) does.

Our precompiled binarines require that from the CPU, so it looks like you'll need
to run on a different platform, or will need to compile everything from source
(DeepVariant, and Tensorflow, and probably CLIF,).

[scott7z]

I'm going to close this issue, since we seem to have gotten to the bottom of it.

[windwang1]

Thanks for your help @scott7z @arostamianfar
According to your suggestion I compiled DeepVariant, Tensorflow, and CLIF from the source.
But nothing changed, still can not get the result.
Then, I ran on another platform which support AVX2 and got the result.

[scott7z]

Did you edit settings.sh? Look where it says:

export DV_COPT_FLAGS="--copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-O3"

you have to remove the options your CPU doesn't support. Something like this, for a minimal case

export DV_COPT_FLAGS=""

or this for something tuned to your particular CPU.

export DV_COPT_FLAGS="--copt=-march=native"

The DV_COPT_FLAGS variable is used by our scripts to pass flags to the build system.
The --copt= construction is a flag for Bazel, the build system that passes flags to the compiler,
The "-march=native" part is a flag for GCC or Clang that says to try to auto-detect the architecture.
So read the GCC manual to see what other options are available.

[scott7z]

You'll probably need to do something similar when building Tensorflow, too.

[windwang1]

Thanks for your reminder @scott7z
I ignored what you mentioned.
But for now I do not plan to make any further changes.
I've converted to a platform that supports AVX2 and completed a 'quick-start' test.