fasterq-dump segfaults immediately

[chrisamiller]

I downloaded some data with prefetch and the status messages show that everything was pulled down successfully. All of the reference files, the sra file, and a vdbcache file are all present and seem of reasonable size:

$ ls -lh | tail
-rw-------    1 c.a.mill domain u   31.2K Jun 18 19:58 GL000243.1
-rw-------    1 c.a.mill domain u   30.4K Jun 18 19:58 GL000244.1
-rw-------    1 c.a.mill domain u   29.6K Jun 18 19:58 GL000245.1
-rw-------    1 c.a.mill domain u   30.0K Jun 18 19:58 GL000246.1
-rw-------    1 c.a.mill domain u   29.5K Jun 18 19:58 GL000247.1
-rw-------    1 c.a.mill domain u   30.4K Jun 18 19:58 GL000248.1
-rw-------    1 c.a.mill domain u   30.0K Jun 18 19:58 GL000249.1
-rw-------    1 c.a.mill domain u   25.9K Jun 18 19:58 J01415.2
-rw-------    1 c.a.mill domain u   38.4G Jun 18 19:56 SRR1168069_dbGaP-29001.sra
-rw-------    1 c.a.mill domain u  395.5M Jun 18 19:56 SRR1168069_dbGaP-29001.sra.vdbcache

When I try to dump fastqs, though, I get an immediate segfault:

$ /usr/local/ncbi/sra-tools/bin/fasterq-dump --ngc ../prj_29001.ngc SRR1168069_dbGaP-29001.sra
2021-06-21T19:06:23 fasterq-dump.2.11.0 fatal: SIGNAL - Segmentation fault
fasterq-dump quit with error code 1

This is true whether I run it in ncbi/sra-tools:latest or use an older version like ncbi/sra-tools:2.10.8. To add to this, I was able to successfully convert one sra file (SRR1168053) to fastq using this command and the latest version, so I suspect a problem with the data

Suggestions for how to proceed or debug would be welcomed - thanks!

[chrisamiller]

Update: Of the 5 files I tried this on, only one was successful. The one above (SRR1168069) failed with one error last time, and different error in subsequent tries with an older docker image. The other one I mentioned in that comment (SRR1168053) succeeded.

using docker image ncbi/sra-tools:2.10.9:

for i in SRR*;do echo "--- $i ---";date;cd $i && rm -rf fasterq.tmp.*;/usr/local/ncbi/sra-tools/bin/fasterq-dump -
-ngc ../prj_29001.ngc *.sra;cd ../;done

--- SRR1168056 ---
Mon Jun 21 19:17:32 UTC 2021
2021-06-21T19:18:07 fasterq-dump.2.10.9 err: lookup_reader.c lookup_reader_get() to_read == 0 at 15068160
2021-06-21T19:51:23 fasterq-dump.2.10.9 err: merge_sorter.c wait_for_and_release_background_file_merger() 1169817041 of 1173310823
2021-06-21T19:51:23 fasterq-dump.2.10.9 err: fasterq-dump.c produce_lookup_files() -> RC(rcVDB,rcNoTarg,rcConstructing,rcSize,rcInvalid)
fasterq-dump quit with error code 3

--- SRR1168069 ---
Mon Jun 21 19:51:23 UTC 2021
2021-06-21T19:51:59 fasterq-dump.2.10.9 err: lookup_reader.c lookup_reader_get() to_read == 0 at 46490340
2021-06-21T20:18:23 fasterq-dump.2.10.9 err: merge_sorter.c wait_for_and_release_background_file_merger() 900579106 of 903549185
2021-06-21T20:18:23 fasterq-dump.2.10.9 err: fasterq-dump.c produce_lookup_files() -> RC(rcVDB,rcNoTarg,rcConstructing,rcSize,rcInvalid)
fasterq-dump quit with error code 3

--- SRR1168092 ---
Mon Jun 21 20:18:24 UTC 2021
2021-06-21T20:18:25 fasterq-dump.2.10.9 fatal: SIGNAL - Segmentation fault
fasterq-dump quit with error code 1

--- SRR1168102 ---
Mon Jun 21 20:18:25 UTC 2021
2021-06-21T20:19:06 fasterq-dump.2.10.9 err: lookup_reader.c lookup_reader_get() to_read == 0 at 42756300
2021-06-21T20:45:56 fasterq-dump.2.10.9 err: merge_sorter.c wait_for_and_release_background_file_merger() 1009027209 of 1012059522
2021-06-21T20:45:56 fasterq-dump.2.10.9 err: fasterq-dump.c produce_lookup_files() -> RC(rcVDB,rcNoTarg,rcConstructing,rcSize,rcInvalid)
fasterq-dump quit with error code 3

using docker image ncbi/sra-tools:latest:

$ for i in SRR*;do echo "--- $i ---";date;cd $i && rm -rf fasterq.tmp.*;/usr/local/ncbi/sra-tools/bin/fasterq-dump -
-ngc ../prj_29001.ngc *.sra;cd /scratch1/fs1/timley/dbgap;done
--- SRR1168056 ---
Mon Jun 21 21:22:59 UTC 2021
2021-06-21T21:23:00 fasterq-dump.2.11.0 fatal: SIGNAL - Segmentation fault
fasterq-dump quit with error code 1
--- SRR1168069 ---
Mon Jun 21 21:23:00 UTC 2021
2021-06-21T21:23:00 fasterq-dump.2.11.0 fatal: SIGNAL - Segmentation fault
fasterq-dump quit with error code 1
--- SRR1168092 ---
Mon Jun 21 21:23:00 UTC 2021
2021-06-21T21:23:01 fasterq-dump.2.11.0 fatal: SIGNAL - Segmentation fault
fasterq-dump quit with error code 1
--- SRR1168102 ---
Mon Jun 21 21:23:01 UTC 2021
2021-06-21T21:23:01 fasterq-dump.2.11.0 fatal: SIGNAL - Segmentation fault
fasterq-dump quit with error code 1

[chrisamiller]

Just did a new prefetch of SRR1168069 and can verify that the md5s match up with the data I was testing on, and that the same error occurs. The problem does not appear to be the result of a corrupted download.

[chrisamiller]

By falling back to 2.10.9, and giving it more cores (4) and more memory (16g), I was able to push these through. using a command like:
/usr/local/ncbi/sra-tools/bin/fasterq-dump -e 4 -p -x --ngc ../prj_29001.ngc SRR1168102_dbGaP-29001.sra

Even then, I've experienced issues where this command would segfault with

lookup :|  0.00%2021-06-28T20:23:48 fasterq-dump.2.10.9 fatal: SIGNAL - Segmentation fault
fasterq-dump quit with error code 1

And then I could clear the temp directories, run the same command again, on the same machine, in the same directory, and it would succeed. Happy to provide more info if it's helpful for debugging, or for adding more informative error messages.

[klymenko]

Did you try the latest toolkit?
Now it's 3.0.0.

[chrisamiller]

Using 3.0.0 outside of docker seems to work fine, but I encounter the same issues from inside the official docker container. At this point, I think it might be some kind of strange environment issue in our HPC cluster's docker implementation that I can't nail down. I suppose this can be resolved, though I may come back and revisit it some day when I have more free time. Thanks

[durbrow]

It segfaults also inside docker?

I would suggest the following:

1. Attach a directory to the container and set it as the working directory.
   docker run ... -v <local dir>:/output:rw -w /output

2. Attach another directory to the container and tell fasterq-dump to use it for temp space. Preferably, this directory will be on a faster volume than the output directory from 1.
   docker run ... -v <fast disk>:/scratch:rw ... fasterq-dump ... --temp /scratch