Thank you for doing this benchmark

[wraetz]

It would be even more interesting if you take the kind of accession ( flat-table vs. cSRA aka aligned ) into consideration.
fasterq-dump was written to speed up the fastq-conversion for cSRA files, there speed up is the highest.
flat tables are processed by fasterq-dump too, but the speed-up is not that big compared to fastq-dump.

You can check what kind it is by running 'vdb-dump SRRXXXXXX --info'
If it has only 1 table ( the SEQUENCE-table ) it is a flat table, aka unaligned data.
If it has a SEQUENCE, a REFERENCE and at least a PRIMARY_ALIGNMENT - table, it is a cSRA aka aligned data.
( cSRA stands for compressed against the reference )

My guess is that the outcome will be:

• for flat tables used parallel fastq-dump
• for cSRA use prefetch + fasterq-dump

However it would be interesting if you can prove that

[Midnighter]

Cheers, thank you for the input. With parallel-fastq-dump I was specifically talking about this Python wrapper. Please also see the ongoing dicussion in the corresponding issue.

Would you happen to have some relevant run accessions for me to test this? I'm not sure right now how I would find them.

[Midnighter]

@wraetz I still don't know how to find sequencing information stored either in a flat format or the cSRA format.

But I did run another comparison that only compares extraction after prefetch and compression.