compare with needletail and kseq.h

[jianshu93]

Hello Rayan,

It would be interesting to compare with biofast: https://github.com/lh3/biofast

It seems for the test dataset: M_abscessus_HiSeq.fq, needtail is still faster than even paralleled version of seq_io. This repo takes about 3 seconds with 12 threads while the needtail takes 0.8 seconds for plain fastq( M_abscessus_HiSeq.fq) file.

Thanks,

Jianshu

[rchikhi]

dear Jianshu, very interesting, thanks much for the pointer!

I was under the impression that the seq_io benchmark was exhaustive in terms of rust implementations, but it's not. That said, M_abscessus_HiSeq.fq is too small a workload. seq_io claims to parse at 2-4 GB/sec so it should also be close to half a second on that file. A larger file would be needed to mitigate overheads.

[rchikhi]

Got around to doing some benchmarks: natir/in_place_fastx#1 (comment)

[jianshu93]

it seems very promising. also the in_place one.

I will also do an experiment on that one. I suggest using metagenomes，like an environmental sample，can be 100G+ and very diverse.

Thanks，

Jianshu

[jianshu93]

I tried your in place fastx commit (main.rs) with the error (inside the in place fastx clone from you fork):

cargo run --release -- ~/Github/test_data/T4AerOil_sbsmpl5.fa
Finished release [optimized + debuginfo] target(s) in 0.06s
Running target/release/in_place_fastx /Users/jianshuzhao/Github/test_data/T4AerOil_sbsmpl5.fa
Error: NotAFastaFile

I can confirm that it is fasta file:

Thanks,

Jianshu