Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix fastp hanging forever on invalid input files #469

Merged
merged 1 commit into from
Mar 22, 2023

Conversation

wdu
Copy link
Contributor

@wdu wdu commented Mar 14, 2023

root cause: the processor does no longer process reads because one of the input files is at EOF while the other still contains reads. When the difference in reads is sufficiently large, the mechanism that pauses a reader when the processor is too far behind kicks in and blocks the reader forever since the processor will no longer process reads.

This will only happen on invalid input. This patch causes fastp to exit with an error when the input is invalid.

  • generate an error if zipped input files are truncated
  • skipping individual reads with errors causes problems for PE since the reads composing a pair wouldn´t match anymore. Instead, generate an error for obviously incorrectly formatted fastq files.
  • generate an error when paired files don´t contain same amount of reads

This fixes issues like #455, possibly #448, #425, #410, #400, #378, #340 (partial), #333, ...

root cause: the processor does no longer process reads because one
of the input files is at EOF while the other still contains reads.
When the difference in reads is sufficiently large, the mechanism
that pauses a reader when the processor is too far behind kicks
in and blocks the reader forever since the processor will no longer
process reads.

This will only happen on invalid input. This patch causes fastp to
exit with an error when the input is invalid.

* generate an error if zipped input files are truncated
* skipping individual reads with errors causes problems for PE
  since the reads composing a pair wouldn´t match anymore. Instead,
  generate an error for obviously incorrectly formatted fastq files.
* generate an error when paired files don´t contain same amount of reads
@sfchen sfchen merged commit d997858 into OpenGene:master Mar 22, 2023
@jolespin
Copy link

Was this fixed here? d997858

I use FastP in the backend of my github.com/jolespin/veba software suite and I've gotten complaints by users that FastP hangs when the input files are a little weird.

Can it just throw a fatal error instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants