Error in dnaio

[sprakashUTH]

When running cutadapt 3.1 to process FASTQ files, I receive the following fatal error:
File "/usr/local/lib/python3.7/site-packages/cutadapt/pipeline.py", line 571, in run
dnaio.read_paired_chunks(f, f2, self.buffer_size)):
File "/usr/local/lib/python3.7/site-packages/dnaio/chunks.py", line 111, in read_paired_chunks
raise ValueError("FASTQ record does not fit into buffer")
ValueError: FASTQ record does not fit into buffer

The error occurs a little over half the time. Other FASTQ files process normally. Have you seen a similar issue? How can I prevent this?

[marcelm]

Can you please say what is in those FASTQ files? In particular: Are these long reads and if so, do you know how long they are?

The buffer size is 4 million by default, so a read of length ~2 million should fit into that buffer (nucleotides plus qualities), assuming that the length of the read name is negligible. I would like to know whether I have to increase that default.

To prevent this, you can increase the buffer size by using the hidden cutadapt option --buffer-size. Just add something like --buffer-size=10000000 to your command-line options.

[marcelm]

Since we cannot reproduce this and there hasn’t been any feedback, I’ll close this now.