pycoqc-ng 3.1.0

pip install --upgrade pycoqc-ng

A report for runs that have not been basecalled, and a long list of
fixes found by reading the code against real Albacore, Guppy and Dorado
output rather than against the test fixtures.

Added

• A report for runs that have not been basecalled. POD5, and a FAST5 holding
  signal only, carry no sequence, so pycoQC used to stop with an error.
  It now reports on the run itself: reads over time, time in pore, activity
  per channel and reads per barcode, with a signal summary table in place of
  the basecall one. There is deliberately no estimated base count, since
  bases can only be guessed from the signal length.
• Pass and fail come from the passes_filtering column when there is no
  quality score to threshold on.

Fixed

• --filter_calibration did nothing on any Guppy 2.2+ run, which names the
  column calibration_strand_genome rather than ..._genome_template.
  Recognising the name alone was not enough: the filter kept a list of the
  markers meaning "no control found", and Guppy's markers are not Albacore's,
  so the whole run would have been discarded as calibration strands.
• Barcodes written as bare numbers, as Deepbinner writes them, were typed as
  numbers and so split across chunks, became 6.0 after a merge, and could
  not be sorted against unclassified.
• The FASTQ and BAM converters counted elapsed time from the first read in
  the directory rather than from the start of each run, so two one hour runs
  recorded two days apart were reported as a single 50 hour run.
• Reads missing an optional column were discarded from the read count, the
  yield and the N50.
• Histogram bins: the shortest read fell outside the log bins, the axis ran a
  quarter of a decade past the longest read, the 1D density was drawn half a
  bin to the right, and the 2D density half a bin left and down.
• The yield on the over time plot disagreed with the summary table, because
  bases were estimated from a sample scaled by the ratio of read counts.
• The two mean coverage figures in one report measured different things.
• The barcode figure's buttons took two clicks and only updated the table;
  at 96 barcodes the legend, the pie and the table were unreadable or cut
  off, in the PDF as well.
• The converters stopped descending into a directory as soon as they found
  one matching file, so a fastq_pass holding unclassified.fastq beside
  one folder per barcode converted the unclassified reads and nothing else.
• --fields and --include_path were accepted and ignored by
  Bam_to_seq_summary, as was --include_path by Pod5_to_seq_summary;
  --basecall_id now says it has no meaning for those formats rather than
  being silently dropped. Fast5_to_seq_summary's command line default
  left out four run level fields its Python default includes.
• FASTQ records were collected into a dict keyed by read id, so repeated
  ids merged, but only when they fell in the same 1,000 read block.
• A bam or fastq directory holding no reads wrote an empty summary or
  raised KeyError.
• --config_file's help has always promised a lookup in ~/.pycoQC and
  ~/.config/pycoQC/config; nothing looked there.
• Pass and fail were split by dropping the passing reads' index labels,
  which takes every row carrying a repeated read id: a run basecalled
  twice reported no failing reads at all.
• Numeric barcodes came out as 6.0, and sorting their names put barcode
  10 between 1 and 2.
• A summary with a read length but no quality score, or the reverse, was
  reported as if nothing had been basecalled instead of saying which
  column was missing.
• Fast5_to_seq_summary left duration and passes_filtering out of its
  defaults, so a raw FAST5 report had no duration plots and nothing
  marked as failing.
• A zero duration made every log bin edge NaN; a blank passes_filtering
  counted as a failure; a barcode too rare to be sampled showed a read
  count beside a blank N50.
• Pass and fail are read only from the part of the path below the
  directory given to the converter, so a run kept under a directory named
  for a failed experiment is no longer a run of failing reads.
• The FASTQ and FAST5 converters sent their workers SIGTERM on the error
  path and never waited for them, so a child that outlived the signal was
  left for Python to join at interpreter exit: the conversion raised the
  error it meant to, and the process then hung on the way out with nothing
  to say why. Cleanup now waits, and SIGKILLs whatever ignored SIGTERM, on
  every exit path rather than for three hand-listed exception types.

Packaging

• The vendored stylesheet directory was not declared as a package, which
  setuptools warned about on every build. The files were shipped anyway,
  by the package-data globs, so no release was missing them.

Full changelog: https://github.com/duceppemo/pycoqc-ng/blob/v3.1.0/CHANGELOG.md

pycoqc-ng 3.0.0

First release of pycoqc-ng, a maintained continuation of pycoQC by Adrien Leger and Tommaso Leonardi, which is no longer developed.

pip install pycoqc-ng

What this adds over pycoQC 2.5.2

• Dorado / unaligned BAM input via Bam_to_seq_summary
• POD5 input via Pod5_to_seq_summary
• FASTQ input via Fastq_to_seq_summary, including %GC metrics and plots
• Per-barcode length and %GC boxplots, and a basecall summary by barcode
• Runs on current dependencies: numpy 2.x, pandas 2.x and 3.x, plotly 5-7, pysam 0.22+, Python 3.9-3.13
• No compiled extension — installs as a pure Python package

Notable fixes

The package could not previously be installed and used: two modules imported a Cython extension built at the repository root, which only worked when the current directory happened to be the repo.

• Fastq_to_seq_summary reused one dict for every read in a file and queued that same object repeatedly. Because mp.Queue pickles in a background thread, entries were overwritten before serialisation: a 200-read file produced 200 rows holding only 2 distinct reads.
• The time-binned plots dropped missing values from one column while reading times from another, pairing values with the wrong time bins.
• FASTQ chunk sizes were counted in lines rather than reads, so a size not divisible by four silently discarded records.
• max_fastq processed one file fewer than requested; N50 failed on read-only arrays; elapsed times broke on pandas 3; plotly 7 compatibility; python -m pycoQC did nothing.

Performance

• FASTQ parsing is 3.3x faster, with byte-identical output. Timestamps now use datetime.fromisoformat rather than dateutil (160-220x faster per read), and mean q-scores use a 256-bin count and one dot product.
• Removing Cython made quality scoring 1.4-14x faster and exact; the old approximate pow drifted up to 0.05 Phred.
• The FASTQ converter no longer nests process pools. The process count was roughly the square of the thread count — 202 processes at -t 16, now 17.

Project

73 tests and GitHub Actions CI across Python 3.9-3.13 and macOS, plus a pyflakes lint gate. See CHANGELOG.md for the full list.

Citing

Please cite the original authors:

Leger A. and Leonardi T. (2019). pycoQC, interactive quality control for Oxford Nanopore Sequencing. Journal of Open Source Software, 4(34), 1236. https://doi.org/10.21105/joss.01236