z-fastq 0.2.1

Faster gzip input, from Zipir 0.3.0's rebuilt DEFLATE decoder: count, stats, and check take 15 to 18% less CPU time on gzip files. process can now move a UMI from the start of the reads into their names, as fastp does. Output bytes, JSON documents, and exit statuses are unchanged, apart from the report fields the UMI options add and the fixes below.

Added

• UMIs. process --umi-length N --umi-mates 1|2|both moves an N-base unique molecular identifier from the start of mate 1, mate 2, or both into the names of both mates, in fastp's @NAME:UMI1_UMI2 form, and --umi-skip S removes the S spacer bases that follow it. Trimming, filters, insert sizes, and the retained statistics see the read after the spacer; the raw statistics and the pair duplication rate see it as read. A pair in which a UMI mate has no base left is dropped as too short. The written records are those fastp 1.4.0 writes with --umi --umi_loc per_read (or read1, read2), except that z-fastq places the UMI before a trailing /1 or /2 and before a tab, updates a header repeated on the plus line, and drops a pair that fastp keeps as a one-base read.
  • The z-fastq/process-v1 report gains the options umi_length, umi_skip, and umi_mates, and a umi object with the pairs whose UMI holds a base other than A, C, G, or T in either case and the short pairs; each is null without the options. Without them, outputs, QC tables, and the QC summary are unchanged.

Changed

• Speed. gzip input decodes through Zipir 0.3.0, whose rebuilt DEFLATE decoder takes 15 to 18% less CPU time for count, stats, and check on gzip input. process with gzip input and output takes about the same time as before, because compression and trimming dominate it. Output bytes, JSON documents, exit statuses, and peak memory are unchanged.

Fixed

• Outputs after a failed write. When a write to one deinterleave output fails, the records already buffered for the other output are written before the command exits with status 3, as process does; they were dropped. When the process report itself cannot be written, the incomplete file is removed, as the QC summary is, instead of being left as truncated JSON.
• QC table folders. A failed process run removes the QC table folders it created, so a rerun is not blocked by them; folders that existed before are kept.

z-fastq 0.2.0

A process command that validates, trims, and filters paired reads into two plain or gzip outputs with a JSON report, faster record handling in every command, AVX2 kernels selected at run time on CPUs that have them, bounded memory for stats and paired sampling on long reads, and stricter handling of standard input and output. Output bytes, JSON documents, and exit classes are unchanged except where a fix below says so.

Added

• Commands. process reads two mate files pair by pair and writes them to two new files.
  • It validates every pair as check --paired does and writes the accepted pairs, plain or gzip (--gzip, Zipir's fast preset by default, --gzip-preset even or dense for smaller files). --max-pairs N stops after the first N pairs. The same inputs and options give byte-identical outputs and report.
  • Trimming, each step off unless its options are given: --adapter1 with --adapter2 removes adapters and mate overlaps, --trim-poly-g N removes 3' poly-G runs, and --trim-tail-window W with --trim-tail-quality Q removes low-quality 3' ends. Trimming uses the same fixed memory at any read length.
  • Filtering, each filter off unless given: --low-quality Q with --max-low-quality-percent P, --max-n N, and --min-length N drop a pair when either mate fails. With any trim or filter option, a pair with a mate trimmed to nothing, or an adapter dimer, is dropped too. The written records are those fastp 1.1.0 writes with the matching options. Dropped pairs never change the exit status.
  • --report writes a z-fastq/process-v1 JSON document, and only with it or the two files below are statistics computed: per-mate statistics and FastQC's core quality modules, computed as FastQC 0.13.0 computes them, for the reads as read and as written; terminal poly-G and poly-A counts; FastQC's Adapter Content for its bundled adapter list, raw and retained; insert sizes from the mate overlap as fastp counts them, with the count of overlapping pairs, peak, and median; FastQC's Sequence Duplication Levels and Overrepresented sequences, raw and retained, and its Per tile sequence quality from Illumina read headers, with each sequence's source named from a FastQC contaminant list given with --contaminants; fastp's pair duplication rate with --pair-duplication; the reads and bases each trimming step removed; the dropped pairs and failed reads by reason; and any failure.
  • --qc-tables DIR and --qc-summary PATH write the same measurements in formats MultiQC reads: QC tables in FastQC's fastqc_data.txt layout per mate for the reads as read and, when trimming or filtering changes them, as written, each module with the status FastQC gives it at its default limits; and a JSON summary in fastp's report layout with the run's totals, filter counts, and per-cycle curves. Both are written only when the run succeeds. The wiki page References names the tools whose rules and formats they follow.
• Library.
  • FileSink.initStreaming writes at the file handle's current position, for standard output and pipes.
  • Stats.ratios returns each ratio of StatsResult as an exact numerator and denominator, so callers can round without floating point.
• Build.
  • Linux aarch64 is supported: -Dtarget=aarch64-linux builds it, and release builds are static for the requested architecture. It uses the portable kernels; the AVX2 object is x86-64 only.
  • -Dstrip sets whether debug information is stripped; it defaults to on for ReleaseFast, so a profiling build can keep its symbols.
  • zig build test-<suite> runs a single test binary, such as test-reader, or test-src for the in-file tests; zig build -l lists them.
• Releases. Each GitHub release carries static, stripped Linux binaries for x86-64 and aarch64 and the source package a Zig dependent fetches, with SHA-256 checksums and build provenance attestations. Both are tested before they are published: each binary is smoke-tested on every command after unpacking, on its own architecture, and the source package is fetched into a fresh package and tested there.

Changed

• Speed.
  • sample, interleave, and deinterleave take complete records straight from the read buffer and write output in whole pages, which roughly halves their time on plain input without raising peak memory beyond a few KiB.
  • Interleaved fraction sampling, exact-count sampling in every mode, and check --paired and --interleaved take records the same way, and the exact-count output pass skips unselected records with less work: 5 to 20% less time on plain input.
  • On CPUs with AVX2, builds whose target lacks AVX2 select 32-byte kernels at run time for newline search past the first 1 KiB of a line, validation of sequences of 1 KiB or more (about a third less time for check on PacBio reads, 15 to 20% for check and sample on ONT reads), validation of lines longer than the read buffer, and stats counting on reads of 1 KiB or more (about a quarter less time on long reads).
  • Fraction sampling validates the records it does not select without copying them: 5 to 25% less time for sample and single-file check on long reads, with no change in memory.
• Memory.
  • stats streams a record larger than the read buffer instead of copying it, so its peak memory no longer grows with read length (about 0.8 MiB on every input shape in the stats benchmark, from up to 1.8 MiB on 300 kb reads), long reads take 5 to 25% less time, and stats allocates nothing per record.
  • Paired and interleaved fraction sampling draw each pair before reading it and validate unselected pairs without keeping them: on 300 kb read pairs, peak memory for sample --paired falls from 2.3 MiB to 0.8 MiB when few pairs are selected. check --interleaved and deinterleave read a mate whose partner already failed only far enough to report it.
• Library. FileSink writes whole pages and keeps a partial last page buffered until the next write or flush.
• Build.
  • -Dkernel-backend=portable builds no AVX2 object and selects no kernel at run time; z-fastq's own kernels then use only the instructions of the build target, as Zipir's do.
  • Debug builds compile with LLVM, as release builds do. With Zig 0.16's own x86-64 code generator, a Debug build on a CPU with PCLMUL could corrupt state after the gzip CRC32 kernel, and an unrelated test then failed.

Fixed

• Standard input and output.
  • count, stats, and check reject standard output appended to one of their inputs (count a.fastq b.fastq >> b.fastq) with status 2 before reading anything, as sample and interleave already did; the input is no longer changed.
  • Standard input named twice, as - /dev/stdin or /dev/stdin /dev/stdin, is rejected when it is a pipe or FIFO instead of being read once and reported as zero records for the second name.
  • A pipe closed by its reader ends every command with status 3 and error: standard output: I/O error, as documented, instead of a SIGPIPE status of 141; sample, stats, count, and the JSON forms print that message on any standard-output failure.
• Inputs and outputs.
  • A directory given as an input is reported as input is a directory before anything is read, instead of I/O error; deinterleave then creates no output files.
  • deinterleave removes the empty --out1 it created when --out2 cannot be created, so a rerun is not blocked by it.
  • Exact-count sampling of a FIFO or directory exits with status 2, as for standard input, instead of 3.
• Options and limits.
  • A value option given twice (--count 1 --count 2) fails with status 2 and may appear only once, as --out1 and --out2 already did, instead of keeping the last value.
  • sample --interleaved and deinterleave accept every --max-line-bytes value the other commands accept, instead of exiting with status 4 before reading for values of 2^62 - 1 and above, sometimes with no message.
  • stats reports an input location past the supported limit as input location exceeds supported limit, as the other commands do.