Repository navigation
Releases
Releases · erdikilic/whittle
1.0.0
whittle 1.0.0 is the version described in the manuscript and the first release since 0.2.0. The full list of changes is in CHANGELOG.md.
New
- Tagged FASTQ input: SAM tags in FASTQ headers are read, updated with the trimmed sequence and written back.
--tag-filterselects reads by tag values, for example[rq]>=0.99 && [np]>=3.--rejected-outputwrites the reads and segments that did not pass, each tagged with the reason.--max-expected-errors,--quality-trim segmentsand--quality-end-cutofffor quality filtering.- Adapter discovery works outward from each read end and also finds barcodes, primers and UMIs next to adapters.
Changed
- Primers and barcodes split chimeric reads, as adapters do.
.fastq.gzoutput is BGZF-compressed and remains standard gzip.
Renamed or removed options
--update-signal-tagsis now--update-moves.--remove-kineticsis now part of--remove-tag.--discover-adaptersis replaced by--adapter-discover.--trim-barcodesis removed; barcode positions recorded by dorado are used whenever adapters are trimmed.
Fixed
- Trimmed and split reads no longer keep fibertools footprint arrays, whose positions referred to the untrimmed read.
- Discovery handles amplicon, cDNA and barcoded libraries more reliably.
0.2.0
whittle 0.2.0, the first release since 0.1.1.
Added
--remove-tag <TAG>and--strip-kinetics: remove auxiliary tags from every
output record.--remove-tagnames one two-character tag and is repeatable;
--strip-kineticsremoves the nine per-base kinetics and alignment-count
arrays (ip,pw,fi,fp,ri,rp,sa,sm,sx) in one flag, and
both fill the same removal set. Removal runs after the rewrite of the tags
whittle keeps in register, so removingMMstill leaves a rebuiltML/MN
and removing one per-base array still slices the others. It applies on every
BAM output path, including the untrimmed fast path, whose records are rebuilt
rather than passed through when tags are removed, and to the tags carried into
a BAM-to-FASTQ header. The resolved set is reported under
params.remove_tags. BAM input only.--trim-barcodes: removes the barcode spans dorado recorded in thebiaux
tag. The positions are read rather than the sequences searched for, so the cut
is the onedorado demuxwould have made, and it runs through the same
trimming machinery as every other stage, keepingMM/ML/MN, per-base
kinetics and the ONT move table in register. It is the outermost stage, so
--head-cropcounts from the first base after the front barcode. A barcode dorado
did not find leaves that end of the read alone; abithat does not describe a
window inside the read leaves it untrimmed and is counted under
warnings.barcode_tag_malformed_reads. BAM input only.--progress {auto,bar,plain,none}selects how progress is reported,
independently of the log level.--progress nonekeeps the banner and the run
summary while reporting nothing in flight, which suits a pipeline log;
--quietdrops the summary as well and outranks it.- Adapter and primer sequences may use the full IUPAC alphabet. A degenerate
primer now matches every variant its wobble positions cover, instead of being
skipped as "non-ACGT".Ufolds toT; non-nucleotide characters are still
skipped with a warning, and a pattern averaging two or more bases per position
is searched but flagged as near-wildcard. An ambiguity code in a read is
treated as a mismatch instead, so it costs error budget rather than matching
for free: a strayNin a real adapter still matches, a run of them is not
excised as one. --summary-json <PATH>: writes a machine-readable JSON summary of the run,
covering the resolved settings (params) and the read, base, and per-reason
segment-drop counters. Written on every dispatch path, folder merges included,
and regardless of--quietor the log level.schema_versionis bumped only
when an existing field changes meaning or disappears.- A
whittle.1man page, checked intoman/and shipped in every release
tarball. It is rendered from the live CLI definition by
cargo run --example gen-man;clap_mangenis a dev-dependency, so the
shipped binary is unchanged. --orderedwrites records in input order when running with more than one
thread, so output is byte-identical between runs and to a single-threaded run.
Without it records are written as they finish, which is faster and uses less
memory; a BAM written that way carriesSO:unsortedin its header.
Changed
- The progress bar shows the output count beside the input count, so a filter
discarding everything is visible while the run is going rather than only in
the summary. The percentage is right-aligned so the bar no longer shifts
sideways as it passes 9% and 99%, and the first frame carries the same fields
as every later one. Still ASCII, so it renders the same over SSH,screenand
a non-UTF-8 console. -vvreports the per-read decisions: which adapter matched where and at what
cost, the resulting action, and why each segment was kept or dropped, each
line attributed to its read. There were previously no trace-level events at
all, so-vvwas indistinguishable from-v.-vgains the resolved thread
budget and the run counters. Log events carry structured fields rather than
preformatted prose. Measured at no cost to throughput at the default level.- BAM to FASTQ no longer re-parses and re-serializes
MM/MLfor a record whose
window is not being trimmed. Over the full window the reconstruction is the
identity, so the source bytes are reused after an allocation-freeMLlength
check. About 29% less CPU on an untrimmed conversion, with byte-identical
output. --summary-jsonreports both adapter counts:params.adapters.configuredis
the set asked for,params.adapters.countthe set trimmed against after
presence detection or inference. The startup banner prints the former, so the
two figures no longer look like a contradiction.- Release tarballs now contain a versioned directory holding the binary,
man/,
and the README, CHANGELOG, and LICENSE, instead of a barewhittle
executable. - Documentation split: the README keeps the overview, install, and quick start;
the full flag reference moved todocs/cli.md, the tag machinery to
docs/tags.md, adapter trimming todocs/adapters.md, and the contributor
policy toCONTRIBUTING.md. - Comments, doc comments, and user-facing strings standardized to American
English prose without em or en dashes. noodles-bam,noodles-sam, andnoodles-bgzfupdated to 0.95, 0.90, and
0.51;clap,anyhow,thiserror,crossbeam-channel,gzp,bstr,
aho-corasick, andjiffto their latest patch releases.- The gzip decoder is zlib-rs rather than zlib-ng, which reads compressed FASTQ
about 7 percent faster and drops a C build dependency from the decode path.
Fixed
--summary-jsonwrote itscommandfield with the banner'sCommand:label
glued to the front of the value, so a consumer re-running the recorded
invocation had to strip it first. The label now belongs to the banner line and
the JSON carries the bare, shell-quoted argv it always documented.thread_budgetallocated zero render workers at exactly-t 3with parallel
decode and uncompressed output, so the banner reportedtrim 0and the
workflow silently fell back to its own default. Every other thread count is
unchanged; a property test now checks that no stage is ever allocated zero.- Every base-modification record missed the untrimmed fast path, because the
MNconsistency check accepted only thei(Int32) subtype while SAM writes
integers at the smallest width that fits, so dorado emitsMN:S. A filter-only
or pass-through BAM run therefore rebuiltMM/MLfor every record and
rewroteMNat the wider subtype. Recognizing every integer width cuts about
27% of the CPU time from those runs and leavesMNat the width it arrived
with. Decoded tag values are unchanged. - Adapter trimming aborted the whole run with a panic, leaving a truncated
output file, when any read contained an ambiguity code (N,Y,R, ...)
near an end. Sassy'sDnaprofile panics during traceback on non-ACGT text,
and the searcher fell back to exactly that profile whenever it detected one.
Every searcher now uses the IUPAC profile, which is what tolerates ambiguity
codes. Output on pure-ACGT input is byte-identical; adapter trimming is about
19% slower. - uBAM records flagged reverse-complemented (
0x4|0x10) are refused rather than
trimmed: htslib decodes theirMMright to left with complemented bases and
their storedSEQis reverse-complemented, so trimming cropped the wrong ends
and relocated every call. - Records carrying the legacy lowercase
Mm/Mltags (old guppy, megalodon)
are refused rather than copied through untouched onto a trimmed sequence.
htslib still reads that spelling, so the calls decoded correctly elsewhere while
pointing at the wrong bases. --summary-jsoncould destroy the run's own output when reads went to a
redirected stdout (whittle --summary-json out.fastq > out.fastq): the
collision guard only covered the-oform.- Folder mode's progress bar was pinned at 0%, because nothing on that path
counted input bytes. Folder readers now wrap their files the way the
single-file path does. - A mistyped
--summary-jsonpath now fails during setup instead of after the
reads are written, and-o /dev/null --summary-json /dev/nullis no longer
refused as a self-overwrite. - Folder mode (
-i <dir>) was missing two advisories the single-file path
emits: the--out-formatextension mismatch and the no-trimming no-op
warning. It also rendered a bare spinner instead of a progress bar with a
percentage and ETA, despite having already measured the input. - Reverse-strand
MMgroups were counted against the complement of the
fundamental base, so every call on a-strand group was relocated onto a
different base, some were dropped, and theMLbytes shifted relative to
their positions. htslib counts the literal base; whittle now matches it.
Uis folded toT(BAM's SEQ encoding has noU) andNcounts every
base, both of which previously deleted the whole group. - BGZF compression and decompression ran single-threaded regardless of
-t.
Thenoodles-bgzf0.48 API ignored its worker count and used Rayon's global
pool, which whittle configured; 0.51 takes an explicit count and builds a
private pool, defaulting to one worker. Measured 3.5x faster BAM-to-BAM at
-t 16. - whittle no longer overwrites a file it is also reading:
--summary-json
pointing at the input, the output, or a folder member;-opointing at
--adapter-fasta; or-onaming the file redirected onto stdin, which the
path-based check could not see. - A cyclic
@PGchain that does not return to its entry node (pgAtopgB
topgCtopgB) hung the run at 100% CPU. The chain walk now tracks
visited IDs. --summary-jsonreportedelapsed_seconds: nullunder--quiet, and was...
0.1.1
whittle 0.1.0
whittle 0.1.0 — first public release.
whittle is a fast, tag-aware long-read trimmer for FASTQ and unaligned BAM (ONT/PacBio). Unlike FASTQ-only trimmers, it keeps position-indexed auxiliary tags in register through every crop, quality trim, and adapter-driven split:
- Base modifications (
MM/ML/MN) are reconstructed for the trimmed window, verified against an independent htslib decoder on real HG002 data. - Per-base kinetics (
ip/pw) are sliced in lockstep with the sequence. - ONT move tables (
mv/ns/ts) are dropped by default, or rewritten with--update-moves. - I/O: FASTQ, gzip/BGZF-compressed FASTQ, and unaligned BAM; parallel across cores; no external
htslibrequired. - Adapter trimming: a user-supplied FASTA, a built-in ONT catalog, or de-novo inference.
Prebuilt binaries for Linux and macOS (x86-64 and arm64, glibc and static musl) are attached below, with SHA-256 checksums. Also available on crates.io. Licensed under Apache-2.0.