Repository navigation
Releases
v1.6.10 — ITS2 evidence and simpler profiling
Highlights
- Add optional host/symbiont ITS2 multi-copy recovery to profiling: linked-read recruitment, observed-read graph paths, quality-aware candidate comparison, conservative phasing, and independent target/decoy and reference-span annotations.
- Preserve unresolved candidates and report phase components, independent variant-site support and rejection reasons. Shared or duplicate evidence cannot force low-frequency haplotypes into acceptance.
- Reduce alignment memory with rolling score rows and compact traceback while preserving full-matrix results.
- Keep three profiling-specific options:
--profile-copy-mode host|symbiont,--profile-group-map, and--profile-decoy. Use Themisto from PATH,-pfor sample concurrency, and--reference-cache-dirfor shared indices. - Fix index reuse across different output directories. Reference changes invalidate the cache.
- Synchronize bilingual READMEs, command-line manuals, output descriptions and profiling documentation.
Migration
Removed profiling k-mer/threshold/memory, tool-path, dedicated cache and force-rebuild flags are rejected. Remove those flags; use PATH and the common cache option where needed. GM2_THEMISTO is no longer read. Integrated defaults remain k=21, threshold=0.8, relevant fraction=0.5, and index memory=2 GB. Without copy mode, profiling remains assembly-free. Experimental non-default pseudoalignment remains available through the standalone marker_profile component.
Validation and limitations
- All 409 workspace tests, strict Clippy, formatting, distribution build and release metadata checks passed.
- The same three complete real WGS samples were run in host and symbiont modes before and after parameter simplification. All six comparisons produced byte-identical biological reports, recruited reads and assembly graphs.
- Combined elapsed time was 160.92 → 161.56 seconds (+0.40%). These single paired runs do not establish a stable speedup or slowdown.
- Shared-cache reuse across output directories, invalidation after reference changes, and assembly-free profiling were also verified.
- Low-frequency recovery is not complete: C3st remains unresolved where independent bridging evidence is insufficient. Read support does not prove all genomic copies have been recovered, and host mode lacks matched independent truth.
Installation
This is a source release, following the existing build/install workflow. No prebuilt binaries are attached.
See the changelog, profiling guide, and WGS comparison.
v1.6.9 — Reliable mitochondrial rescue and simpler defaults
Highlights
- Preserve both mates during mitochondrial refiltering so graph and junction checks receive valid read pairs.
- Verify rescue assemblies before replacing primary results. Empty, partial or failed rescues restore the primary assembly; only a read-supported circle replaces it.
- Remove stale finalization outputs on reruns, report current empty/failed states, and prevent short-circle standardization from accessing bases beyond the sequence.
- Honor one-pass
--max-reads. Support cleanup previews and limit cleanup to confirmed samples from the current invocation, preserving other samples' historical intermediates. - Use
tipseek mito -f samples.tsv -r reference.gb -o outputwith automatic defaults. The legacy--mito-genbankalias and expert overrides remain supported.
Validation and limitations
- All 372 workspace tests passed; release builds, formatting and diff checks passed.
- Real paired-read validation confirmed that the simplified reference option preserves outputs and that partial runs retain their reads without cleanup.
- The full real-read validation remained a partial multi-contig assembly, not a verified circle. This release fixes workflow correctness; it does not claim complete mitochondrial recovery for that dataset.
- Given identical valid reads and contigs, all six finalization output files were byte-identical before and after the fixes.
Installation
This is a source release, following the existing build/install workflow. No prebuilt binaries are attached.
See the changelog and mitochondrial guide.
v1.6.8 — Faster population processing and reliable stage reuse
Highlights
- Validate population stage dependencies and completion receipts; reuse raw joint calls when only genotype/site filters change.
- Stream fixmate, coordinate sorting and duplicate marking; create BAI indexes while writing final BAMs.
- Use indexed reference intervals for coverage QC, preserving complete flagstat statistics and single-thread fallback behavior.
- Reduce final BAM compression overhead with lossless level 1. In the four-library real UCE benchmark, this change reduced median runtime from 9.203 to 7.108 seconds, with final BAM files 5.28% larger. These are subset/host-specific measurements, not a universal speed guarantee.
- Simplify pipeline lifecycle handling and shared tool arguments. Downstream failures terminate and reap remaining producer processes.
- Fix empty R2 fields, normalized sample-ID collisions, same-file reference copies, executable resolution and stage-reuse validation.
Validation
- 368 workspace tests passed, along with Clippy and formatting checks; GitHub CI passed for the implementation commit.
- The final refactoring comparison preserved all 4,000,012 alignment records, all variant stages, 263 filtered SNPs, 9 one-SNP-per-UCE loci, full QC reports, PLINK panels and PCA results on the same real UCE subset.
- Real-tool regression checks cover single-thread execution, skipped duplicate marking, indexed region queries, coverage edge cases and raw-call reuse. ADMIXTURE and full-library performance were not tested.
Installation and compatibility
This source release follows the existing build/install workflow. External tools remain separate dependencies; samtools sort/markdup must support --write-index (validated with samtools 1.16.1). Existing exon backends and reference-ordered SCO output remain available.
See CHANGELOG and the validation report.
v1.6.7 — Independent exon backend and read-supported SCO recovery
Highlights
- Added exonerate-rs as an independent exon annotation backend, invoked from the current environment by command name. Miniprot remains available.
- Added reference-coordinate SCO reconstruction with one sequence per SCO: recovered sample bases follow reference order, while missing positions and unresolved conflicts remain N.
- Improved cross-reference evidence projection, short exon candidate screening, and multi-contig conflict handling with read support and auditable rejection reasons.
- Preserved reads containing internal N bases for evidence evaluation and added conservative recovery of observed terminal stop codons.
- Added bounded family scheduling and reduced redundant candidate-chain comparisons while preserving selection and ambiguity handling.
- Fixed output-path overlap checks involving symbolic links and parent-directory components.
- Updated bilingual documentation and generic validation tools; removed sample-specific study materials from the release tree.
Validation
- 361 workspace tests passed; Clippy with warnings denied and Python script compilation passed.
- GitHub CI and CodeQL passed for the integrated implementation, including the distribution build and CLI smoke check.
Usage notes
- Install the selected annotation backend separately and make it available on PATH.
- Reference-coordinate SCO output preserves unknown and conflicting positions; it does not imply a fully recovered biological CDS or a physically validated contig join.
v1.6.6 — Table-based UCE evidence and Apple silicon tuning
Highlights
- Replaces FM-index extension in UCEFilter exact-match evidence with an exact run-k table when run-k ≤ 32 (default 19): each run-k window costs one lookup, and a MEM extends by comparing 32 packed bases per step against each occurrence of its run-k prefix. Larger run-k keeps the FM index.
- Stores the run-k table compactly (two bits per reference base, one fingerprinted
u32per hash slot); peak memory on the benchmark below is about 16 MB above the FM index. - Detects the host processor and memory:
-p autoreports performance and efficiency cores on Apple silicon, macOS measures available memory instead of falling back to a 512 MiB UCEFilter fragment bank, and compute threads ask macOS to prefer performance cores. - Assembler k-mer counting workers claim small blocks instead of fixed equal slices, so faster cores take more work.
- Leaves scheduling unchanged: stage thread budgets, stage priorities and sample order still follow the GeneMiner2 dispatcher.
Performance
575,435-pair UCE benchmark used for v1.6.5 (2,211 bivalve loci; on-target pairs simulated from real DK40 UCE contigs, off-target pairs from a real WGS library; Apple M2, -p 8, default filter assemble --assembly-mode uce), two runs each:
| Stage | v1.6.5 | v1.6.6 |
|---|---|---|
| Total | 47.3–49.8 s | 32.3–33.2 s |
| First-pass filter | 4.8–5.0 s | 1.8–2.0 s |
| Rescue-round filter | 18.8–20.0 s | 5.5–6.7 s |
Assembly time is unchanged. UCEFilter alone on a 675,435-pair variant of the benchmark: exact-match evidence 4.7 s → 0.85 s, peak RSS 331 MB → 347 MB.
Verification
- All result files were byte-identical to v1.6.5 on the benchmark above (5,556 files) and on real phyluce Gallus UCE reads (5,123 files); only run logs and manifests differ.
- New randomized equivalence tests compare the run-k table path with the FM index (seeds, occurrence counts, strand masks, MEM statistics), packed with bytewise prefix comparison, and k-mer counts across worker counts.
- 286 workspace tests passed;
cargo fmt --checkand Clippy with warnings denied passed; release build completed.
v1.6.5 — Faster UCE recruitment, rescue and assembly
Highlights
- Prunes exact-match (MEM) evidence in UCEFilter: a read start stops extending past run-k once its longest possible match can no longer reach the current best seed, and unique run-k strands come from a per-row table instead of a sampled suffix-array walk.
- Rewrites rescue internal-gap QC on packed 15-mer keys with one pass over reads for both the before and after contigs, and lists rescue read directories once per round instead of once per restored locus.
- Counts assembler k-mers with 64-bit keys when k ≤ 32, including the k=21 rescue assembly.
- Leaves scheduling unchanged: stage thread budgets, stage priorities and sample order still follow the GeneMiner2 dispatcher.
Performance
575,435-pair UCE benchmark (2,211 bivalve loci; on-target pairs simulated from real DK40 UCE contigs, off-target pairs from a real WGS library; Apple M2, -p 8, default filter assemble --assembly-mode uce):
| Stage | v1.6.4 | v1.6.5 |
|---|---|---|
| Total | 82.6 s | 49.1–49.6 s |
| First-pass filter | 8.3 s | 5.2–5.4 s |
| Rescue round | 70.7 s | 39.5–40.5 s |
Verification
- All result files were byte-identical to v1.6.4 on the benchmark above and on real phyluce Gallus UCE reads; only run logs and manifests differ.
- New randomized equivalence tests compare each optimized path with the previous implementation (MEM seeds, counts and strands; per-base gap-support profiles; u64 vs u128 k-mer counts) and were mutation-checked.
- 283 workspace tests passed;
cargo fmt --checkand Clippy with warnings denied passed; release build completed.
v1.6.4 — Exon reference derivation and validated fragment joining
Highlights
- Derives auditable protein references from coding nucleotide baits in
--assembly-mode exon; external.faafiles remain optional per-family overrides. - Simplifies complementary-fragment screening: query overlap is no longer an independent veto, unique coverage follows
--gene-min-model-coverage, and a one-residue margin is used only to decide whether padded reannotation should be attempted. - Keeps the final safety boundary unchanged: a padded join is accepted only after unique complete-model reannotation places the entire synthetic interval inside an intron and outside every exon.
- Validates the behavior across 56 real multi-exon Patiria pectinifera genes with 2.0–20.0-kb target introns; all 29 accepted padded CDS results match their unbroken-candidate results.
- Adds the integrated GeneMiner2-to-TipSeek innovation schematic and harmonizes the Chinese-primary, English-switchable documentation across exon, UCE, mitochondrial, RAD, population, profiling, and repeatome workflows.
Verification
- 280 workspace tests passed.
- Clippy passed with warnings denied.
- Release build completed successfully.
- 198 local documentation links were validated.
v1.6.3 — Integrated exon annotation
Integrated exon annotation
Highlights
- Added
--assembly-mode exonas the complete protein-guided extension of the default gene-family recovery workflow. - Added deterministic miniprot-based exon model selection and structural QC for coverage, CDS phase, translation, frameshifts, internal stops, splice classes, and competing models.
- Added coordinate-preserving CDS, protein, exon, intron, gene, supercontig, GFF3, model, and audit outputs.
- Added conservative N-padded joining for one unambiguous pair of complementary terminal partial models. A derived join is accepted only after reannotation produces one complete resolve-eligible model and places the entire synthetic interval inside an intron without exon overlap.
- Simplified the public interface:
tipseekis the default gene workflow,--assembly-mode exonadds structural annotation, and--assembly-mode uceremains independent. - Added focused Chinese and English exon documentation and separated candidate recovery, exon annotation, and phylogenetic resolution into non-overlapping guides.
Exon-mode input
Each nucleotide family reference (.fa or .fasta) must have a matching protein reference with the same family stem and the .faa extension. Exon annotation requires miniprot 0.18 or later.
cli/tipseek --assembly-mode exon \
-f samples.tsv \
-r family_reference \
--gene-protein-reference family_proteins \
-o exon_output \
-p 8Validation
cargo test --workspace: 272 passedcargo clippy --workspace --all-targets -- -D warnings: passed- Documentation links, code fences, deprecated invocations, and whitespace checks: passed
Full Changelog: v1.6.2...v1.6.3
v1.6.2 — TipSeek identity and evidence-bounded UCE recovery
TipSeek identity and evidence-bounded UCE recovery
Highlights
- Renamed the public project from TStools to TipSeek and changed the primary executable from
cli/geneminer2tocli/tipseek. - Changed UCE defaults to k=23/step=4 automatic recruitment followed by one evidence-constrained rescue round.
- Strengthened fallback-locus assignment and rescue eligibility with full-panel probe checks, explicit provisional-core states, long-inverted-repeat rejection, internal-gap review, and per-locus audit tables.
- Added the TipSeek UCE manuscript and supplementary methods, including the 11-sample coral benchmark of TipSeek R0/R1/R2, GeneMiner2, and SPAdes + PHYLUCE.
Command change
The main command is now:
cli/tipseekExisting scripts that invoke cli/geneminer2 must update that executable path. Workflow names and their task-specific options remain available through the new entry point.
UCE defaults and evidence controls
- UCE mode now defaults to
-kf 23, step 4,--uce-recruit-mode auto, and one rescue round. - Recruitment k is independent from exact-match verification k, allowing more sensitive recruitment without weakening the verification threshold.
- Automatic recruitment retries only fast-pass unresolved loci, then evaluates gated fragments against the complete probe panel before locus assignment.
- Fallback-only candidates must meet length, target coverage, identity, and panel-uniqueness gates before becoming provisional cores.
- Exact long inverted repeats are rejected. Internal read-chain gaps are reported as review evidence; review-only cores are retained but cannot seed rescue.
- Rescue remains reversible per locus, and a failed extension restores the previously accepted candidate.
- The previous k=31 fast-pass workflow without rescue remains reproducible through explicit overrides.
Manuscript and benchmark
- Restructured the UCE manuscript to follow the Bioinformatics original-paper format.
- Retained the coral-only benchmark and full TipSeek R0/R1/R2 results, including sample–locus recovery, panel coverage, shared loci, sequence length, wall time, and peak RSS.
- Added supplementary methods, reproducible command placeholders, and the complete benchmark table.
Validation
cargo test --workspacegit diff --check
Full Changelog: v1.6.1...v1.6.2
v1.6.1 — Automatic UCE recruitment and manuscript draft
Automatic UCE recruitment and manuscript foundation
Highlights
- Added the opt-in
--uce-recruit-mode autoworkflow for conservative recovery of unresolved UCE loci. The default remainsfast. - Renamed the public project to TStools while retaining repository URLs, the
cli/geneminer2entry point, legacy output identifiers, and scholarly citations for compatibility. - Added the Chinese TStools/UCE manuscript draft under
manuscript/TStools_UCE_draft_ZH.mdfor collaborative revision and provenance tracking.
Conservative automatic recruitment
- Retries only fast-pass unresolved loci with a sensitive k=21/step=1 coarse-recruitment gate.
- Requires fallback reads to provide a 45 bp alignment at 80% identity and merges only newly recovered locus FASTQs.
- Requires fallback-only contigs to be at least 200 bp, cover at least 80% of the target probe at 80% identity, and avoid near-tied panel loci.
- Preserves acceptance and rejection evidence in auditable TSV sidecars and archived outputs.
Validation
- GitHub CI passed, including MSRV checking, formatting, workspace tests, Clippy, distribution build, release metadata, and CLI entry-point validation.
- GitHub CodeQL analysis passed.
Full Changelog: v1.6.0...v1.6.1