Skip to content

v1.6.0

Choose a tag to compare

@github-actions github-actions released this 13 May 23:02

A broad feature release on top of 1.5.0. Highlights: bigBed / bigWig
and 2bit support via a vendored libBigWig and a hand-rolled 2bit
reader; full TUI exploration suite (column stats, sort, show/hide,
live filter, copy cell); generic Parquet range queries with column
auto-detection; markdown output and LociSSD validation; coordinate-
convention selector with UCSC and NCBI labels; color themes; LTO +
ccache static builds; and ARM64 release artifacts.

Changed

  • Smarter inline truncation for list / map cells. The
    truncator now walks every top-level comma and picks the largest
    leading-element prefix that fits the column. Previous behavior
    always dropped to [first, …] after the first comma, even when
    [first, second] would have fit. Most visible on VCF INFO
    expansions, BAM tags, and other multi-element list cells common
    in real data. Falls back to char-truncation only when even one
    element + ellipsis doesn't fit.

Added

  • TUI copy cell (y key) — copies the top-left visible cell to
    the system clipboard via OSC52. Works through SSH and tmux 3.3+
    with no xclip / pbcopy helper required (modern terminal
    emulators intercept the escape directly). Status bar flashes
    copied: <preview> until the next key; auto-clears.

  • UCSC 2bit (.2bit) — sequence container used for genome
    references (hg38.2bit, mm10.2bit). vv exposes the sequence index
    rather than decoded bases (chromosome-scale strings are too large
    to materialise): name, length, n_blocks (unknown-base runs),
    mask_blocks (soft-masked-region runs). Endianness is detected
    from the signature; long-offset 64-bit 2bit is rejected with a
    clear message (use twoBitToFa for that variant).

  • --theme <name> — color palette selector. Built-in themes:
    default, dark, light, solarized-dark, solarized-light
    (solarized accepted as a synonym for solarized-dark). Affects
    both the non-interactive ASCII table (ANSI escapes) and the
    ncurses TUI. Each theme provides a 256-color palette and a
    16-color fallback for older terminals; the light theme also
    suppresses zebra stripes so the table reads cleanly on a bright
    background. Unknown theme names produce a clear error listing the
    available choices.

  • TUI live filter (& key) — long-requested. Opens a
    &<expression> input bar; expression grammar matches the
    --filter CLI flag. Rows not matching the predicate are hidden
    from the display; status bar shows filter:<expr> N/M (rows
    kept / total). Composes with sort (s): one full-file pass
    rebuilds the combined view via the existing source_row(display)
    indirection. Empty input or Esc (in normal mode) clears.

    Internals: a new rebuild_display_order() orchestrates both
    sort and filter into the single sort_order_ permutation. When
    both are active, rows that pass the filter are collected with
    their sort keys in one pass and sorted at the end.

  • --decode-threads N — separately size Arrow's CPU thread pool
    (Parquet column decode and CSV / TSV parallel parsing) without
    touching htslib's thread count. Defaults to --threads; useful
    when cold reads bottleneck on column decompression. Bounded at
    2 × hardware_concurrency().

Changed

  • Release pipeline: now builds both x86_64 and aarch64
    (ARM64) static Linux binaries on every tag. The release workflow
    matrices over ubuntu-latest and the new GitHub-hosted
    ubuntu-22.04-arm runner — native builds on each architecture,
    no QEMU emulation. Tarballs are named
    vv-<ver>-linux-<arch>.tar.gz; a merged SHA256SUMS covers both.
  • AlmaLinux 8 Docker build: dropped the -march=x86-64
    baseline flag from the build-app CMake invocation so the same
    Dockerfile builds on aarch64. (-march=x86-64 is the default
    target on x86_64 gcc anyway, so the change is a no-op there.)
  • AlmaLinux 8 Docker build: rebuild every static dep + vv itself
    with -flto=auto. gcc-ar / gcc-ranlib preserve LTO IR in archives
    so the final link can drop unused code across library boundaries
    (not just within each .a). Significant text-segment shrink vs
    the previous non-LTO 15 MB static binary.
  • AlmaLinux 8 Docker build: install ccache (via EPEL) and put
    its compiler-symlink dir ahead of gcc-toolset-12 on PATH. A
    BuildKit cache mount (id=vv-ccache) persists the compile cache
    across docker builds. Requires Docker BuildKit + buildx.

Added

  • TUI interactive-exploration batch — three new keybindings turn
    the ncurses viewer into a real exploration tool:
    • S opens a per-column stats popup (count, nulls, min, max,
      mean, distinct count) for the leftmost visible column. Full-file
      scan with progress in the status line.
    • s sorts by the leftmost visible column. Repeated s on the
      same column toggles ascending / descending; s on a different
      column re-sorts ascending; u clears the sort. Numeric types
      sort by raw value, others lexicographically; nulls last. Search
      and the detail pane follow the sorted display order via a single
      source_row(display) indirection that routes all cache lookups
      through the sort permutation when active.
    • c opens a column show/hide picker overlay; Space toggles
      the column under the cursor, hidden columns drop from the
      layout, and the status bar shows a hidden:N indicator.
  • --tail N — show the last N rows instead of the first N.
    Mirrors -n but operates on the tail. Reuses the
    MemoryTableSource adapter: every chunk is read (through any
    active --filter) and the last N rows are sliced off, so the
    result renders identically through every view / export mode
    (table, TUI, TSV, CSV, JSON, Markdown, Parquet).
  • --coords UCSC|NCBI — pick the coordinate convention for
    -r. UCSC (default; aliases 0-based, bed) means 0-based
    half-open as in BED, bigBed, bigWig, BAM and LociSSD — the
    convention introduced by Jim Kent's UCSC Genome Browser source
    tree in 2000. NCBI (aliases GenBank, 1-based, tabix)
    means 1-based inclusive as in GenBank (1982), VCF, GFF, and the
    samtools / bcftools / tabix command lines. vv -r chr1:101-200 --coords NCBI file matches the same rows as the UCSC default
    vv -r chr1:100-200 file. Conversion happens once in
    apply_region_modifiers; downstream sources see normalized
    UCSC. --regions-file is always BED (UCSC) per the spec.
  • --parquet - — write Parquet to stdout. Parquet's
    footer-at-end requires seekable writes, so the data is spooled
    to an mkstemps temp file under /tmp, then streamed to stdout
    and unlinked. Bit-identical to writing to a file path. Enables
    vv ... --parquet - | tool pipelines.
  • Generic Parquet range queries-r chr1:1000-2000 file.parquet
    now works on any Parquet file with chrom/start/end columns, not just
    LociSSD. Chromosome / Start / End column names are auto-detected
    (common variants: Chromosome/Chrom/Chr, Start/POS/
    chromStart, End/Stop/chromEnd). When auto-detection isn't
    clear (or your columns have unusual names) use
    --region-cols Chr,Start,End. Row-group pruning uses Parquet
    ByteArray statistics on chrom plus Int statistics on Start/End;
    per-row filtering inside surviving row groups is correct for
    unsorted files too, just less efficient. Dictionary-encoded chrom
    columns are handled transparently.
  • --validate — LociSSD invariants checker. Verifies the
    lociSSD_manifest footer parses, the required columns
    (Chromosome / Start / End / MaxEndSoFar) have the right types,
    the manifest's per-chromosome row ranges cover every row
    contiguously, rows are sorted by (Start, End) within each
    chromosome, MaxEndSoFar[i] matches max(End[chrom_first..i]),
    and every row's Chromosome label agrees with the manifest's
    window. Prints PASS / FAIL per check, caps repeat violations
    at 5 lines per category, exits non-zero on any failure.
  • --md / --markdown — write a GitHub-flavored markdown table
    to stdout. Useful for pasting query results into issues, READMEs,
    and reports. Cells escape | as \| and turn embedded newlines
    into <br>. Honours --select, --filter, -n, --no-header,
    -r, --sample. Numbers keep their _ thousands separators
    so the rendered table stays readable.
  • bigBed / bigWig support via a vendored copy of
    libBigWig (0.4.8, MIT-licensed,
    compiled with -DNOCURL so only zlib is required). Source lives
    under vendored/libBigWig/; CMake compiles it as a private static
    library (libbigwig.a) and links it into vv. The static AlmaLinux 8
    Docker build picks it up automatically — no new external dependency
    for any distribution channel.
  • bigBed: the embedded autoSql definition is parsed into typed
    Arrow columns (signalValue, pValue, qValue, …). A small
    ~150-LOC hand-rolled parser handles primitives, char[N] fixed-
    width strings, enum{…} / set{…} (as strings), and arrays
    (int[N] / int[fieldName] → Arrow list<int>).
  • bigWig: rendered as four columns (chrom, start, end, value).
  • Range queries (-r chr1:78-99) use libBigWig's overlap APIs
    (bbGetOverlappingEntries / bwGetOverlappingIntervals); no
    external index needed.
  • User manual at docs/USAGE.md, with worked examples for every
    flag, every output mode, the LociSSD / tabix / BCF range-query
    flows, and a "by-workflow" cookbook section. docs/build_docs.sh
    renders it to self-contained HTML and PDF via pandoc (with a
    texlive → chromium-via-HTML fallback).
  • BCF range queries-r chr1:100-200 file.bcf now works, using
    bcf_itr_querys over an existing .csi / .tbi index. Missing
    index produces a clear hint pointing at bcftools index.
  • --regions-file <BED> — read additional windows from a BED file's
    first three columns. Combines with explicit -r; both are taken.
  • --slop N — pad every window by N bp on each side (start clamped
    at 0). Equivalent to bedtools slop inline.
  • --stats — Parquet metadata dump (row groups, codecs, per-column
    compressed / uncompressed sizes, null counts) without reading any
    data. Falls back to --schema for non-Parquet sources.
  • --unique COL[,COL,...] — distinct value counts per column.
    Top-50 values per column with overflow indicator. Honours --filter.
  • --sample N — reservoir-sample N rows uniformly instead of
    head-N. Honours --filter (filter first, then sample). Combines
    with every view / export mode through a new in-memory
    MemoryTableSource adapter that wraps the sampled Arrow Table and
    routes it through the normal rendering pipeline.
  • --schema — print the schema + file metadata footer and exit.
    Cheap, useful for inherited files.
  • --describe — per-column statistics: count, nulls, min, max,
    mean (numeric), distinct count (string, capped at 16). Respects
    --select and --filter.
  • Column projection by name--select Chromosome,Start,Score.
    Unknown names produce a clear error. Works across the ASCII table,
    vertical-head, TSV, CSV, JSON, and Parquet output paths. Display
    modes still hide format-derived columns (e.g. LociSSD's
    MaxEndSoFar); export modes preserve the user's explicit list.
  • --filter <expr> — value-predicate filter: <col> <op> <value>
    joined by AND / OR. Ops: == != < <= > >=. Values: integer,
    float, single- or double-quoted string. Evaluated per row.
  • JSON / NDJSON output--json writes a JSON array, --ndjson
    writes one JSON object per row. Streams chunk-by-chunk.
  • LociSSD range queries: -r chr1:78-99 file.lociss (also
    --window). Pruning uses the embedded manifest to locate the
    chromosome's row range, Parquet column statistics on Start and
    MaxEndSoFar to skip row groups outside the window, and a per-row
    predicate (Start < end AND End > start) inside surviving row
    groups. Open-ended forms (chr1:, chr1:78-, chr1:-99) and
    comma-separated multi-windows supported. Plain Parquet without a
    LociSSD manifest is rejected with a hint.
  • TUI: help overlay (H or F1) — centred panel listing every
    keybinding. Any key dismisses it.
  • TUI: frozen first column (z toggle) — pin column 0 at the left
    edge while scrolling sideways. Useful for wide tables where the key
    column (CHROM, Name, ...) should stay visible.
  • TUI: mouse-wheel scrolling — scroll three rows per wheel notch.
  • LociSSD support — Parquet files carrying a lociSSD_manifest key
    in their footer (see FORMAT_SPEC.md) are detected
    automatically. The derived MaxEndSoFar technical column is hidden
    from the ASCII table, the vertical-head view, and the interactive TUI;
    it remains intact in --tsv, --csv, and --parquet output so the
    data round-trips losslessly. New virtual hidden_for_display() hook
    on TabularSource provides the mechanism.