figwig 0.2.0

figwig 0.2.0 adds the figwig file format: any number of tracks over one genome in one file, read many windows at a time from any subset of them, with converters from bigWigs and BAMs, a per-cell converter for fragments files, and a Claude Code skill that covers it. The bigWig reader and writer are faster, and figwig bam2bw gains a coverage mode. For anyone who reads training data from many tracks, or turns reads into count tracks.

Commits

• 787cc7e FIX the coverage mode's changelog entry moves from the released 0.1.0 section, where PR #21 appended it, to 0.2.0
• 758ec5a RELEASE bump version to 0.2.0 with changelog entries
• b02f0c7 DOC the first-call times after the kernels pass constants as plain values, and the changelog entry
• 8b10618 UPDATE the kernels pass constants to each other as bool() and int(), so numba compiles one copy of a callee per type rather than one per literal value: a quarter fewer specializations to compile on a first call
• 8261395 FIX CI compiles the kernels for a generic CPU, so numba's cache loads on any runner of an OS
• 4d467ab FIX the commit-time step uses touch -t, which macOS's touch takes too
• ee48d7b FIX the CI numba cache key is the hash of the kernel files with no fallback, since numba does not notice a constant or kernel used from another file changing
• 9caaa5f UPDATE the test suite runs in a quarter of the time and CI keeps numba's cache: bam2bw's tests run the command in the test process, the damage test samples the chunks, pytest-xdist in the dev extra and on every runner core
• 836c987 DOC the skill, docs and README follow the dense-path and converter-check PRs: the defaults for dense float tracks, the BAM header and bw2fw sizes checks where the commands are described, and six corrected sentences
• 7fb5d57 FIX figwig bam2bw warns, rather than refuses, when a BAM's header disagrees with the sizes, keeping bam2bw 0.5.1's discarding of reads outside them
• ec4ed35 ADD figwig bam2fw and bam2bw check each BAM's header against the sizes: a shared chromosome of another length, or no shared chromosome, is refused before any read
• 11fc36e UPDATE dense float tracks read 3 to 6 times faster and write up to twice as fast on many threads: runs chunks decode in place, and read batches and write tasks are cut per thread
• c91109c FIX bw2fw refuses a sizes file sharing no chromosome with a bigWig, an empty track round-trips through fw2bw and bw2fw, and a bad BAM is a message from bam2fw and bam2bw
• 474f3be UPDATE the statistics' minimum and maximum are those of the nonzero bases, each summary stream carries a CRC-32, and the specification gains worked chunks and the rules a clean-room reader asked for
• 70a1c17 DOC the figwig file format, version 1: a specification, a reader written from it, a Kaitai Struct definition and tests against the files the package writes
• 86cbd8f UPDATE the bigWig writer plans each batch on its own thread and encodes and compresses it on the pool, and confirms sorted input in one compiled pass
• 653f352 ADD engine='isal' on the bigWig reader: ISA-L's inflate through python-isal's extension, from numba without the GIL, after a one-block self-check
• 58f7f90 ADD tests of the coverage mode's edges: the run decoder, every BED reader, pooled and separate files, shifts that clip or empty reads, small write batches, empty inputs and the command line
• d097dcc DOC the coverage mode against deepTools on ten experiments: the same value at every base, 3 to 8 times faster
• d0ebdee FIX the coverage helpers are private names
• 16d45b4 ADD figwig bam2bw -c and bam2fw -c, the coverage of every aligned base, as deepTools counts it
• 6bc259a DOC the per-cell conversion's numbers after the third optimization pass
• 1260cd7 UPDATE chunk CRCs in one numba call per batch, and entry batches of at least a million entries bounded by their tiles
• b6c4473 UPDATE the cell scanner inflates a unit in one call, joins chromosomes on the pool, and the writer folds a batch's statistics in one pass
• 66accbb DOC the per-cell conversion's numbers after the optimizations, every method timed again at a load of 31 to 40
• cb2946e UPDATE the entries encoder visits only the tracks a tile touches
• dcf9114 UPDATE the cell scanner finishes each chromosome on the pool, and the writer's presorted path makes no pass per entry
• 3cd0603 UPDATE the cell scanner skips an unlisted barcode in the kernel, so the scan gains to 8 threads
• 3610bcc UPDATE bam2fw --barcode-column hands the writer its entries in the encoders' order, which the writer then does not sort
• c8eb849 UPDATE bam2fw --barcode-column groups a file's cells within a chunk's limit, hands on 32-bit arrays, and is timed against pandas, scipy and SnapATAC2
• 71ee8cb FIX the key-packing test builds its expectation per sampled chromosome, not a 56 GB array
• 874208f FIX bam2fw --barcode-column refuses a truncated gzip or BGZF file with a message, and tests every malformed case
• 1b0fbbb ADD figwig bam2fw --barcode-column, a fragments file as one track per cell
• da379e8 UPDATE the skill as a router to operations, with recipes for whole jobs and no machine-bound timings
• 4c4b4ae DOC unify the README, docs, changelog and skill after the format work
• 3375917 Merge pull request #17 from jmschrei/figwig_format
• 5549e12 DOC changelog entry for the zoom-record kernel holding the open record in locals, and its comment in the present tense
• e27aa29 Keep the open zoom record in locals in _zoom_records
• b1abdc7 DOC list every figwig bam2bw argument in a README dropdown
• ad3da25 Update README.md

figwig files

New — the figwig file (.fw) holds the per-base values of any number of tracks over one genome, with a JSON record of attributes per track and the name of the genome, and reads many windows at a time from any subset of its tracks into one (windows, tracks, width) array in the tracks' own dtype. FigWigReader, read_figwig, FigWigWriter and write_figwig mirror the bigWig API with a tracks= argument; select(**attributes) gives the tracks whose records match, stats each track's sum, nonzero bases, minimum and maximum over its nonzero bases, and written_by the figwig that wrote the file. Version 1 of the format is specified and frozen on the format page of the docs, with tests/format_reference.py, a reader written from the page with numpy and zlib alone, and docs/figwig.ksy, a Kaitai Struct definition, both run over files of every layout; a reader refuses a file of a later version or with a required feature it does not know.

New — layout. Every track lies on a grid of tile bases, and the tiles of a group of tracks are one chunk: nothing when every value is zero, else dense, sparse, runs or a per-tile matrix, whichever is smallest, in the narrowest element type that holds the values exactly, with a one type that stores no values when every nonzero value is 1 and a float64 type for an integer above 2**24 beside a fraction. Chunks are compressed with zlib or libdeflate on n_jobs threads, or stored raw when deflate would grow them (level=0 stores every chunk raw). The index holds only the tiles with values, so a file of 100,000 single-track groups opens in a tenth of a second. order='tracks' stores each track's nonzero bases in blocks per chromosome instead, for reading a few tracks over the genome; figwig reorder rewrites a file in the other order.

New — reads. read(..., sparse=True) returns the nonzero bases as a SparseWindows of entries, labels= sums the tracks per label (cells into pseudobulks), bins= sums each window per bin of bases, read_entries(tracks, chroms) returns every nonzero base of tracks over chromosomes, and channels=/items= arrange the tracks read by two attributes into (windows, items, channels, width). A read of mixed tracks comes back in the narrowest type that holds every track exactly, and a dense read whose output would pass 8 GiB raises instead of allocating it. sums(chroms, starts, ends, tracks=None, nan='propagate', paired=False) sums intervals of any lengths; nan='skip' leaves NaN bases out and nan='count' gives the bases that are not NaN, so a mean over covered bases as pyBigWig and deepTools give it is one over the other; paired=True sums interval i over track i alone. tile_sums gives each track's sum per tile from the file's summaries, and tile_sums(level=k) the zoom levels a file written with zooms= keeps, without decoding a chunk.

New — writes. write takes windows or intervals of whole groups, write_sparse(chroms, positions, tracks, values) nonzero bases as entries, both encoded and compressed on the n_jobs threads. A track's dtype is declared (uint8 to uint32, int8 to int32, float32, float64) or inferred; a float32 track keeps float32's rounding, and a value beyond float32, a 64-bit integer beyond 2**53 or an infinite value is refused before anything is written. The file is written beside its path and moved there at close, so a failed write, a writer never closed (which warns) or a conversion over its own input leaves the path as it was. FigWigWriter(path, None, tracks, append=True) adds tracks in place under the file's lock, writing the new chunks after its end and the header last, so a writer that fails or is killed leaves the file readable as it was; add_file and figwig merge copy the tracks of files written with the same chromosomes, tile, level and order.

New — integrity. Every chunk carries a CRC-32, checked with verify=True; the header, the tables and each index record carry CRC-32s checked at open, and every count and pointer is bounded against the file before its bytes are read, so a damaged, crafted or truncated file raises ValueError rather than a MemoryError or a crash. A reader checks on every read that the file at its path is the one it opened, so a file replaced by a conversion raises instead of reading with a stale index.

New — device="cuda" on read, sums an...

figwig 0.1.0

figwig 0.1.0 is the first release of figwig, a fast, multithreaded reader and writer of bigWig files, into and out of numpy. It is for anyone who reads the signal under many genomic windows, such as the training data of a sequence-to-function model, writes bigWigs from numpy arrays or model predictions, or converts reads to per-base count tracks with figwig bam2bw.

Pull requests

• #1 Prepare figwig 0.1.0: read_windows, missing windows, Windows and macOS reading, fixes and CI
• #2 Add BigWigWriter and write_bigwig: multithreaded bigWig writing with zoom levels
• #3 Make writing mirror reading: read_bigwig, write_bigwig and BigWigWriter.write
• #4 FIX measure each writer benchmark run's own peak memory, not its parent's
• #5 UPDATE rename BigWig to BigWigReader
• #6 ADD figwig bam2bw, bam2bw with the speed search's readers and figwig's writer
• #7 UPDATE figwig bam2bw: read each file on every core, SAM-like files in a pool, no joblib
• #9 UPDATE figwig bam2bw: drop counted keys before the keys array grows
• #10 FIX figwig bam2bw's BGZF inflate kernel using its scratch buffer after freeing it
• #11 DOC split the README into a shorter README, user-guide docs and a Claude Code skill
• #12 DOC fix the figwig skill and docs where agent tests found them wrong or short
• #13 FIX np.str_ names in messages; DOC figwig bam2bw on Windows and a comment's timings
• #14 RELEASE figwig 0.1.0

The full history is git log v0.1.0.

Reading

New: BigWigReader and read_bigwig.

• BigWigReader(path).read(chroms, starts, width, out=None, n_jobs=8, missing=0.0) reads the per-base values of many windows of one width into a float32 array of shape (n, width).
• read_bigwig given a list of bigWigs reads them all into one array of shape (n, len(bigwigs), width), which is the (batch, channels, length) layout that sequence models take.
• The work runs on up to n_jobs threads. These run in parallel because zlib's uncompress() and figwig's numba kernels release the GIL.
• On 167,750 windows of 1,000 bp from an ENCODE ATAC-seq bigWig, figwig took 0.135 s on 8 threads, against pybigtools' 1.25 s, on a 2x AMD EPYC 9575F.

The values are those pybigtools' values() gives, bit for bit:

• a base that no interval covers is missing, 0 unless given;
• a base past the end of its chromosome is NaN;
• a window on a chromosome the file does not have is missing throughout, with a warning.

What it reads:

• bedGraph, varStep and fixedStep sections, compressed or not.
• A file it cannot read exactly raises a ValueError that says what it found, such as a bigBed, a big-endian file, a corrupt index or overlapping intervals.
• A BigWigReader can be read from several threads at once, and pickled into DataLoader workers.

The reader began inside tangermeme's extract_loci (tangermeme #107).

Writing

New: BigWigWriter and write_bigwig.

• write_bigwig writes what read_bigwig reads: windows, intervals or single bases, from (n, width) or (n, len(paths), width) arrays.
• BigWigWriter writes the same a batch at a time, so a track larger than memory can be written in pieces.
• Data blocks are compressed on several threads, with zlib, or with libdeflate through the new fast extra.
• Each window gets the smallest section type: fixedStep, varStep or bedGraph.
• Writing 15.9 million per-base counts took 0.18 s on 8 threads, against pyBigWig's 4.20 s.

Files match pyBigWig's. Files are laid out as libBigWig lays them out. A file of intervals written with missing=numpy.nan, zlib at level 6 and no zoom levels is byte for byte the file pyBigWig writes. Where libBigWig writes a wrong value, figwig writes the right one:

• the header's maximum;
• the end of the last fixedStep block of each call;
• the sums of the last zoom record of each zoom block.

figwig bam2bw

New: the figwig bam2bw command. It is bam2bw 0.5.1 with the same arguments, outputs and messages.

• A BAM file is read with libdeflate and a numba kernel on -p threads, and a BED/tsv file with a numba kernel.
• The counts are written by BigWigWriter.
• On a 2.4 GB ATAC-seq BAM it took 4.65 s and 910 MB at -p 2, against bam2bw's 58.48 s and 3,039 MB.
• It gave the same exit code, messages and decoded entries as bam2bw in all 3,781 runs that bam2bw completed, over 591 public test files.
• It needs the new bam2bw extra, and runs on Linux and macOS.

Claude Code skill

New: figwig install-skill. It installs a Claude Code skill that teaches a coding agent how to use figwig:

• reading windows, and data loaders;
• writing predictions, intervals and counts;
• converting reads with figwig bam2bw;
• the rules a write follows, and what each error message means;
• the measured gain from threads.

Run figwig install-skill --force after upgrading figwig, to replace the installed copy.

Platforms

• Tested on Linux with Python 3.10 to 3.14, and on macOS and Windows with Python 3.10 and 3.13.
• It depends only on numpy (1.23 or later) and numba (0.58 or later).

---

This is the first release. Reading the per-base signal under hundreds of thousands of windows, from one or many bigWigs, used to mean a Python call per window that more threads did not speed up. With figwig it is one call whose threads run in parallel. Writing bigWigs from model predictions, and converting reads to count tracks, are each faster than the tools they replace. There is no earlier version to be compatible with.

🤖 Generated with Claude Code