Skip to content

v0.16.0

Choose a tag to compare

@teaguesterling teaguesterling released this 12 Nov 03:18
· 45 commits to main since this release
493bf5a

v0.16.0 — ATTACH surface, the sample axis (read and write), and the V2 matrix writer and reader; v2.0.0-only

This release opens pfile_catalog to ATTACH, completes the sample axis for both
reading and writing, and adds both the writer and the reader for the two-axis
matrix format. It builds for DuckDB v2.0.0 only (the v1.5.6 line stays on v0.14.2).

New — ATTACH (TYPE pfile_catalog)
ATTACH a catalog as a read-only database and query its relations by name.
- Works for every catalog kind: a single fileset, a v1 sharded set, and
the v2 matrix, resolved by an explicit path, a prefix, or a bare fileset.
- The seven relations: c.variants, c.samples, c.genotypes,
c.genotypes_by_sample, c.cells, c.variant_blocks, c.sample_blocks.
- Read-only, pushdown-aware, DETACH-clean; a hole that matches no cell
throws.

New — the sample axis, reading
read_pfile gains combine_samples for sample-sharded cohorts.
- combine_samples := 'concatenate' reads a cohort split across per-sample
shards that share one .pvar. It checks the variant count only (a
mismatch throws); documented as such.
- combine_samples := 'disjoint' plus parquet .psam companions supports a
cohort whose shards hold disjoint samples.

New — the sample-sharding writer
COPY (...) TO '...' (FORMAT pfile, shard_samples {...}) writes a cohort as
one shared .pvar, a per-shard .pgen and .psam, and a concat_dim:"samples"
catalog. A bind-time guard checks the file-descriptor and memory cost
against the shard count.

New — the V2 matrix writer (pfile_catalog : 2)
COPY (...) TO '...' (FORMAT pfile, shard_by '' shard_samples {...})
writes the two-axis grid: the variant roll composed with the sample
fan-out. It opens N writers per variant block (not M by N), stages every
file until finalize, and emits the catalog last. Variant keys may be
compound (CHROM plus POS). The reserved key names GROUP and CHUNK, a bare
LIST, a sample orientation, and non-contiguous input are each rejected at
bind.

New — the V2 matrix reader (the two-dimensional read)
read_pfile(catalog := '.pfile.json') reads a version-2 matrix as
one cohort across its variant and sample blocks. It reads version-2 matrix
catalogs only.
- All three orients, all genotype modes, and the sample, region and count
filters work; counts and frequencies use the whole cohort.
- With more than one variant block, select variants with region := (each
block has its own variant ordinals, so variants := and variant_range
are errors); with one variant block, both work.
- catalog := names every input, so a positional prefix and pgen/pvar/psam
overrides are errors with it.
ATTACH c.genotypes_by_sample on a matrix now uses this read, and is correct
for a complete matrix. A holed matrix still errors for the by-sample view
only when it has more than one sample block; a holed single-sample-block
matrix reads through the list path.

Fixes
- ATTACH c.genotypes_by_sample on a matrix with more than one sample block
is now guarded. It previously stacked the sample blocks and returned
wrong genotypes.
- ATTACH c.genotypes on a matrix now reads each cell with that cell's
templated .pvar and .psam, so a template that shares one .pvar per
variant block is read correctly.

Known limitation
read_pfile(catalog := …) needs a complete grid: a catalog with a hole (an
absent cell) is an error, because a 2-D read of a holed grid is not defined
yet. Each cell's .pgen must hold exactly the variant and sample counts its
blocks declare. The read rejects variants := and variant_range across more
than one variant block (use region :=), and it reads version-2 matrix
catalogs only.