adata-cli 0.6.1

Makes the outputs of subset, split and concat the size anndata writes; they were about twice that. Also makes split read each matrix once rather than once per group. In 0.6.0 a hand-written anndata loop split the benchmark 7× faster than split did, and split is now the faster of the two.

Fixed

Sparse outputs were about twice as large as they needed to be. Every sparse output of subset, split and concat had two problems:

• indices and indptr were widened to int64, whatever the source used.
• The source's compression was dropped. With an lzf input, data and indices were written uncompressed.

Outputs now keep the source's index dtypes and storage settings. concat widens a dtype only when the combined matrix would not fit in it, and checks against that dtype's own limit, so a narrow int16 indptr is widened rather than wrapped.

split read the whole matrix once per group. It ran a separate subset for each group. When groups are interleaved through the file, as samples usually are, each subset read nearly all of X.

• split now reads each matrix once and shares every block among all the outputs. It holds up to a quarter of the file-descriptor limit open at once, and makes one pass per batch beyond that.
• Dataframe columns are no longer read through an h5py fancy index, which cost 0.38 s per column per group on 50,000 cells.
• subset shares the new writer. It gathers rows in one vectorised step and skips blocks with nothing selected.

Measured on the ci tier (50k × 20k, lzf) on one laptop:

Case	0.6.0	0.6.1	anndata
split-by-sample	630.9 MB	1.6 s, 302.6 MB	2.3 s, 301.7 MB
subset-query	309.7 MB	1.0 s, 147.1 MB	1.4 s, 146.6 MB
concat-inner	1232 MB	3.0 s, 583 MB	2.2 s, 832 MB (concat_on_disk)

The tag's own run on the CI runner will be published to docs/BENCHMARKS.md.

Behaviour change

Row selections passed to the subset writers must be sorted and unique, and anything else raises ValueError. Every command already produced sorted selections, and h5py required the same.

Testing

A new complexity guard checks that split's reads of X do not grow with the number of groups; the old implementation fails it with 32× growth. Split output is checked against per-group subset output and against anndata slicing, across CSR, CSC and dense, h5ad and zarr, and both axes.

Full changes: #19

Benchmark vs anndata/scanpy

Results — 0.6.1

Tier	ci, 50,000 obs x 20,000 var, 5% dense CSR
Var-heavy shape	2,000 obs x 36,601 var
Page cache	dropped between runs
Address-space ceiling	12.9 GB per process
Repeats	3 (fastest shown)
Platform	Linux x86_64, Python 3.12.3
Versions	adata-cli 0.6.1, anndata 0.13.4, scanpy 1.12.4
Generated	2026-09-24T15:41:03Z

startup

What does each contender cost before it does any work?

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.56 s	69.4 MB	-	no change
anndata	1.52 s	132.8 MB	-	no change
scanpy	3.30 s	265.2 MB	-	no change

concat-inner

Concatenate two stores on the obs axis, inner join.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	12.26 s	177.2 MB	583.3 MB	time +108%
anndata (concat_on_disk)	6.18 s	451.0 MB	831.9 MB	no change
anndata (in memory)	12.26 s	1.8 GB	580.3 MB	no change
scanpy	n/a	n/a	n/a	-
	scanpy has no concat of its own; it re-exports anndata's

concat-merge-same

Concatenate carrying var columns forward -- the REQ-71798 case.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.74 s	195.6 MB	45.7 MB	time +40%
anndata (concat_on_disk)	2.26 s	247.6 MB	63.2 MB	no change
anndata (in memory)	2.67 s	288.0 MB	44.6 MB	no change

subset-query

Keep the obs rows matching a predicate on an existing column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	4.10 s	212.0 MB	147.1 MB	time +26%, RSS -39%
anndata (backed)	4.94 s	626.5 MB	146.6 MB	no change
anndata (in memory)	5.21 s	984.4 MB	146.6 MB	no change
scanpy (filter_cells)	9.02 s	1.6 GB	291.2 MB	no change

h5ad-to-zarr

Convert a store to Zarr.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	7.81 s	232.9 MB	151.6 MB in 4717 files	time +29%
anndata (in memory)	6.15 s	814.9 MB	136.8 MB in 55 files	no change
scanpy	n/a	n/a	n/a	-
	no streaming converter; scanpy defers to anndata

inspect

Report what is in the store, without reading the matrix.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.58 s	71.4 MB	-	no change
anndata (read_elem on obs)	1.76 s	159.9 MB	-	no change
anndata (full load)	3.20 s	583.7 MB	-	no change

export-obs

Write the obs table out as CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.74 s	85.1 MB	1.8 MB	no change
anndata (read_elem on obs)	1.77 s	160.5 MB	1.8 MB	no change
anndata (full load)	3.28 s	583.9 MB	1.8 MB	no change

split-by-sample

Write one store per distinct value of an obs column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	6.65 s	198.4 MB	302.6 MB in 9 files	time -88%, RSS -74%
anndata (hand-written loop)	7.81 s	684.9 MB	301.7 MB in 8 files	no change

ls

List everything in the store.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.59 s	70.5 MB	-	no change
h5py (visit)	1.66 s	136.3 MB	-	no change
h5ls -r	n/a	n/a	n/a	-
	h5ls is not installed

create

Write an empty store with a given obs/var shape.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.64 s	84.5 MB	3.4 MB	no change
anndata	8.73 s	228.1 MB	49.2 MB	no change

export-array

Write a dense element out as .npy.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.60 s	82.6 MB	10.0 MB	no change
anndata (read_elem)	1.67 s	154.3 MB	10.0 MB	no change
anndata (full load)	3.27 s	583.8 MB	10.0 MB	no change

export-sparse

Write X out as Matrix Market text.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	25.52 s	76.3 MB	761.3 MB	no change
anndata (sparse_dataset)	7.12 s	968.7 MB	661.6 MB	no change
anndata (full load)	7.23 s	775.1 MB	661.6 MB	no change

import-dataframe

Replace obs from a CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.21 s	114.2 MB	296.0 MB	no change
anndata	6.88 s	596.5 MB	291.2 MB	no change

concat-outer

Concatenate two stores keeping the union of variables.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	12.20 s	172.0 MB	583.3 MB	time +106%
anndata (concat_on_disk)	6.26 s	450.5 MB	831.9 MB	no change
anndata (in memory)	12.18 s	1.8 GB	580.3 MB	no change

convert-dtype

Rewrite X as float32.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	5.57 s	90.2 MB	291.1 MB	no change
anndata (in memory)	6.91 s	983.8 MB	291.2 MB	no change
scanpy	n/a	n/a	n/a	-
	no dtype rewrite; scanpy defers to anndata

convert-layout

Transpose X from CSR to CSC.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli (streaming)	13.62 s	369.8 MB	1.2 GB	no change
adata-cli (--in-memory)	9.52 s	2.3 GB	297.0 MB	no change
anndata (in memory)	7.22 s	984.2 MB	297.0 MB	no change

rss-vs-size

Does peak memory track input size, at a fixed --chunk? This is the README's claim, stated as a measurement.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	12.13 s	173.8 MB	583.3 MB	time +105%
anndata (in memory)	12.18 s	1.8 GB	580.3 MB	no change

---

Rows marked n/a are operations the baseline does not offer; that is a result, not a gap in the measurement. Where a baseline is faster, the row stands as it is -- the trade this tool makes is memory for time, and hiding the cost would make the table useless.

Output size is the file as the filesystem reports it, which for HDF5 includes allocation slack. On a small tier that slack can be several times the stored bytes and the column says more about the writer's allocation strategy than about the data; at ci and above it is noise. Zarr stores report a file count alongside, because a directory of small chunks measures much larger than it holds.