Release 0.10.0 · cellgeni/adata-cli

Adds Parse Biosciences split-pipe output to adata convert, read in two
streaming passes like the MTX folders of 0.8.0. Features are now named by
gene id by default, for every format: ids are unique where symbols repeat,
so a gene-level matrix no longer needs --make-unique (a probe-level Flex
matrix still does).

Added

adata convert reads Parse Biosciences split-pipe output
(#30). A run folder, a
sample folder or a DGE_filtered/DGE_unfiltered folder is recognised
whatever its name:
- count_matrix.mtx (cells x genes) becomes X through the streaming MTX
  reader; integer counts stay integers;
- cell_metadata.csv becomes obs, indexed by bc_wells, each column
  typed as pandas reads it and stored as anndata writes it;
- all_genes.csv becomes var, with gene_symbols and genome;
- --sample NAME reads one sample, --all-samples writes one output per
  sample, and --unfiltered reads every barcode;
- --genome G keeps G's genes and the cells split-pipe called as G.
Checked against Parse's own scanpy recipe in CI, and on a real split-pipe
v1.6.1 run: X, obs and var all match, for 29,588 cells and for the
431,061 barcodes of the unfiltered folder, at a peak of about 200 MB.
A streaming CSV table reader (#55),
formats/table.py: two passes over the file, columns typed as
pd.read_csv types them, the obs dataframe written a block of rows at
a time. It reads Parse's cell metadata, and is there for #25's delimited
tables.

Changed

Features are named by gene id by default
(#54). This changes what
adata convert writes from 10x HDF5, MTX folders, CellBender and Parse:
var names are the gene ids, and the symbols are in var["gene_symbols"],
the layout scanpy.read_10x_mtx(var_names="gene_ids") gives. Symbols
repeat (Cell Ranger's GRCh38-2020-A reference repeats 10), so naming by
them needed --make-unique. --var-names gene_symbols gives the old
names, as scanpy and CellBender's loaders give them. A probe-level (Flex)
matrix still repeats gene ids, and still warns.

Fixed

Categories are ordered as anndata orders them
(#55). anndata sorts the
categories it makes from strings naturally (A2 before A10); convert
sorted them plainly, so a categorical var column holding numbers in its
values came out in a different order from scanpy's.

Benchmark vs anndata/scanpy

Results — `0.10.0`


Tier	`ci`, 50,000 obs x 20,000 var, 5% dense CSR
Var-heavy shape	2,000 obs x 36,601 var
Page cache	dropped between runs
Address-space ceiling	12.9 GB per process
Repeats	3 (fastest shown)
Platform	Linux x86_64, Python 3.12.3
Versions	adata-cli 0.10.0, anndata 0.13.4, scanpy 1.12.4
Generated	2026-10-19T10:48:49Z

`startup`

What does each contender cost before it does any work?

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.59 s	69.9 MB	-	no change
anndata	1.55 s	133.5 MB	-	no change
scanpy	3.26 s	260.0 MB	-	no change

`concat-inner`

Concatenate two stores on the obs axis, inner join.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	12.13 s	172.9 MB	583.3 MB	no change
anndata (concat_on_disk)	6.07 s	448.3 MB	831.9 MB	no change
anndata (in memory)	12.14 s	1.8 GB	580.3 MB	no change
scanpy	n/a	n/a	n/a	-
	scanpy has no concat of its own; it re-exports anndata's

`concat-merge-same`

Concatenate carrying var columns forward -- the REQ-71798 case.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.75 s	196.4 MB	45.7 MB	no change
anndata (concat_on_disk)	2.27 s	247.3 MB	63.2 MB	no change
anndata (in memory)	2.66 s	289.0 MB	44.6 MB	no change

`subset-query`

Keep the obs rows matching a predicate on an existing column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	4.12 s	227.8 MB	147.1 MB	no change
anndata (backed)	4.93 s	653.9 MB	146.6 MB	no change
anndata (in memory)	5.20 s	984.0 MB	146.6 MB	no change
scanpy (filter_cells)	8.82 s	1.6 GB	291.2 MB	no change

`h5ad-to-zarr`

Convert a store to Zarr.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	6.53 s	125.2 MB	151.6 MB in 4717 files	no change
anndata (in memory)	5.91 s	815.4 MB	136.8 MB in 55 files	no change
scanpy	n/a	n/a	n/a	-
	no streaming converter; scanpy defers to anndata

`inspect`

Report what is in the store, without reading the matrix.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.59 s	72.4 MB	-	no change
anndata (read_elem on obs)	1.71 s	160.5 MB	-	no change
anndata (full load)	3.22 s	584.3 MB	-	no change

`export-obs`

Write the obs table out as CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.76 s	85.7 MB	1.8 MB	no change
anndata (read_elem on obs)	1.72 s	160.8 MB	1.8 MB	no change
anndata (full load)	3.20 s	584.4 MB	1.8 MB	no change

`split-by-sample`

Write one store per distinct value of an obs column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	6.57 s	199.7 MB	302.6 MB in 9 files	no change
anndata (hand-written loop)	7.74 s	684.2 MB	301.7 MB in 8 files	no change

`ls`

List everything in the store.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.60 s	71.6 MB	-	no change
h5py (visit)	1.62 s	136.4 MB	-	no change
h5ls -r	n/a	n/a	n/a	-
	h5ls is not installed

`create`

Write an empty store with a given obs/var shape.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.65 s	85.0 MB	3.4 MB	no change
anndata	8.63 s	228.7 MB	49.2 MB	no change

`export-array`

Write a dense element out as .npy.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.63 s	83.6 MB	10.0 MB	no change
anndata (read_elem)	1.64 s	154.6 MB	10.0 MB	no change
anndata (full load)	3.13 s	584.3 MB	10.0 MB	no change

`export-sparse`

Write X out as Matrix Market text.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	10.88 s	257.1 MB	661.3 MB	no change
anndata (sparse_dataset)	7.05 s	968.2 MB	661.6 MB	no change
anndata (full load)	7.12 s	773.7 MB	661.6 MB	no change

`import-dataframe`

Replace obs from a CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.22 s	114.9 MB	296.0 MB	no change
anndata	6.92 s	600.5 MB	291.2 MB	no change

`import-10x-mtx`

Read a gzipped Cell Ranger 3 MTX folder of the same counts into an h5ad. adata-cli streams it in two passes and keeps integer counts; scanpy loads it and converts them to float32.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	15.74 s	345.3 MB	279.2 MB	no change
scanpy	13.65 s	3.3 GB	284.6 MB	no change

`concat-outer`

Concatenate two stores keeping the union of variables.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	12.12 s	189.1 MB	583.3 MB	no change
anndata (concat_on_disk)	6.13 s	450.4 MB	831.9 MB	no change
anndata (in memory)	12.10 s	1.8 GB	580.3 MB	no change

`recast-narrow`

Narrow X from float64 values with int64 indices to float32 and int32, as issue #13 asked. X is uncompressed, as write_h5ad leaves it by default, so the output keeps that and the baseline writes it the same way.

Contender	Wall time	Peak RSS	Input → output	vs previous
adata-cli	2.35 s	120.3 MB	815.6 MB → 415.4 MB (-49%)	no change
anndata (in memory)	3.92 s	1.8 GB	815.6 MB → 417.3 MB (-49%)	no change
scanpy	n/a	n/a	n/a	-
	no dtype rewrite; scanpy defers to anndata

`recast-narrow-lzf`

The same narrowing on an lzf-compressed X. Integer counts in float64 are mostly zero bytes, which lzf already removes, so expect a small saving on disk; the halving is in memory, once the matrix is loaded.

Contender	Wall time	Peak RSS	Input → output	vs previous
adata-cli	7.80 s	127.7 MB	302.5 MB → 291.1 MB (-4%)	no change
anndata (in memory)	8.04 s	1.8 GB	302.5 MB → 291.2 MB (-4%)	no change

`convert-layout`

Transpose X from CSR to CSC. The output should be the size of the input: 0.7.0 fixed a --layout that wrote it three times larger.

Contender	Wall time	Peak RSS	Input → output	vs previous
adata-cli (streaming)	14.67 s	385.3 MB	291.2 MB → 297.0 MB (+2%)	no change
adata-cli (--in-memory)	9.36 s	2.3 GB	291.2 MB → 297.0 MB (+2%)	no change
anndata (in memory)	7.14 s	984.6 MB	291.2 MB → 297.0 MB (+2%)	no change

`rss-vs-size`

Does peak memory track input size, at a fixed --chunk? This is the README's claim, stated as a measurement.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	12.10 s	181.6 MB	583.3 MB	no change
anndata (in memory)	12.18 s	1.8 GB	580.3 MB	no change

Rows marked n/a are operations the baseline does not offer; that is a result, not a gap in the measurement. Where a baseline is faster, the row stands as it is -- the trade this tool makes is memory for time, and hiding the cost would make the table useless.

Output size is the file as the filesystem reports it, which for HDF5 includes allocation slack. On a small tier that slack can be several times the stored bytes and the column says more about the writer's allocation strategy than about the data; at ci and above it is noise. Zarr stores report a file count alongside, because a directory of small chunks measures much larger than it holds. Where a case exists to change a file's size, the column reads input → output, with the change.

Navigation Menu

Repository navigation

0.10.0

Choose a tag to compare

Sorry, something went wrong.

Sorry, something went wrong.

Uh oh!

No results found

Added

Changed

Fixed

Results — `0.10.0`

`startup`

`concat-inner`

`concat-merge-same`

`subset-query`

`h5ad-to-zarr`

`inspect`

`export-obs`

`split-by-sample`

`ls`

`create`

`export-array`

`export-sparse`

`import-dataframe`

`import-10x-mtx`

`concat-outer`

`recast-narrow`

`recast-narrow-lzf`

`convert-layout`

`rss-vs-size`

Uh oh!

0.10.0

Added

Changed

Fixed

Results — 0.10.0

startup

concat-inner

concat-merge-same

subset-query

h5ad-to-zarr

inspect

export-obs

split-by-sample

ls

create

export-array

export-sparse

import-dataframe

import-10x-mtx

concat-outer

recast-narrow

recast-narrow-lzf

convert-layout

rss-vs-size

Uh oh!

Results — `0.10.0`

`startup`

`concat-inner`

`concat-merge-same`

`subset-query`

`h5ad-to-zarr`

`inspect`

`export-obs`

`split-by-sample`

`ls`

`create`

`export-array`

`export-sparse`

`import-dataframe`

`import-10x-mtx`

`concat-outer`

`recast-narrow`

`recast-narrow-lzf`

`convert-layout`

`rss-vs-size`