Release 0.11.0 · cellgeni/adata-cli

Adds loom to adata convert, both ways. A velocyto, loompy or
kallisto|bustools loom converts a slab of cells at a time, with the
matrices sparse, their dtypes kept, and the graphs and global attributes
that anndata.read_loom drops; an AnnData exports to a loom that loompy
validates.

Added

adata convert reads looms (#26),
spec 2.0.1 and 3.0.0, as anndata.read_loom reads them:
- X is the spliced layer when there is one, with the main matrix kept as
  layers["matrix"]; --layer picks another;
- attributes become obs and var columns, or obsm and varm when they have
  several columns; graphs become obsp and varp, and global attributes
  uns["loom"];
- genes are named by id (Accession, or kallisto|bustools' gene_id),
  the symbols kept as a column, unless --var-names gene_symbols;
  --obs-names-attr and --var-names-attr name any other attribute;
- X and the layers are CSR, or dense with --dense, and keep their dtype:
  velocyto's integer counts stay integers.
On a public 10k-cell velocyto loom this took 26 s at a 159 MB peak, where
anndata.read_loom took 29 s and 9.9 GB, and gave the same matrices,
names and columns. SCope's compound attributes are left out with a
warning, for #27.
adata convert writes looms (#26):
spec 3.0.0 as loompy 3 writes it, which loompy's validator accepts. X (or
--layer) is the main matrix, with every layer, obs and var as attributes
(CellID, Gene, Accession), 2-D obsm and varm, obsp and varp as
graphs, and uns scalars as global attributes. A loom's matrices are dense:
the summary gives their size before compression and warns above 10 GB.

Benchmark vs anndata/scanpy

Results — `0.11.0`


Tier	`ci`, 50,000 obs x 20,000 var, 5% dense CSR
Var-heavy shape	2,000 obs x 36,601 var
Page cache	dropped between runs
Address-space ceiling	12.9 GB per process
Repeats	3 (fastest shown)
Platform	Linux x86_64, Python 3.12.3
Versions	adata-cli 0.11.0, anndata 0.13.4, scanpy 1.12.4
Generated	2026-10-19T17:02:30Z

`startup`

What does each contender cost before it does any work?

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.63 s	70.4 MB	-	time +11%
anndata	1.73 s	133.2 MB	-	time +16%
scanpy	3.46 s	260.7 MB	-	no change

`concat-inner`

Concatenate two stores on the obs axis, inner join.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	12.24 s	176.9 MB	583.3 MB	no change
anndata (concat_on_disk)	6.31 s	447.1 MB	831.9 MB	no change
anndata (in memory)	12.25 s	1.8 GB	580.3 MB	no change
scanpy	n/a	n/a	n/a	-
	scanpy has no concat of its own; it re-exports anndata's

`concat-merge-same`

Concatenate carrying var columns forward -- the REQ-71798 case.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.82 s	210.9 MB	45.7 MB	no change
anndata (concat_on_disk)	2.35 s	245.5 MB	63.2 MB	no change
anndata (in memory)	2.86 s	289.1 MB	44.6 MB	no change

`subset-query`

Keep the obs rows matching a predicate on an existing column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	4.18 s	212.5 MB	147.1 MB	no change
anndata (backed)	5.18 s	655.2 MB	146.6 MB	no change
anndata (in memory)	5.43 s	984.8 MB	146.6 MB	no change
scanpy (filter_cells)	9.26 s	1.6 GB	291.2 MB	no change

`h5ad-to-zarr`

Convert a store to Zarr.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	6.52 s	125.4 MB	151.6 MB in 4717 files	no change
anndata (in memory)	6.05 s	813.5 MB	136.8 MB in 55 files	no change
scanpy	n/a	n/a	n/a	-
	no streaming converter; scanpy defers to anndata

`inspect`

Report what is in the store, without reading the matrix.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.65 s	72.6 MB	-	no change
anndata (read_elem on obs)	1.81 s	160.7 MB	-	no change
anndata (full load)	3.27 s	584.4 MB	-	no change

`export-obs`

Write the obs table out as CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.82 s	86.9 MB	1.8 MB	no change
anndata (read_elem on obs)	1.90 s	160.4 MB	1.8 MB	time +11%
anndata (full load)	3.39 s	584.2 MB	1.8 MB	no change

`split-by-sample`

Write one store per distinct value of an obs column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	6.64 s	199.5 MB	302.6 MB in 9 files	no change
anndata (hand-written loop)	7.92 s	685.9 MB	301.7 MB in 8 files	no change

`ls`

List everything in the store.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.65 s	71.6 MB	-	no change
h5py (visit)	1.74 s	136.5 MB	-	no change
h5ls -r	n/a	n/a	n/a	-
	h5ls is not installed

`create`

Write an empty store with a given obs/var shape.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.70 s	85.3 MB	3.4 MB	no change
anndata	8.77 s	225.4 MB	49.2 MB	no change

`export-array`

Write a dense element out as .npy.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.67 s	84.0 MB	10.0 MB	no change
anndata (read_elem)	1.76 s	154.5 MB	10.0 MB	no change
anndata (full load)	3.32 s	584.4 MB	10.0 MB	no change

`export-sparse`

Write X out as Matrix Market text.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	10.96 s	253.8 MB	661.3 MB	no change
anndata (sparse_dataset)	7.17 s	968.6 MB	661.6 MB	no change
anndata (full load)	7.34 s	773.7 MB	661.6 MB	no change

`import-dataframe`

Replace obs from a CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.29 s	115.2 MB	296.0 MB	no change
anndata	7.01 s	596.8 MB	291.2 MB	no change

`import-10x-mtx`

Read a gzipped Cell Ranger 3 MTX folder of the same counts into an h5ad. adata-cli streams it in two passes and keeps integer counts; scanpy loads it and converts them to float32.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	15.93 s	356.8 MB	279.2 MB	no change
scanpy	14.09 s	3.3 GB	284.6 MB	no change

`concat-outer`

Concatenate two stores keeping the union of variables.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	12.24 s	173.6 MB	583.3 MB	no change
anndata (concat_on_disk)	6.32 s	450.7 MB	831.9 MB	no change
anndata (in memory)	12.30 s	1.8 GB	580.3 MB	no change

`recast-narrow`

Narrow X from float64 values with int64 indices to float32 and int32, as issue #13 asked. X is uncompressed, as write_h5ad leaves it by default, so the output keeps that and the baseline writes it the same way.

Contender	Wall time	Peak RSS	Input → output	vs previous
adata-cli	2.42 s	120.5 MB	815.6 MB → 415.4 MB (-49%)	no change
anndata (in memory)	4.05 s	1.8 GB	815.6 MB → 417.3 MB (-49%)	no change
scanpy	n/a	n/a	n/a	-
	no dtype rewrite; scanpy defers to anndata

`recast-narrow-lzf`

The same narrowing on an lzf-compressed X. Integer counts in float64 are mostly zero bytes, which lzf already removes, so expect a small saving on disk; the halving is in memory, once the matrix is loaded.

Contender	Wall time	Peak RSS	Input → output	vs previous
adata-cli	7.89 s	128.2 MB	302.5 MB → 291.1 MB (-4%)	no change
anndata (in memory)	8.14 s	1.8 GB	302.5 MB → 291.2 MB (-4%)	no change

`convert-layout`

Transpose X from CSR to CSC. The output should be the size of the input: 0.7.0 fixed a --layout that wrote it three times larger.

Contender	Wall time	Peak RSS	Input → output	vs previous
adata-cli (streaming)	14.94 s	368.9 MB	291.2 MB → 297.0 MB (+2%)	no change
adata-cli (--in-memory)	9.58 s	2.3 GB	291.2 MB → 297.0 MB (+2%)	no change
anndata (in memory)	7.27 s	983.4 MB	291.2 MB → 297.0 MB (+2%)	no change

`rss-vs-size`

Does peak memory track input size, at a fixed --chunk? This is the README's claim, stated as a measurement.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	12.15 s	178.0 MB	583.3 MB	no change
anndata (in memory)	12.23 s	1.8 GB	580.3 MB	no change

Rows marked n/a are operations the baseline does not offer; that is a result, not a gap in the measurement. Where a baseline is faster, the row stands as it is -- the trade this tool makes is memory for time, and hiding the cost would make the table useless.

Output size is the file as the filesystem reports it, which for HDF5 includes allocation slack. On a small tier that slack can be several times the stored bytes and the column says more about the writer's allocation strategy than about the data; at ci and above it is noise. Zarr stores report a file count alongside, because a directory of small chunks measures much larger than it holds. Where a case exists to change a file's size, the column reads input → output, with the change.

Navigation Menu

Repository navigation

0.11.0

Choose a tag to compare

Sorry, something went wrong.

Sorry, something went wrong.

Uh oh!

No results found

Added

Results — `0.11.0`

`startup`

`concat-inner`

`concat-merge-same`

`subset-query`

`h5ad-to-zarr`

`inspect`

`export-obs`

`split-by-sample`

`ls`

`create`

`export-array`

`export-sparse`

`import-dataframe`

`import-10x-mtx`

`concat-outer`

`recast-narrow`

`recast-narrow-lzf`

`convert-layout`

`rss-vs-size`

Uh oh!

0.11.0

Added

Results — 0.11.0

startup

concat-inner

concat-merge-same

subset-query

h5ad-to-zarr

inspect

export-obs

split-by-sample

ls

create

export-array

export-sparse

import-dataframe

import-10x-mtx

concat-outer

recast-narrow

recast-narrow-lzf

convert-layout

rss-vs-size

Uh oh!

Results — `0.11.0`

`startup`

`concat-inner`

`concat-merge-same`

`subset-query`

`h5ad-to-zarr`

`inspect`

`export-obs`

`split-by-sample`

`ls`

`create`

`export-array`

`export-sparse`

`import-dataframe`

`import-10x-mtx`

`concat-outer`

`recast-narrow`

`recast-narrow-lzf`

`convert-layout`

`rss-vs-size`