Release adata-cli 0.7.0 · cellgeni/adata-cli

This release adds format conversion. adata convert now changes a store's format: h5ad ↔ Zarr v2/v3, and 10x Genomics HDF5 matrices in and out. The matrix command that used to be convert is now recast.

Upgrading from 0.6

convert used to change a matrix's dtype, layout or density. That command is now recast, with the same flags and behaviour:

adata recast data.h5ad X -o small.h5ad --dtype float32

Your existing scripts keep working until 1.0.0. If you pass any of --dtype, --indices-dtype, --layout, --all or --inplace to convert, it runs recast as before and prints a deprecation warning.

Added

adata convert changes a store's format. The output's extension picks the format.

adata convert data.h5ad -o data.zarr
adata convert data.zarr -o data.h5ad --compression gzip
adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad
adata convert raw_feature_bc_matrix.h5 -o rna.h5ad --feature-type "Gene Expression"
adata convert counts.h5ad -o for_cellbender.h5 --layer counts

h5ad ↔ Zarr:
- Every element is copied as it is, streamed, so the store's size isn't a limit.
- Chunking and compression carry over where the target can express them. --compression overrides them.
- A Zarr output keeps the source's Zarr version unless --zarr-format says otherwise.
Reading 10x HDF5:
- Covers Cell Ranger's *_feature_bc_matrix.h5, v2 and v3+, and the same layout from Space Ranger, Xenium and CellBender.
- You get what scanpy.read_10x_h5(gex_only=False) gives, column for column. CI checks this against scanpy itself.
- Two differences from scanpy: X keeps the integer counts, and the run's metadata is kept in uns["cellranger"].
- --feature-type keeps one feature type. --split-feature-types writes one file per type, reading the matrix once. --genome, --var-names and --make-unique behave like scanpy's options of the same name.
Writing 10x HDF5:
- Writes the Cell Ranger 3+ layout from X, raw/X or any layer, for CellBender, SoupX, DropletUtils and Seurat.
- The output uses gzip and fixed-width strings, as Cell Ranger writes them; PyTables, which CellBender reads with, needs those.
- Converting a Cell Ranger file to h5ad and back reproduces it dataset for dataset.
Safety:
- Every check runs before anything is written.
- The output is written beside its destination and moved into place only once complete, so a failed conversion leaves nothing behind.
- An existing output is refused unless you pass --overwrite.
- A flag that doesn't apply to the conversion you asked for is refused rather than ignored.

Fixed

recast --layout (formerly convert --layout) left h5ad outputs up to three times too large.
- The streaming transpose kept its working data in the output file. HDF5 never gives back the space of deleted data, so it stayed there.
- On a 20,000 × 2,000 matrix at 5% density, the output was 55 MB where 17 MB was needed.
- Zarr outputs weren't affected.
Writing an h5ad that contains a None to Zarr failed.
- anndata stores None as an empty HDF5 dataset. Any None in uns made subset, split, import and recast fail with TypeError: ... not 'Empty' when the output was Zarr.
Zarr written to h5ad came out uncompressed. A compressed Zarr array now becomes gzip in the h5ad.
Empty arrays chunked larger than themselves couldn't be written to h5ad. This also broke recast on an all-zero matrix from a compressed h5ad.

Full changes: #32, #33, #34, #41

Benchmark vs anndata/scanpy

Results — `0.7.0`


Tier	`ci`, 50,000 obs x 20,000 var, 5% dense CSR
Var-heavy shape	2,000 obs x 36,601 var
Page cache	dropped between runs
Address-space ceiling	12.9 GB per process
Repeats	3 (fastest shown)
Platform	Linux x86_64, Python 3.12.3
Versions	adata-cli 0.7.0, anndata 0.13.4, scanpy 1.12.4
Generated	2026-10-15T12:48:08Z

`startup`

What does each contender cost before it does any work?

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.63 s	67.9 MB	-	time +13%
anndata	1.65 s	133.5 MB	-	no change
scanpy	3.57 s	260.3 MB	-	no change

`concat-inner`

Concatenate two stores on the obs axis, inner join.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	9.29 s	170.3 MB	583.3 MB	time -24%
anndata (concat_on_disk)	5.94 s	447.3 MB	831.9 MB	no change
anndata (in memory)	9.09 s	1.8 GB	580.3 MB	time -26%
scanpy	n/a	n/a	n/a	-
	scanpy has no concat of its own; it re-exports anndata's

`concat-merge-same`

Concatenate carrying var columns forward -- the REQ-71798 case.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.50 s	193.1 MB	45.7 MB	time -14%
anndata (concat_on_disk)	2.24 s	246.3 MB	63.2 MB	no change
anndata (in memory)	2.42 s	288.9 MB	44.6 MB	no change

`subset-query`

Keep the obs rows matching a predicate on an existing column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	3.35 s	209.5 MB	147.1 MB	time -18%
anndata (backed)	4.26 s	683.4 MB	146.6 MB	time -14%
anndata (in memory)	4.49 s	985.2 MB	146.6 MB	time -14%
scanpy (filter_cells)	7.69 s	1.6 GB	291.2 MB	time -15%

`h5ad-to-zarr`

Convert a store to Zarr.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	6.14 s	154.2 MB	151.6 MB in 4717 files	time -21%, RSS -34%
anndata (in memory)	5.70 s	778.4 MB	136.8 MB in 55 files	no change
scanpy	n/a	n/a	n/a	-
	no streaming converter; scanpy defers to anndata

`inspect`

Report what is in the store, without reading the matrix.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.61 s	69.8 MB	-	no change
anndata (read_elem on obs)	1.76 s	160.3 MB	-	no change
anndata (full load)	3.19 s	584.3 MB	-	no change

`export-obs`

Write the obs table out as CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.77 s	83.3 MB	1.8 MB	no change
anndata (read_elem on obs)	1.78 s	161.1 MB	1.8 MB	no change
anndata (full load)	3.22 s	584.4 MB	1.8 MB	no change

`split-by-sample`

Write one store per distinct value of an obs column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	4.99 s	196.5 MB	302.6 MB in 9 files	time -25%
anndata (hand-written loop)	6.34 s	684.4 MB	301.7 MB in 8 files	time -19%

`ls`

List everything in the store.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.63 s	68.9 MB	-	no change
h5py (visit)	1.67 s	136.4 MB	-	no change
h5ls -r	n/a	n/a	n/a	-
	h5ls is not installed

`create`

Write an empty store with a given obs/var shape.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.66 s	82.5 MB	3.4 MB	no change
anndata	9.00 s	229.1 MB	49.2 MB	no change

`export-array`

Write a dense element out as .npy.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.66 s	81.3 MB	10.0 MB	time +10%
anndata (read_elem)	1.68 s	154.6 MB	10.0 MB	no change
anndata (full load)	3.22 s	584.3 MB	10.0 MB	no change

`export-sparse`

Write X out as Matrix Market text.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	21.22 s	74.7 MB	761.3 MB	time -17%
anndata (sparse_dataset)	6.53 s	968.5 MB	661.6 MB	no change
anndata (full load)	6.72 s	774.4 MB	661.6 MB	no change

`import-dataframe`

Replace obs from a CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.96 s	112.4 MB	296.0 MB	time +63%
anndata	5.36 s	600.5 MB	291.2 MB	time -22%

`concat-outer`

Concatenate two stores keeping the union of variables.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	9.15 s	175.6 MB	583.3 MB	time -25%
anndata (concat_on_disk)	5.93 s	449.6 MB	831.9 MB	no change
anndata (in memory)	9.07 s	1.8 GB	580.3 MB	time -26%

`convert-dtype`

Rewrite X as float32.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	4.02 s	88.2 MB	291.1 MB	time -28%
anndata (in memory)	5.35 s	984.0 MB	291.2 MB	time -23%
scanpy	n/a	n/a	n/a	-
	no dtype rewrite; scanpy defers to anndata

`convert-layout`

Transpose X from CSR to CSC.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli (streaming)	18.18 s	362.2 MB	297.0 MB	time +33%
adata-cli (--in-memory)	8.85 s	2.3 GB	297.0 MB	no change
anndata (in memory)	6.04 s	984.7 MB	297.0 MB	time -16%

`rss-vs-size`

Does peak memory track input size, at a fixed --chunk? This is the README's claim, stated as a measurement.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	9.19 s	180.6 MB	583.3 MB	time -24%
anndata (in memory)	8.99 s	1.8 GB	580.3 MB	time -26%

Rows marked n/a are operations the baseline does not offer; that is a result, not a gap in the measurement. Where a baseline is faster, the row stands as it is -- the trade this tool makes is memory for time, and hiding the cost would make the table useless.

Output size is the file as the filesystem reports it, which for HDF5 includes allocation slack. On a small tier that slack can be several times the stored bytes and the column says more about the writer's allocation strategy than about the data; at ci and above it is noise. Zarr stores report a file count alongside, because a directory of small chunks measures much larger than it holds.

Navigation Menu

Repository navigation

adata-cli 0.7.0

Choose a tag to compare

Sorry, something went wrong.

Sorry, something went wrong.

Uh oh!

No results found

Upgrading from 0.6

Added

Fixed

Results — `0.7.0`

`startup`

`concat-inner`

`concat-merge-same`

`subset-query`

`h5ad-to-zarr`

`inspect`

`export-obs`

`split-by-sample`

`ls`

`create`

`export-array`

`export-sparse`

`import-dataframe`

`concat-outer`

`convert-dtype`

`convert-layout`

`rss-vs-size`

Uh oh!

adata-cli 0.7.0

Upgrading from 0.6

Added

Fixed

Results — 0.7.0

startup

concat-inner

concat-merge-same

subset-query

h5ad-to-zarr

inspect

export-obs

split-by-sample

ls

create

export-array

export-sparse

import-dataframe

concat-outer

convert-dtype

convert-layout

rss-vs-size

Uh oh!

Results — `0.7.0`

`startup`

`concat-inner`

`concat-merge-same`

`subset-query`

`h5ad-to-zarr`

`inspect`

`export-obs`

`split-by-sample`

`ls`

`create`

`export-array`

`export-sparse`

`import-dataframe`

`concat-outer`

`convert-dtype`

`convert-layout`

`rss-vs-size`