adata-cli 0.7.1

A patch release that fixes several ways subset, recast, split and concat could lose or corrupt data while reporting success. Upgrading is recommended if you use subset on Zarr stores or --inplace.

pip install -U pyadata-cli

Fixed

• subset x.zarr -o x.zarr emptied the store (#35). An output that names the input is now refused; use --inplace to replace a store.
• A failed selection left an empty output (#35). It also truncated an existing -o target. The selection is now resolved before anything is written.
• --inplace deleted the source before its replacement was in place (#35). This affected subset and recast: an interrupt at the wrong moment lost the data. The source is now renamed aside and deleted only afterwards.
• A failure part-way through left a partial store that looked finished (#35). This affected subset, recast, split and concat.
  • All four now write beside their output (out.adata-tmp.zarr) and move it into place only once it's complete, as convert already did.
  • split writes its manifest last.
  • If a run is killed outright, the next run refuses to start until the leftover *.adata-tmp* or *.adata-old* is dealt with, rather than deleting what may be the only copy.
• concat wrote Zarr v3 whatever its inputs' version (#36). It now keeps the version its Zarr inputs share. --zarr-format still overrides it.
• subset and concat now check --zarr-format themselves (#37), and six --help examples that still said h5ad are corrected (#38). Both contributed by @iamcipherdev, who opened their first pull requests here: thank you.

Benchmarks

The recast benchmark now narrows a float64/int64 store, and reports file size before and after. Narrowing halves an uncompressed X. On an lzf- or gzip-compressed X it saves only a few percent on disk, because the compressor had already removed the zero bytes; the halving shows up in memory once the matrix is loaded.

Full changelog: CHANGELOG.md

Benchmark vs anndata/scanpy

Results — 0.7.1

Tier	ci, 50,000 obs x 20,000 var, 5% dense CSR
Var-heavy shape	2,000 obs x 36,601 var
Page cache	dropped between runs
Address-space ceiling	12.9 GB per process
Repeats	3 (fastest shown)
Platform	Linux x86_64, Python 3.12.3
Versions	adata-cli 0.7.1, anndata 0.13.4, scanpy 1.12.4
Generated	2026-10-15T14:55:32Z

startup

What does each contender cost before it does any work?

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.30 s	76.1 MB	-	time +108%, RSS +12%
anndata	3.65 s	133.4 MB	-	time +121%
scanpy	5.81 s	260.4 MB	-	time +62%

concat-inner

Concatenate two stores on the obs axis, inner join.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	15.37 s	172.4 MB	583.3 MB	time +65%
anndata (concat_on_disk)	8.35 s	449.9 MB	831.9 MB	time +41%
anndata (in memory)	10.20 s	1.8 GB	580.3 MB	time +12%
scanpy	n/a	n/a	n/a	-
	scanpy has no concat of its own; it re-exports anndata's

concat-merge-same

Concatenate carrying var columns forward -- the REQ-71798 case.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.83 s	193.5 MB	45.7 MB	time +22%
anndata (concat_on_disk)	3.83 s	246.2 MB	63.2 MB	time +71%
anndata (in memory)	3.90 s	289.0 MB	44.6 MB	time +61%

subset-query

Keep the obs rows matching a predicate on an existing column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	5.30 s	193.4 MB	147.1 MB	time +58%
anndata (backed)	6.98 s	674.9 MB	146.6 MB	time +64%
anndata (in memory)	7.77 s	985.1 MB	146.6 MB	time +73%
scanpy (filter_cells)	11.88 s	1.6 GB	291.2 MB	time +54%

h5ad-to-zarr

Convert a store to Zarr.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	10.21 s	121.9 MB	151.6 MB in 4717 files	time +66%, RSS -21%
anndata (in memory)	7.97 s	775.5 MB	136.8 MB in 55 files	time +40%
scanpy	n/a	n/a	n/a	-
	no streaming converter; scanpy defers to anndata

inspect

Report what is in the store, without reading the matrix.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.21 s	69.7 MB	-	time +98%
anndata (read_elem on obs)	3.72 s	160.7 MB	-	time +111%
anndata (full load)	4.68 s	584.0 MB	-	time +47%

export-obs

Write the obs table out as CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.66 s	83.2 MB	1.8 MB	time +116%
anndata (read_elem on obs)	3.95 s	161.1 MB	1.8 MB	time +122%
anndata (full load)	9.18 s	584.4 MB	1.8 MB	time +185%

split-by-sample

Write one store per distinct value of an obs column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	7.63 s	197.3 MB	302.6 MB in 9 files	time +53%
anndata (hand-written loop)	6.32 s	684.6 MB	301.7 MB in 8 files	no change

ls

List everything in the store.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.86 s	68.9 MB	-	time +37%
h5py (visit)	1.81 s	136.5 MB	-	no change
h5ls -r	n/a	n/a	n/a	-
	h5ls is not installed

create

Write an empty store with a given obs/var shape.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.32 s	82.5 MB	3.4 MB	time +101%
anndata	8.44 s	226.1 MB	49.2 MB	no change

export-array

Write a dense element out as .npy.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.33 s	81.3 MB	10.0 MB	time +101%
anndata (read_elem)	4.05 s	154.8 MB	10.0 MB	time +142%
anndata (full load)	5.90 s	584.4 MB	10.0 MB	time +83%

export-sparse

Write X out as Matrix Market text.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	35.18 s	74.6 MB	761.3 MB	time +66%
anndata (sparse_dataset)	12.00 s	968.5 MB	661.6 MB	time +84%
anndata (full load)	14.57 s	773.4 MB	661.6 MB	time +117%

import-dataframe

Replace obs from a CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	2.27 s	112.3 MB	296.0 MB	time +16%
anndata	6.91 s	597.0 MB	291.2 MB	time +29%

concat-outer

Concatenate two stores keeping the union of variables.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	16.37 s	175.3 MB	583.3 MB	time +79%
anndata (concat_on_disk)	12.00 s	448.9 MB	831.9 MB	time +102%
anndata (in memory)	16.64 s	1.8 GB	580.3 MB	time +83%

recast-narrow

Narrow X from float64 values with int64 indices to float32 and int32, as issue #13 asked. X is uncompressed, as write_h5ad leaves it by default, so the output keeps that and the baseline writes it the same way.

Contender	Wall time	Peak RSS	Input → output	vs previous
adata-cli	6.00 s	117.7 MB	815.6 MB → 415.4 MB (-49%)	-
anndata (in memory)	14.61 s	1.8 GB	815.6 MB → 417.3 MB (-49%)	-
scanpy	n/a	n/a	n/a	-
	no dtype rewrite; scanpy defers to anndata

recast-narrow-lzf

The same narrowing on an lzf-compressed X. Integer counts in float64 are mostly zero bytes, which lzf already removes, so expect a small saving on disk; the halving is in memory, once the matrix is loaded.

Contender	Wall time	Peak RSS	Input → output	vs previous
adata-cli	10.85 s	125.5 MB	302.5 MB → 291.1 MB (-4%)	-
anndata (in memory)	9.40 s	1.8 GB	302.5 MB → 291.2 MB (-4%)	-

convert-layout

Transpose X from CSR to CSC. The output should be the size of the input: 0.7.0 fixed a --layout that wrote it three times larger.

Contender	Wall time	Peak RSS	Input → output	vs previous
adata-cli (streaming)	19.18 s	357.8 MB	291.2 MB → 297.0 MB (+2%)	no change
adata-cli (--in-memory)	10.51 s	2.3 GB	291.2 MB → 297.0 MB (+2%)	time +19%
anndata (in memory)	7.69 s	984.4 MB	291.2 MB → 297.0 MB (+2%)	time +27%

rss-vs-size

Does peak memory track input size, at a fixed --chunk? This is the README's claim, stated as a measurement.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	16.12 s	174.3 MB	583.3 MB	time +75%
anndata (in memory)	12.82 s	1.8 GB	580.3 MB	time +43%

---

Rows marked n/a are operations the baseline does not offer; that is a result, not a gap in the measurement. Where a baseline is faster, the row stands as it is -- the trade this tool makes is memory for time, and hiding the cost would make the table useless.

Output size is the file as the filesystem reports it, which for HDF5 includes allocation slack. On a small tier that slack can be several times the stored bytes and the column says more about the write...

adata-cli 0.7.0

This release adds format conversion. adata convert now changes a store's format: h5ad ↔ Zarr v2/v3, and 10x Genomics HDF5 matrices in and out. The matrix command that used to be convert is now recast.

Upgrading from 0.6

convert used to change a matrix's dtype, layout or density. That command is now recast, with the same flags and behaviour:

adata recast data.h5ad X -o small.h5ad --dtype float32

Your existing scripts keep working until 1.0.0. If you pass any of --dtype, --indices-dtype, --layout, --all or --inplace to convert, it runs recast as before and prints a deprecation warning.

Added

adata convert changes a store's format. The output's extension picks the format.

adata convert data.h5ad -o data.zarr
adata convert data.zarr -o data.h5ad --compression gzip
adata convert filtered_feature_bc_matrix.h5 -o counts.h5ad
adata convert raw_feature_bc_matrix.h5 -o rna.h5ad --feature-type "Gene Expression"
adata convert counts.h5ad -o for_cellbender.h5 --layer counts

• h5ad ↔ Zarr:
  • Every element is copied as it is, streamed, so the store's size isn't a limit.
  • Chunking and compression carry over where the target can express them. --compression overrides them.
  • A Zarr output keeps the source's Zarr version unless --zarr-format says otherwise.
• Reading 10x HDF5:
  • Covers Cell Ranger's *_feature_bc_matrix.h5, v2 and v3+, and the same layout from Space Ranger, Xenium and CellBender.
  • You get what scanpy.read_10x_h5(gex_only=False) gives, column for column. CI checks this against scanpy itself.
  • Two differences from scanpy: X keeps the integer counts, and the run's metadata is kept in uns["cellranger"].
  • --feature-type keeps one feature type. --split-feature-types writes one file per type, reading the matrix once. --genome, --var-names and --make-unique behave like scanpy's options of the same name.
• Writing 10x HDF5:
  • Writes the Cell Ranger 3+ layout from X, raw/X or any layer, for CellBender, SoupX, DropletUtils and Seurat.
  • The output uses gzip and fixed-width strings, as Cell Ranger writes them; PyTables, which CellBender reads with, needs those.
  • Converting a Cell Ranger file to h5ad and back reproduces it dataset for dataset.
• Safety:
  • Every check runs before anything is written.
  • The output is written beside its destination and moved into place only once complete, so a failed conversion leaves nothing behind.
  • An existing output is refused unless you pass --overwrite.
  • A flag that doesn't apply to the conversion you asked for is refused rather than ignored.

Fixed

• recast --layout (formerly convert --layout) left h5ad outputs up to three times too large.
  • The streaming transpose kept its working data in the output file. HDF5 never gives back the space of deleted data, so it stayed there.
  • On a 20,000 × 2,000 matrix at 5% density, the output was 55 MB where 17 MB was needed.
  • Zarr outputs weren't affected.
• Writing an h5ad that contains a None to Zarr failed.
  • anndata stores None as an empty HDF5 dataset. Any None in uns made subset, split, import and recast fail with TypeError: ... not 'Empty' when the output was Zarr.
• Zarr written to h5ad came out uncompressed. A compressed Zarr array now becomes gzip in the h5ad.
• Empty arrays chunked larger than themselves couldn't be written to h5ad. This also broke recast on an all-zero matrix from a compressed h5ad.

Full changes: #32, #33, #34, #41

Benchmark vs anndata/scanpy

Results — 0.7.0

Tier	ci, 50,000 obs x 20,000 var, 5% dense CSR
Var-heavy shape	2,000 obs x 36,601 var
Page cache	dropped between runs
Address-space ceiling	12.9 GB per process
Repeats	3 (fastest shown)
Platform	Linux x86_64, Python 3.12.3
Versions	adata-cli 0.7.0, anndata 0.13.4, scanpy 1.12.4
Generated	2026-10-15T12:48:08Z

startup

What does each contender cost before it does any work?

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.63 s	67.9 MB	-	time +13%
anndata	1.65 s	133.5 MB	-	no change
scanpy	3.57 s	260.3 MB	-	no change

concat-inner

Concatenate two stores on the obs axis, inner join.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	9.29 s	170.3 MB	583.3 MB	time -24%
anndata (concat_on_disk)	5.94 s	447.3 MB	831.9 MB	no change
anndata (in memory)	9.09 s	1.8 GB	580.3 MB	time -26%
scanpy	n/a	n/a	n/a	-
	scanpy has no concat of its own; it re-exports anndata's

concat-merge-same

Concatenate carrying var columns forward -- the REQ-71798 case.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.50 s	193.1 MB	45.7 MB	time -14%
anndata (concat_on_disk)	2.24 s	246.3 MB	63.2 MB	no change
anndata (in memory)	2.42 s	288.9 MB	44.6 MB	no change

subset-query

Keep the obs rows matching a predicate on an existing column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	3.35 s	209.5 MB	147.1 MB	time -18%
anndata (backed)	4.26 s	683.4 MB	146.6 MB	time -14%
anndata (in memory)	4.49 s	985.2 MB	146.6 MB	time -14%
scanpy (filter_cells)	7.69 s	1.6 GB	291.2 MB	time -15%

h5ad-to-zarr

Convert a store to Zarr.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	6.14 s	154.2 MB	151.6 MB in 4717 files	time -21%, RSS -34%
anndata (in memory)	5.70 s	778.4 MB	136.8 MB in 55 files	no change
scanpy	n/a	n/a	n/a	-
	no streaming converter; scanpy defers to anndata

inspect

Report what is in the store, without reading the matrix.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.61 s	69.8 MB	-	no change
anndata (read_elem on obs)	1.76 s	160.3 MB	-	no change
anndata (full load)	3.19 s	584.3 MB	-	no change

export-obs

Write the obs table out as CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.77 s	83.3 MB	1.8 MB	no change
anndata (read_elem on obs)	1.78 s	161.1 MB	1.8 MB	no change
anndata (full load)	3.22 s	584.4 MB	1.8 MB	no change

split-by-sample

Write one store per distinct value of an obs column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	4.99 s	196.5 MB	302.6 MB in 9 files	time -25%
anndata (hand-written loop)	6.34 s	684.4 MB	301.7 MB in 8 files	time -19%

ls

List everything in the store.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.63 s	68.9 MB	-	no change
h5py (visit)	1.67 s	136.4 MB	-	no change
h5ls -r	n/a	n/a	n/a	-
	h5ls is not installed

create

Write an empty store with a given obs/var shape.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.66 s	82.5 MB	3.4 MB	no change
anndata	9.00 s	229.1 MB	49.2 MB	no change

export-array

Write a dense element out as .npy.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.66 s	81.3 MB	10.0 MB	time +10%
anndata (read_elem)	1.68 s	154.6 MB	10.0 MB	no change
anndata (full load)	3.22 s	584.3 MB	10.0 MB	no change

export-sparse

Write X out as Matrix Market text.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	21.22 s	74.7 MB	761.3 MB	time -17%
anndata (sparse_dataset)	6.53 s	968.5 MB	661.6 MB	no change
anndata (full load)	6.72 s	774.4 MB	661.6 MB	no change

import-dataframe

Replace obs from a CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.96 s	112.4 MB	296.0 MB	time +63%
anndata	5.36 s	600.5 MB	291.2 MB	time -22%

concat-outer

Concatenate two stores keeping the union of variables.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	9.15 s	175.6 MB	583.3 MB	time -25%
anndata (concat_on_disk)	5.93 s	449.6 MB	831.9 MB	no change
anndata (in memory)	9.07 s	1.8 GB	580.3 MB	time -26%

convert-dtype

Rewrite X as float32.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	4.02 s	88.2 MB	291.1 MB	time -28%
anndata (in memory)	5.35 s	984.0 MB	291.2 MB	time -23%
scanpy	n/a	n/a	n/a	-
	no dtype rewrite; scanpy defers to anndata

convert-layout

Transpose X from CSR to CSC.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli (streaming)	18.18 s	362.2 MB	297.0 MB	time +33%
adata-cli (--in-memory)	8.85 s	2.3 GB	297.0 MB	no change
anndata (in memory)	6.04 s	984.7 MB	297.0 MB	time -16%

rss-vs-size

Does peak memory track input size, at a fixed --chunk? This is the README's claim, stated as a measurement.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	9.19 s	180.6 MB	583.3 MB	time -24%
anndata (in memory)	8.99 s	1.8 GB	580.3 MB	time -26%

---

Rows marked n/a are operations the baseline does not offer; that is a result, not a gap in the measurement. Where a baseline is faster, the row stands as it is -- the trade this tool makes is memory for time, and hiding ...

adata-cli 0.6.1

Makes the outputs of subset, split and concat the size anndata writes; they were about twice that. Also makes split read each matrix once rather than once per group. In 0.6.0 a hand-written anndata loop split the benchmark 7× faster than split did, and split is now the faster of the two.

Fixed

Sparse outputs were about twice as large as they needed to be. Every sparse output of subset, split and concat had two problems:

• indices and indptr were widened to int64, whatever the source used.
• The source's compression was dropped. With an lzf input, data and indices were written uncompressed.

Outputs now keep the source's index dtypes and storage settings. concat widens a dtype only when the combined matrix would not fit in it, and checks against that dtype's own limit, so a narrow int16 indptr is widened rather than wrapped.

split read the whole matrix once per group. It ran a separate subset for each group. When groups are interleaved through the file, as samples usually are, each subset read nearly all of X.

• split now reads each matrix once and shares every block among all the outputs. It holds up to a quarter of the file-descriptor limit open at once, and makes one pass per batch beyond that.
• Dataframe columns are no longer read through an h5py fancy index, which cost 0.38 s per column per group on 50,000 cells.
• subset shares the new writer. It gathers rows in one vectorised step and skips blocks with nothing selected.

Measured on the ci tier (50k × 20k, lzf) on one laptop:

Case	0.6.0	0.6.1	anndata
split-by-sample	630.9 MB	1.6 s, 302.6 MB	2.3 s, 301.7 MB
subset-query	309.7 MB	1.0 s, 147.1 MB	1.4 s, 146.6 MB
concat-inner	1232 MB	3.0 s, 583 MB	2.2 s, 832 MB (concat_on_disk)

The tag's own run on the CI runner will be published to docs/BENCHMARKS.md.

Behaviour change

Row selections passed to the subset writers must be sorted and unique, and anything else raises ValueError. Every command already produced sorted selections, and h5py required the same.

Testing

A new complexity guard checks that split's reads of X do not grow with the number of groups; the old implementation fails it with 32× growth. Split output is checked against per-group subset output and against anndata slicing, across CSR, CSC and dense, h5ad and zarr, and both axes.

Full changes: #19

Benchmark vs anndata/scanpy

Results — 0.6.1

Tier	ci, 50,000 obs x 20,000 var, 5% dense CSR
Var-heavy shape	2,000 obs x 36,601 var
Page cache	dropped between runs
Address-space ceiling	12.9 GB per process
Repeats	3 (fastest shown)
Platform	Linux x86_64, Python 3.12.3
Versions	adata-cli 0.6.1, anndata 0.13.4, scanpy 1.12.4
Generated	2026-09-24T15:41:03Z

startup

What does each contender cost before it does any work?

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.56 s	69.4 MB	-	no change
anndata	1.52 s	132.8 MB	-	no change
scanpy	3.30 s	265.2 MB	-	no change

concat-inner

Concatenate two stores on the obs axis, inner join.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	12.26 s	177.2 MB	583.3 MB	time +108%
anndata (concat_on_disk)	6.18 s	451.0 MB	831.9 MB	no change
anndata (in memory)	12.26 s	1.8 GB	580.3 MB	no change
scanpy	n/a	n/a	n/a	-
	scanpy has no concat of its own; it re-exports anndata's

concat-merge-same

Concatenate carrying var columns forward -- the REQ-71798 case.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.74 s	195.6 MB	45.7 MB	time +40%
anndata (concat_on_disk)	2.26 s	247.6 MB	63.2 MB	no change
anndata (in memory)	2.67 s	288.0 MB	44.6 MB	no change

subset-query

Keep the obs rows matching a predicate on an existing column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	4.10 s	212.0 MB	147.1 MB	time +26%, RSS -39%
anndata (backed)	4.94 s	626.5 MB	146.6 MB	no change
anndata (in memory)	5.21 s	984.4 MB	146.6 MB	no change
scanpy (filter_cells)	9.02 s	1.6 GB	291.2 MB	no change

h5ad-to-zarr

Convert a store to Zarr.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	7.81 s	232.9 MB	151.6 MB in 4717 files	time +29%
anndata (in memory)	6.15 s	814.9 MB	136.8 MB in 55 files	no change
scanpy	n/a	n/a	n/a	-
	no streaming converter; scanpy defers to anndata

inspect

Report what is in the store, without reading the matrix.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.58 s	71.4 MB	-	no change
anndata (read_elem on obs)	1.76 s	159.9 MB	-	no change
anndata (full load)	3.20 s	583.7 MB	-	no change

export-obs

Write the obs table out as CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.74 s	85.1 MB	1.8 MB	no change
anndata (read_elem on obs)	1.77 s	160.5 MB	1.8 MB	no change
anndata (full load)	3.28 s	583.9 MB	1.8 MB	no change

split-by-sample

Write one store per distinct value of an obs column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	6.65 s	198.4 MB	302.6 MB in 9 files	time -88%, RSS -74%
anndata (hand-written loop)	7.81 s	684.9 MB	301.7 MB in 8 files	no change

ls

List everything in the store.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.59 s	70.5 MB	-	no change
h5py (visit)	1.66 s	136.3 MB	-	no change
h5ls -r	n/a	n/a	n/a	-
	h5ls is not installed

create

Write an empty store with a given obs/var shape.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.64 s	84.5 MB	3.4 MB	no change
anndata	8.73 s	228.1 MB	49.2 MB	no change

export-array

Write a dense element out as .npy.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.60 s	82.6 MB	10.0 MB	no change
anndata (read_elem)	1.67 s	154.3 MB	10.0 MB	no change
anndata (full load)	3.27 s	583.8 MB	10.0 MB	no change

export-sparse

Write X out as Matrix Market text.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	25.52 s	76.3 MB	761.3 MB	no change
anndata (sparse_dataset)	7.12 s	968.7 MB	661.6 MB	no change
anndata (full load)	7.23 s	775.1 MB	661.6 MB	no change

import-dataframe

Replace obs from a CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.21 s	114.2 MB	296.0 MB	no change
anndata	6.88 s	596.5 MB	291.2 MB	no change

concat-outer

Concatenate two stores keeping the union of variables.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	12.20 s	172.0 MB	583.3 MB	time +106%
anndata (concat_on_disk)	6.26 s	450.5 MB	831.9 MB	no change
anndata (in memory)	12.18 s	1.8 GB	580.3 MB	no change

convert-dtype

Rewrite X as float32.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	5.57 s	90.2 MB	291.1 MB	no change
anndata (in memory)	6.91 s	983.8 MB	291.2 MB	no change
scanpy	n/a	n/a	n/a	-
	no dtype rewrite; scanpy defers to anndata

convert-layout

Transpose X from CSR to CSC.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli (streaming)	13.62 s	369.8 MB	1.2 GB	no change
adata-cli (--in-memory)	9.52 s	2.3 GB	297.0 MB	no change
anndata (in memory)	7.22 s	984.2 MB	297.0 MB	no change

rss-vs-size

Does peak memory track input size, at a fixed --chunk? This is the README's claim, stated as a measurement.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	12.13 s	173.8 MB	583.3 MB	time +105%
anndata (in memory)	12.18 s	1.8 GB	580.3 MB	no change

---

Rows marked n/a are operations the baseline does not offer; that is a result, not a gap in the measurement. Where a baseline is faster, the row stands as it is -- the trade this tool makes is memory for time, and hiding the cost would make the table useless.

Output size is the file as the filesystem reports it, which for HDF5 includes allocation slack. On a small tier that slack can be several times the stored bytes and the column says more about the writer's allocation strategy than about the data; at ci and above it is noise. Zarr stores report a file count alongside, because a directory of small chunks measures much larger than it holds.

adata-cli 0.6.0

Adds adata convert, and fixes four quadratic paths that made concat and split appear to hang on real data. Three of the four were found by a new suite of complexity guards rather than by users; the first was reported from a pipeline that had to kill twelve tasks after 98 minutes.

adata convert

Change a matrix's dtype, layout or density on disk, streaming. Closes #13.

adata convert data.h5ad X -o out.h5ad --dtype float32
adata convert data.h5ad X -o out.h5ad --layout csc
adata convert data.h5ad --all -o out.h5ad --dtype float32 --indices-dtype int32

Counts stored as float64 halve with --dtype float32. --layout csr|csc transposes between the sparse encodings; --layout dense|sparse changes the density. Works on X, any layer, raw/X, or any 2-D array by path.

Two things are refused before anything is written: a cast that would not round-trip — checked by casting every value and casting it back, not by comparing dtypes — and a densification that would inflate the store beyond 4×. --force overrides either, and both messages say what was measured.

Transposing streams by default and works on matrices too large to load; --in-memory is faster when the matrix fits.

Fixed

Where	Cost before
concat --merge re-read a var column once per variable	hours at 36,601 vars
concat category union used list membership	2,096,128 comparisons at 1,024 categories
split --by rescanned each chunk once per label	~10⁹ comparisons at 1M cells × 1k samples
concat built a Python object per obs row	three extra passes per column

Plus two found by automated review and confirmed by measurement: copy_dataset ignored its own read budget for variable-length strings (827 MB against a stated 32 MiB), and the benchmark's own peak-RSS measurement was floored by the runner's memory on Linux.

--merge drop and --uns-merge drop are now accepted — drop was the documented default but was rejected as a value.

New testing machinery

Complexity guards (tests/test_performance.py) covering every subcommand. They count operations rather than seconds, so none can fail because a CI runner was busy. Two claims are now enforced rather than described: view and ls read zero data elements at any store size, and streaming stays far below the input curve at a fixed --chunk.

A comparative benchmark (benchmarks/) against anndata and scanpy, published to docs/BENCHMARKS.md on every tag. Report-only — it never fails a build, and it publishes the rows where adata-cli is the slower of the two.

See docs/TESTING.md for both.

Upgrading

No breaking changes. pip install -U pyadata-cli, or quay.io/cellgeni/adata-cli:0.6.0.

Benchmark vs anndata/scanpy

Results — 0.6.0

Tier	ci, 50,000 obs x 20,000 var, 5% dense CSR
Var-heavy shape	2,000 obs x 36,601 var
Page cache	dropped between runs
Address-space ceiling	12.9 GB per process
Repeats	3 (fastest shown)
Platform	Linux x86_64, Python 3.12.3
Versions	adata-cli 0.6.0, anndata 0.13.4, scanpy 1.12.4
Generated	2026-09-24T10:35:13Z

startup

What does each contender cost before it does any work?

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.61 s	69.2 MB	-	-
anndata	1.65 s	132.5 MB	-	-
scanpy	3.65 s	264.8 MB	-	-

concat-inner

Concatenate two stores on the obs axis, inner join.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	5.90 s	176.2 MB	1.2 GB	-
anndata (concat_on_disk)	6.40 s	450.6 MB	831.9 MB	-
anndata (in memory)	12.27 s	1.8 GB	580.3 MB	-
scanpy	n/a	n/a	n/a	-
	scanpy has no concat of its own; it re-exports anndata's

concat-merge-same

Concatenate carrying var columns forward -- the REQ-71798 case.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.24 s	193.5 MB	92.6 MB	-
anndata (concat_on_disk)	2.37 s	247.3 MB	63.2 MB	-
anndata (in memory)	2.84 s	286.7 MB	44.6 MB	-

subset-query

Keep the obs rows matching a predicate on an existing column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	3.25 s	345.2 MB	309.7 MB	-
anndata (backed)	5.16 s	674.7 MB	146.6 MB	-
anndata (in memory)	5.42 s	984.2 MB	146.6 MB	-
scanpy (filter_cells)	9.31 s	1.6 GB	291.2 MB	-

h5ad-to-zarr

Convert a store to Zarr.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	6.04 s	246.0 MB	115.3 MB in 2133 files	-
anndata (in memory)	6.28 s	778.6 MB	136.8 MB in 55 files	-
scanpy	n/a	n/a	n/a	-
	no streaming converter; scanpy defers to anndata

inspect

Report what is in the store, without reading the matrix.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.62 s	71.5 MB	-	-
anndata (read_elem on obs)	1.82 s	159.9 MB	-	-
anndata (full load)	3.31 s	583.7 MB	-	-

export-obs

Write the obs table out as CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.77 s	85.6 MB	1.8 MB	-
anndata (read_elem on obs)	1.88 s	159.7 MB	1.8 MB	-
anndata (full load)	3.45 s	583.6 MB	1.8 MB	-

split-by-sample

Write one store per distinct value of an obs column.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	56.72 s	753.9 MB	630.9 MB in 9 files	-
anndata (hand-written loop)	7.82 s	685.2 MB	301.7 MB in 8 files	-

ls

List everything in the store.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.59 s	70.6 MB	-	-
h5py (visit)	1.65 s	136.4 MB	-	-
h5ls -r	n/a	n/a	n/a	-
	h5ls is not installed

create

Write an empty store with a given obs/var shape.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.65 s	84.2 MB	3.4 MB	-
anndata	8.83 s	228.0 MB	49.2 MB	-

export-array

Write a dense element out as .npy.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	0.65 s	82.8 MB	10.0 MB	-
anndata (read_elem)	1.82 s	154.5 MB	10.0 MB	-
anndata (full load)	3.42 s	583.7 MB	10.0 MB	-

export-sparse

Write X out as Matrix Market text.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	25.59 s	76.1 MB	761.3 MB	-
anndata (sparse_dataset)	7.14 s	967.9 MB	661.6 MB	-
anndata (full load)	7.56 s	773.6 MB	661.6 MB	-

import-dataframe

Replace obs from a CSV.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	1.27 s	113.8 MB	296.0 MB	-
anndata	7.02 s	596.3 MB	291.2 MB	-

concat-outer

Concatenate two stores keeping the union of variables.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	5.91 s	176.6 MB	1.2 GB	-
anndata (concat_on_disk)	6.49 s	450.7 MB	831.9 MB	-
anndata (in memory)	12.46 s	1.8 GB	580.3 MB	-

convert-dtype

Rewrite X as float32.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	5.68 s	90.4 MB	291.1 MB	-
anndata (in memory)	7.00 s	983.8 MB	291.2 MB	-
scanpy	n/a	n/a	n/a	-
	no dtype rewrite; scanpy defers to anndata

convert-layout

Transpose X from CSR to CSC.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli (streaming)	14.00 s	366.0 MB	1.2 GB	-
adata-cli (--in-memory)	9.75 s	2.3 GB	297.0 MB	-
anndata (in memory)	7.40 s	983.8 MB	297.0 MB	-

rss-vs-size

Does peak memory track input size, at a fixed --chunk? This is the README's claim, stated as a measurement.

Contender	Wall time	Peak RSS	Output	vs previous
adata-cli	5.93 s	171.1 MB	1.2 GB	-
anndata (in memory)	12.56 s	1.8 GB	580.3 MB	-

---

Rows marked n/a are operations the baseline does not offer; that is a result, not a gap in the measurement. Where a baseline is faster, the row stands as it is -- the trade this tool makes is memory for time, and hiding the cost would make the table useless.

Output size is the file as the filesystem reports it, which for HDF5 includes allocation slack. On a small tier that slack can be several times the stored bytes and the column says more about the writer's allocation strategy than about the data; at ci and above it is noise. Zarr stores report a file count alongside, because a directory of small chunks measures much larger than it holds.

adata-cli 0.5.1

adata-cli 0.5.1

The container image now works inside a Nextflow pipeline, and copying a
row-chunked store is no longer bottlenecked on read latency.

The Python package is unchanged apart from the performance fix below — if you
use adata from PyPI and not from the image, the upgrade is optional.

---

⚠️ Breaking: the image no longer sets an entrypoint

You must now name the command explicitly:

-docker run --rm -v /data:/data quay.io/cellgeni/adata-cli:0.5.1 view /data/f.h5ad
+docker run --rm -v /data:/data quay.io/cellgeni/adata-cli:0.5.1 adata view /data/f.h5ad

This affects Docker and Podman users only. If you run the image through
Apptainer or Singularity, nothing changes — singularity exec has always
ignored the entrypoint, so your commands already named adata.

The entrypoint had to go because Nextflow requires /bin/bash to be the
container entrypoint. Anything that relied on the old implicit form — a shell
alias, a wrapper script, a .bashrc function — needs the extra word.

Nothing was republished under the 0.5.0 tag, so pinned workflows keep
working until you move them.

---

Nextflow support

The image can now be used from a process directly:

process INSPECT {
    container 'quay.io/cellgeni/adata-cli:0.5.1'

    input:  path h5ad
    output: path 'summary.txt'

    script:
    """
    adata view ${h5ad} > summary.txt
    """
}

With either runtime enabled in nextflow.config:

// Docker
docker.enabled    = true
docker.runOptions = '-u $(id -u):$(id -g)'

// or Apptainer, e.g. on an HPC farm
apptainer.enabled     = true
apptainer.autoMounts  = true

What was wrong

• Every Docker- and Podman-backed task failed outright. Nextflow invokes
  docker run IMG /bin/bash -ue .command.sh; the old ENTRYPOINT ["adata"]
  turned that into adata /bin/bash -ue .command.sh, which died with
  No such command '/bin/bash'.
• Task metrics were silently missing. Nextflow shells out to ps to
  collect them, and ps was not in the image — so -with-trace,
  -with-report and Seqera Platform all showed empty CPU and memory columns.
  The tools Nextflow requires (bash, ps, awk, date, grep, sed,
  tail, tee) are now installed, and asserted at build time so they cannot
  silently disappear again.
• A bind-mounted $HOME could break the install. Apptainer mounts your
  home directory by default, so a ~/.local/lib/python3.12/site-packages
  holding a different zarr, h5py or numpy could shadow the image's own.
  The image now sets PYTHONNOUSERSITE=1.
• Running under an arbitrary UID left no writable cache. -u $(id -u):$(id -g)
  gives a container with no writable $HOME; the image now points
  XDG_CACHE_HOME at a world-writable directory.

Verified against Nextflow 26.04.6: a pipeline completes and reports a full
trace row.

---

Faster copies of row-chunked stores

subset, split, concat, import and the HDF5 ⇄ Zarr conversions all
stream through the same copy path, which used to read one source chunk at a
time. For a store
chunked (1, n_cols) — common in files written row-by-row — that meant one
row per read.

On a local disk this is merely wasteful. On Lustre or NFS every read is a
network round-trip costing milliseconds, so a million-row copy spent almost all
of its wall-clock waiting rather than working.

Reads are now sized to a 32 MiB budget, rounded down to a whole number of
source chunks:

store	source chunks	rows per read before	after
(1_000_000, 30_000) float32	(1, 30_000)	1	279
(10_000_000,) int64	(65_536,)	65,536	4,194,304

Variable-length string columns are sized correctly too. h5py reports
itemsize 8 for them because the value is a pointer rather than the text,
which previously overestimated how many rows fit in memory by an order of
magnitude.

No output changes — this is purely how much is read per call.

---

Also in this release

• Added a .dockerignore. Building the image locally no longer copies your
  working .venv, .git and .pytest_cache into the build context.
• The image's layers are ordered so that editing source no longer re-downloads
  duckdb on every rebuild.
• README.md and the documentation site were updated for the entrypoint
  change.

Upgrading

# PyPI
pip install --upgrade pyadata-cli

# Container
docker pull quay.io/cellgeni/adata-cli:0.5.1

quay.io/cellgeni/adata-cli:latest now points at 0.5.1.

Full changelog: https://github.com/cellgeni/adata-cli/blob/main/CHANGELOG.md

adata-cli 0.5.0

First release under the new name. pip install pyadata-cli — the command is adata.

The distribution is pyadata-cli because adata-cli was claimed on PyPI by an unrelated project. The import package and the command are both adata.

Why this release matters

The tool could not read any file written by anndata ≥ 0.11. Since pandas' future.infer_string became the default, anndata writes obs/_index as a nullable-string-array group rather than a dataset, and axis_len required a dataset — so info, export dataframe and subset all failed outright on essentially every file produced in the last year.

That is fixed, and now verified in CI against stores written by anndata 0.8, 0.9, 0.10, 0.11, 0.12 and 0.13, in both .h5ad and .zarr.

Two further faults fixed along the way:

• HDF5 → Zarr conversion failed on any store with string columns, i.e. all of them.
• subset silently dropped raw/. It is now carried over and matched against its own var axis.

Renamed

h5ad → adata-cli; the command is adata; info is now view. The h5ad command and the info subcommand remain as deprecating aliases and will be removed in 1.0.0.

New commands

adata view data.h5ad                     # AnnData-aware inspection
adata ls data.h5ad --long                # any HDF5/Zarr store, incl. .loom
adata create out.h5ad --n-obs 5000 --n-var 2000
adata subset data.h5ad -o out.h5ad -q "cluster == Cortex_2 and n_counts > 1000"
adata split data.h5ad --by sample -o per_sample/
adata concat per_sample/*.h5ad -o merged.h5ad --join outer --label sample

• ls — tree listing of any HDF5 or Zarr store, no AnnData assumptions. -1 emits bare paths for piping.
• create — write a valid empty store for import to fill in.
• split --by — one store per distinct value of a column, with a CSV manifest. Closes #2.
• concat — obs-axis concatenation with inner/outer join and merge strategies. Verified to agree with anndata.concat.
• subset --obs-query / --var-query — a small predicate language evaluated while streaming; no new dependency.

Closes #4: results go to stdout with status on stderr, export dataframe accepts any dataframe path, and export array can write to stdout.

Correctness

Everything written is now tagged with its encoding-type and encoding-version; text is variable-length UTF-8 as the spec requires; categoricals keep ordered and survive a CSV round-trip; None round-trips via anndata's null encoding. Sparse subsetting streams in blocks (byte-exact against scipy). All four HDF5/Zarr pairings convert, across Zarr v2 and v3.

Testing

1029 tests, coverage 92% with a 90% floor enforced in CI. The compatibility suite builds a store with each pinned anndata release in its own environment and checks that what this tool writes reopens in that same release.

Full detail in CHANGELOG.md. Documentation at https://cellgeni.github.io/adata-cli/.

Hot fix: root attributes

Summary

This hotfix improves AnnData on-disk schema compliance and subset-output interoperability, especially for downstream tools that validate element metadata.

Highlights

1) Root AnnData metadata enforcement

• Enforced root attributes on writable store operations:
  • encoding-type: "anndata"
  • encoding-version: "0.1.0"
• Added non-fatal read-time warning when these root attributes are missing.
• Applied enforcement consistently through store copy/convert paths.

2) Subset output schema hardening

• subset now guarantees presence of optional AnnData mapping groups when absent in source:
  • layers, obsm, obsp, varm, varp
• These mapping groups now carry mapping metadata for compatibility with this codebase’s documented schema:
  • encoding-type: "dict"
  • encoding-version: "0.1.0"

3) Documentation updates

• Updated element specification docs for HDF5 and Zarr to align with latest requirements and preserve explicit legacy compatibility notes.
• Clarified handling of historical mapping metadata (dict v0.1.0) for backward compatibility.

Test Coverage Added/Updated

• Added tests for root AnnData attribute enforcement and warning behavior.
• Added/updated subset integration tests to validate optional group creation and mapping metadata.

Impact

• No intended breaking CLI changes.
• Improves compatibility with schema-aware downstream readers.
• Reduces risk of malformed subset outputs due to missing mapping metadata.

Notes for users

• Some anndata versions (notably <= 0.12.0) may still fail in experimental.concat_on_disk(..., merge='same') when var contains columns due to upstream behavior. This is version-specific to anndata, not specific to this hotfix.

Included changes (since 0.3.1 baseline)

• Source: storage metadata enforcement and subset schema fixes
• Docs: element specification updates (HDF5/Zarr)
• Tests: new/expanded coverage for storage/subset behavior
• Lockfile: updated uv.lock

Hot-fix dataframe export

Summary

This hot-fix release resolves dataframe export issues affecting categorical obs/var columns:

• Fixes crashes when categorical labels contain non-ASCII UTF-8 characters (e.g. Greek letters).
• Fixes incorrect/empty values when exporting multiple categorical columns with h5ad export dataframe -c ....

Fixed

1) UTF-8 categorical decoding

decode_str_array now decodes byte/object arrays robustly as UTF-8 (with safe fallback), preventing errors like:

'ascii' codec can't decode byte ...

2) Categorical cache cross-column leakage

Categorical value caching now uses a stable per-column key, avoiding accidental cache reuse across different columns in the same export pass.

This fixes cases where one categorical column could appear blank or show values that belong to another categorical column.

Tests added

• Core read regression for non-ASCII byte decoding.
• Core read regression for multiple categorical columns sharing one export path.
• CLI regression for export dataframe with multiple categorical columns, including UTF-8 values.

User impact

Commands such as:

h5ad export dataframe subset_1k.h5ad obs -c "age,cell_type" --head 10

and

h5ad export dataframe subset_1k.h5ad obs -c "age,sample_ID,cell_type" --head 10

now return stable, correctly populated categorical values without decode failures.

Upgrade

Install/update to h5ad==0.3.1.

Notes

• No CLI interface changes.
• No data format migration required.
• This release is backward-compatible and focused on dataframe export correctness.

0.3.0 Export/Import commands

Breaking Changes

• Subset CLI output is now an option: h5ad subset INPUT --output OUTPUT (positional output removed); use --inplace to modify the source file directly.
• Info CLI flag rename: --types is now --tree.
• Export CLI uses export dataframe instead of the old export table flow (table command removed).
• Export/import entry paths are now positional arguments (e.g., h5ad export dataframe FILE obs ...).

New Features

• Full .zarr support alongside .h5ad, including backend detection and store abstraction.
• Import commands for dataframe/array/sparse/dict data types with consistent CLI UX.
• Expanded info capabilities: tree view, depth control, and entry-path inspection.

Subset Improvements

• Sparse + dense matrix handling now works across layers, obsm, varm, obsp, and varp.
• Safer copying of HDF5 groups (improves robustness for complex uns content).
• Optional in-place subsetting with safe temp-store replacement.

Export/Import Improvements

• Better handling for awkward arrays and sparse exports.
• More robust CSV/MTX/NPY import and export behavior, with clearer errors.

Compatibility & Legacy Support

• Added support for legacy v0.1.0 categorical/dataframe formats.
• Expanded entry type detection and read logic for older formats.

Docs

• New GET_STARTED.md plus updated README and element specs documentation for .h5ad/.zarr.

Tests & CI

• Larger test matrix, improved coverage reporting, and expanded test suites for export/import/subset/info and zarr.

Migration Guide (quick examples)

# info: old -> new
h5ad info --types data.h5ad
h5ad info --tree data.h5ad

# subset: old -> new
h5ad subset data.h5ad out.h5ad --obs cells.txt
h5ad subset data.h5ad --output out.h5ad --obs cells.txt
h5ad subset data.h5ad --obs cells.txt --inplace

# export table -> export dataframe
h5ad export table data.h5ad obs --output obs.csv
h5ad export dataframe data.h5ad obs --output obs.csv

CLI v0.2.0 Release Notes

h5ad CLI v0.2.0 Release Notes

Release Date: January 15, 2026

We're excited to announce a major update to the h5ad CLI tool! This release adds powerful new capabilities for working with large .h5ad (AnnData) files while maintaining the memory-efficient streaming approach you know and love.

🎉 What's New

New Command: subset - Filter Your Data

The biggest addition in this release! You can now create filtered subsets of your .h5ad files without loading them into memory.

Filter by cells (observations):

h5ad subset input.h5ad output.h5ad --obs cell_names.txt

Filter by genes (variables):

h5ad subset input.h5ad output.h5ad --var gene_names.txt

Filter by both:

h5ad subset input.h5ad output.h5ad --obs cells.txt --var genes.txt

Key Features:

• ✅ Works with dense and sparse matrices (CSR/CSC formats)
• ✅ Preserves all metadata and attributes
• ✅ Memory-efficient chunked processing
• ✅ Beautiful progress bars show you what's happening
• ✅ Handles large files (multi-GB) with ease

Input Format:
Your cell/gene name files should contain one name per line:

AAACCTGAGAAACCAT-1
AAACCTGAGACAGACC-1
AAACCTGAGGCATGGT-1

⚠️ Breaking Changes

Table Command Arguments

We've renamed two flags in the table command for better clarity:

• --cols → --columns (short form -c unchanged)
• --out → --output (short form -o unchanged)

Old syntax (v0.1.x):

h5ad table data.h5ad --axis obs --cols cell_type,donor --out metadata.csv

New syntax (v0.2.0):

h5ad table data.h5ad --axis obs --columns cell_type,donor --output metadata.csv

If you're using the short forms -c and -o, your existing scripts will continue to work without changes.

✨ Improvements

Enhanced info Command

The info command now provides:

• Better formatted output with colors
• Clearer file structure visualization
• More detailed group and dataset information

Example:

h5ad info data.h5ad

Improved table Command

The table export command now includes:

• Better memory management for large files
• Improved column validation
• More informative error messages
• Clearer argument names (--columns and --output)

Example:

h5ad table data.h5ad --axis obs --columns cell_type,donor --output metadata.csv

🚀 Performance & Reliability

• Better Memory Management: All commands use chunked streaming to handle files of any size
• Progress Feedback: Long-running operations now show progress bars
• Error Handling: More informative error messages help you troubleshoot issues
• Standards Compliant: Fixed encoding attributes to match h5ad specification

📦 Installation

Using uv (Recommended)

# Clone the repository
git clone <your-repo-url>
cd h5ad

# Install dependencies
uv sync

# Run commands
uv run h5ad --help

Using Docker

# Pull the latest image
docker pull quay.io/cellgeni/h5ad-cli:latest

# Run commands
docker run -v $(pwd):/data quay.io/cellgeni/h5ad-cli:latest info /data/file.h5ad

📚 Documentation

• README: Updated with all new features and examples
• Testing Guide: New documentation for developers at docs/TESTING.md

🔧 Technical Details

New Dependencies

• numpy>=2.3.5 - For efficient array operations
• rich>=14.2.0 - For beautiful terminal output

Docker Images

Images are now automatically built and published to Quay.io:

• Tagged releases: quay.io/cellgeni/h5ad-cli:0.2.0
• Latest stable: quay.io/cellgeni/h5ad-cli:latest
• Preview versions: quay.io/cellgeni/h5ad-cli:0.2.0-preview3

🐛 Bug Fixes

• Fixed encoding-type attribute to comply with h5ad standard (now writes as bytes)
• Improved categorical data handling
• Better sparse matrix format preservation

💡 Usage Tips

Working with Large Files

The subset command is particularly useful for:

• Creating test datasets: Extract a small subset of cells/genes for quick testing
• Cell type analysis: Filter specific cell populations for downstream analysis
• Gene panel studies: Extract expression data for specific gene lists
• Memory constraints: Reduce file size before loading into memory-intensive tools

Example Workflow

# 1. Check file structure
h5ad info large_dataset.h5ad

# 2. Export cell metadata to find interesting cells
h5ad table large_dataset.h5ad --axis obs --output cells_metadata.csv

# 3. Create a subset with specific cells (after selecting in external tool)
h5ad subset large_dataset.h5ad filtered_dataset.h5ad --obs selected_cells.txt

# 4. Verify the subset
h5ad info filtered_dataset.h5ad

🔜 Coming Soon

We're working on:

• Additional matrix format support
• More export options
• Performance optimizations for very large files
• Additional filtering options (by metadata values)

🙏 Feedback

We'd love to hear from you! If you encounter any issues or have feature requests, please:

• Open an issue on GitHub
• Contact the development team
• Check the documentation for troubleshooting tips

📝 Full Changelog

See PR_DEV_TO_MAIN.md for the complete technical changelog with all 24 commits and detailed changes.

---

Upgrade Instructions

From v0.1.x

Update your installation:

# With uv
cd h5ad
git pull
uv sync

# With Docker
docker pull quay.io/cellgeni/h5ad-cli:latest

Update your scripts:

If you're using the table command with --cols or --out flags, update them to --columns and --output. Alternatively, use the short forms -c and -o which work in both versions.

The info command and new subset command work as documented with no migration needed.

---

Happy subsetting! 🎊