Read performance tuning

[jeromekelleher]

This discussion is to document some read performance benchmarks and to discuss results of various perf approaches.

[jeromekelleher]

Starting with the baseline measurements from the benchmark suite introduced in #308

vcztools benchmark — full-suite findings

Date: 2026-04-23
Dataset: 100 000 diploid samples × 496 414 variants (synthetic, np.arange % N augmented fields)
Chunk shape: (1000, 10000) for 2-D arrays
Suite: 8 tasks × 6 backends × 3 repeats = 144 task-runs, all record-count assertions passed
Output: performance/results/full-suite.jsonl

Median elapsed_s

Task	local-dir	local-dir-zv3	local-zip	local-http	obstore-file	icechunk
iter_no_fields	0.006	0.006	0.008	0.020	0.008	0.003
iter_info_only	18.93	17.72	10.88	34.13	19.51	6.85
region_info_and_format	9.99	9.98	9.94	10.36	9.65	9.40
subset_10_samples	55.62	55.71	53.61	61.56	56.42	51.25
region_variant_position	0.057	0.053	0.049	0.076	0.058	0.049
filter_info_dp_gt_80	9.95	9.03	6.54	18.17	10.56	3.89
region_filter_format_gq_gt_50	0.79	0.82	0.73	0.83	0.74	0.75
region_and_sample_subset	0.079	0.081	0.078	0.092	0.080	0.074

Backend geomean across all 8 tasks

Rank	Backend	Geomean elapsed_s
1	icechunk	0.765
2	local-zip	0.971
3	local-dir-zv3	1.072
4	local-dir	1.112
5	obstore-file	1.151
6	local-http	1.609

Key findings

1. icechunk wins on metadata-heavy scans

Icechunk is the fastest backend on every task except two ties. The lead is largest where the cost is chunk-plan assembly rather than bulk decompression:

• iter_info_only: 2.8× faster than local-dir (6.9 s vs 18.9 s).
• filter_info_dp_gt_80: 2.6× faster than local-dir (3.9 s vs 10.0 s).

Icechunk's virtual chunk manifest means opening an array and planning reads doesn't pay per-chunk filesystem or metadata costs.

2. local-http is consistently slowest

local-http is 1.6× the geomean of icechunk, and specifically 4.7×–5.9× slower on the high-chunk-count scans (iter_info_only, filter_info_dp_gt_80, iter_no_fields). Python's built-in http.server with per-chunk HEAD/GET round-trips is what it says on the tin — no keep-alive, no range-request batching.

3. FMT-dominated tasks are backend-agnostic

region_info_and_format (9.4–10.4 s) and subset_10_samples (51–62 s) span < 10 % and < 20 % across backends respectively. Once the task reads hundreds of MB of compressed genotype data, the bottleneck is blosc decompression CPU — the storage backend barely matters.

4. Zarr v2 vs v3 directory is nearly a wash

local-dir vs local-dir-zv3 are within a few percent, with v3 edging out v2 on INFO-heavy tasks (iter_info_only 17.7 s vs 18.9 s, filter_info_dp_gt_80 9.0 s vs 9.9 s). v3's codec stack is slightly cheaper per chunk, but it's not the dominant cost anywhere.

5. obstore over file:// is slightly slower than raw fsspec

obstore-file trails local-dir by 3–10 % on the heavy tasks. The trip through obstore's Rust layer adds latency that only pays off over the network.

6. local-zip beats everything except icechunk

Zip's sequential layout and near-zero metadata overhead plays well with bulk reads: it's within 50 % of icechunk's geomean and actually fastest on two tasks (region_variant_position, region_filter_format_gq_gt_50). Worth keeping as the default easy-to-ship format.

Target band check

Target: tasks land in [0.5 s, 30 s] on local-dir.

Status	Tasks
In band, all backends	region_info_and_format, filter_info_dp_gt_80, region_filter_format_gq_gt_50
Below the floor (intentional noise-level probes)	iter_no_fields, region_variant_position, region_and_sample_subset
In band except on local-http	iter_info_only (34.1 s on HTTP)
Above the ceiling on every backend	subset_10_samples (51–62 s)

subset_10_samples reads one samples-chunk (10 % of the samples axis) across every variant-chunk — ~10 GB of uncompressed genotype data. Staying above the band is inherent at 100 k × 496 k.

Peak memory

Peak RSS stayed consistent across backends; the heaviest tasks are the ones that materialise large FMT chunks:

Task	Peak RSS (max across backends)
region_info_and_format	~1.7 GB
region_filter_format_gq_gt_50	~1.0 GB
subset_10_samples	~640 MB
Others	≤ 650 MB

[jeromekelleher]

This is the updated version after #313 landed.

vcztools benchmark — full-suite findings

Date: 2026-04-23 (rerun with cpu_s, bytes_retrieved, data_rate_mib_s; task catalogue now 9 tasks)
Dataset: 100 000 diploid samples × 496 414 variants (synthetic, np.arange % N augmented fields)
Chunk shape: (1000, 10000) for 2-D arrays
Suite: 9 tasks × 6 backends × 3 repeats = 162 task-runs, all record-count assertions passed
Output: performance/results/full-suite.jsonl

Median wall time (elapsed_s)

Task	local-dir	local-dir-zv3	local-zip	local-http	obstore-file	icechunk
iter_no_fields	0.010	0.008	0.009	0.018	0.008	0.005
iter_info_only	21.38	17.67	9.94	35.24	21.61	7.00
region_info_and_format	10.18	10.72	10.51	10.23	10.08	10.14
first_samples_chunk	19.91	20.19	20.39	22.61	21.03	18.13
first_variant_chunks	17.39	19.28	16.96	15.65	16.34	22.41
region_variant_position	0.069	0.062	0.056	0.088	0.062	0.051
filter_info_dp_gt_80	10.94	11.24	6.85	18.37	10.63	3.59
region_filter_format_gq_gt_50	0.808	0.783	0.751	0.831	0.793	0.799
region_and_sample_subset	0.082	0.082	0.080	0.087	0.081	0.076

Median data rate (data_rate_mib_s)

MiB/s of returned data (the user-visible output of variant_chunks). Reading pattern differences show up as wide spreads across the three ~1 GB-returning tasks.

Task	Bytes returned	local-dir	local-dir-zv3	local-zip	local-http	obstore-file	icechunk
iter_info_only	21.8 MB	0.97	1.18	2.10	0.59	0.96	2.98
region_info_and_format	496.5 MB	46.5	44.2	45.1	46.3	46.96	46.7
first_samples_chunk	992.8 MB	47.6	46.9	46.4	41.9	45.0	52.2
first_variant_chunks	1000.0 MB	54.8	49.5	56.2	60.9	58.4	42.6
filter_info_dp_gt_80	0.59 MB	0.05	0.05	0.08	0.03	0.05	0.16
region_filter_format_gq_gt_50	99.3 MB	117.2	121.0	126.1	114.0	119.4	118.6
region_and_sample_subset	1.99 MB	23.1	23.1	23.6	21.7	23.5	24.8

CPU/wall ratio

Ratio above 1.0 indicates multi-threaded blosc decompression; well below 1.0 indicates I/O wait. On our 100k × 496k dataset:

Task	local-dir	local-dir-zv3	local-zip	local-http	obstore-file	icechunk
iter_info_only	0.59	0.51	0.59	0.63	0.81	0.98
region_info_and_format	0.99	0.99	1.00	0.98	0.99	1.01
first_samples_chunk	0.87	0.83	0.83	0.81	0.88	1.02
first_variant_chunks	0.98	0.98	1.00	0.99	1.00	1.02
filter_info_dp_gt_80	0.72	0.53	0.53	0.65	0.82	0.91
region_filter_format_gq_gt_50	0.97	0.95	0.98	0.93	0.97	0.98
region_and_sample_subset	0.88	0.81	0.83	0.83	0.88	1.03

Peak RSS (MB, max across 3 repeats)

Task	local-dir	local-dir-zv3	local-zip	local-http	obstore-file	icechunk
iter_info_only	135	134	151	147	147	149
region_info_and_format	1692	1693	1707	1696	1702	1706
first_samples_chunk	403	403	406	406	406	407
first_variant_chunks	1275	1276	1295	1276	1278	1279
region_filter_format_gq_gt_50	846	852	875	877	860	857

Backend geomean ranking

Geometric mean of elapsed_s across all 9 tasks (lower is better):

Rank	Backend	Geomean elapsed_s
1	icechunk	1.056
2	local-zip	1.249
3	local-dir-zv3	1.429
4	obstore-file	1.441
5	local-dir	1.495
6	local-http	1.857

[tomwhite]

This is awesome @jeromekelleher!

Interesting that Icechunk does so well - intuitively it has an extra layer of indirection, but that can actually help for some styles of query.

Zip files look great as a read-only format (as long as they are not too big I suppose).

obstore over file:// is slightly slower than raw fsspec

Is it using fsspec or just Python file handling? This benchmark doesn't test obstore for remote files, but we might want to make it the default for those at some point.

[jeromekelleher]

I'm not sure the original comment was correct - I think we'd need to look at the details carefully.

[jeromekelleher]

vcztools perf branch — what we did and what we learned

A condensed summary of the performance work done in #315

Headline

first_variant_chunks on icechunk goes from 62 MiB/s to 1280 MiB/s (~20×) on a 4-CPU box; first_samples_chunk now delivers a full 10 000-sample chunk (10× the baseline payload) in ~4 s instead of 17 s for 1 000 samples — call it ~3–4× per-byte. Bulk reads on the fast backends now sit close to the 4-core blosc decompression ceiling.

The single most important finding

The 2026-04-23 baseline report blamed single-threaded blosc decompression for 50–60 MiB/s throughput. Profiling showed Zarr I/O + decompression was only ~10 % of wall time. The real cost was Python-level array munging: a no-op arr[:, np.arange(N)] fancy-index copy followed by a non-contiguous np.concatenate and a final np.ascontiguousarray — about 57 % of wall time, before any concurrency was in play. Fixing the index pattern alone gave 8×.

Optimization phases

• Phase 0.5 — collapse no-op fancy indexing (8.0× alone). samples.build_chunk_plan now emits selection=None for full chunks and slice(start, stop) for contiguous ranges; ndarray only when truly needed. CachedChunk skips the gather + non-contiguous concat in those cases.
• Phase 1 — async fan-out (further ~2×). Prefetch dispatches through Zarr's own event loop via AsyncArray.getitem, so blosc decompression parallelises across cores without us managing threads. A sync fast-path for 1-read prefetches avoids ~1 ms of asyncio dispatch overhead per chunk on tasks with one read per chunk.
• Phase 2 — cross-chunk readahead (further ~1.1×). _ReadaheadPipeline keeps a byte budget of in-flight prefetches (default 256 MiB, tunable). Byte budget rather than chunk count because chunks vary 2 MB – 1 GB depending on the field and the selection — a count-based depth either starves fan-out or blows RSS.
• Slice fix. np.ascontiguousarray deferred from the reader to the VCF encoder boundary (c_chunk_to_vcf), the only downstream consumer that requires C-contiguous layout. Slice-based selections now match full-chunk selections within ~1 % on bulk throughput.
• Filter pipeline. Two-stage prefetch (read filter fields, then query fields after rejecting whole chunks) was wasted complexity: whole-chunk rejection on a 1000-variant chunk under any realistic filter is effectively zero-probability (~10⁻⁹⁶ for DP>80). Now a single unified _ReadaheadPipeline covers both. +117 % on filter_info_dp_gt_80_genotypes / icechunk.

[jeromekelleher]

vcztools — current benchmark numbers

Snapshot of the post-perf-work benchmark matrix.

Setup

• Source run: performance/reports/run-readerkwargs.jsonl, recorded 2026-04-27 against the per-pipeline-executor implementation (final state of the perf branch).
• Hardware: 4-CPU host claude-worker1.
• Dataset: 100 000 diploid samples × 496 414 variants, chunk shape (1000, 10000). Synthetic deterministic fields (np.arange- derived variant_DP, variant_QUAL, variant_IMPACT, call_DP, call_GQ); record counts asserted against bench.vcz.benchmark_meta.json.
• Matrix: 11 tasks × 6 backends × 3 repeats = 198 runs. All record-count assertions passed.
• Reproduce:

  uv run python performance/benchmarks.py run \
      --dataset performance/data/bench.vcz \
      --output performance/results/run.jsonl
  uv run python performance/benchmarks.py compare \
      performance/reports/run-phase0-baseline.jsonl \
      performance/results/run.jsonl
  

Headline gains vs the 2026-04-23 baseline

Task / Backend	Baseline	Current	Speedup
first_variant_chunks / icechunk	62 MiB/s	1280 MiB/s	20.6×
first_variant_chunks / local-zip	62 MiB/s	1211 MiB/s	19.4×
first_samples_chunk / icechunk¹	17.2 s	4.06 s	4.2× wall, ~42× per byte
first_samples_chunk / local-zip¹	19.4 s	6.05 s	3.2× wall, ~32× per byte

¹ first_samples_chunk was reshaped during the perf round: the baseline took 1 000 samples (~947 MB output); the current task takes the full 10 000-sample chunk (~9.47 GiB). Per-byte throughput is the fair comparison; wall-time is given alongside for reference.

Median elapsed time (seconds)

Task	icechunk	local-dir	local-dir-zv3	local-http	local-zip	obstore-file
iter_no_fields	0.002	0.007	0.006	0.015	0.006	0.007
iter_info_only	5.16	13.36	13.25	26.80	25.54	13.37
region_info_and_format	1.27	1.68	1.47	1.49	1.59	1.35
region_variant_position	0.045	0.057	0.052	0.082	0.053	0.056
region_and_sample_subset	0.040	0.040	0.038	0.050	0.042	0.037
region_filter_format_gq_gt_50	0.611	0.799	0.727	0.787	0.793	0.733
first_samples_chunk	4.06	4.01	3.76	6.21	6.05	4.57
first_samples_chunk_slice	4.17	4.05	3.94	6.36	6.29	4.51
first_variant_chunks	0.745	0.787	0.749	0.849	0.788	0.765
filter_info_dp_gt_80	2.51	6.11	5.33	12.54	10.61	6.55
filter_info_dp_gt_80_genotypes	5.80	11.28	9.73	22.55	24.72	10.61

Median throughput (MiB/s of returned data)

Tasks whose output is below ~1 MB are omitted (throughput is dominated by metadata overhead there).

Task	Output	icechunk	local-dir	local-dir-zv3	local-http	local-zip	obstore-file
first_samples_chunk	9.47 GiB	2330	2362	2520	1526	1565	2073
first_samples_chunk_slice	9.47 GiB	2270	2338	2401	1489	1505	2099
first_variant_chunks	953 MiB	1280	1212	1273	1123	1211	1247
region_info_and_format	473 MiB	374	281	322	317	298	351
region_filter_format_gq_gt_50	95 MiB	155	119	130	120	119	129
region_and_sample_subset	1.9 MiB	48	48	50	38	45	51
filter_info_dp_gt_80_genotypes	188 MiB	32	17	19	8	8	18

CPU/wall ratio (4 cores available; ceiling ≈ 4)

Task	icechunk	local-dir	local-dir-zv3	local-http	local-zip	obstore-file
first_samples_chunk	3.06	2.85	2.86	2.08	1.74	2.82
first_samples_chunk_slice	3.04	2.82	2.78	2.05	1.72	2.73
filter_info_dp_gt_80_genotypes	2.69	1.62	1.62	1.16	0.77	1.83
first_variant_chunks	2.46	2.17	2.24	2.02	2.08	2.20
region_info_and_format	1.38	1.20	1.24	1.13	1.12	1.25
region_and_sample_subset	1.46	1.36	1.35	1.19	1.17	1.46
region_filter_format_gq_gt_50	1.12	1.08	1.07	1.06	1.02	1.10
filter_info_dp_gt_80	1.21	0.90	0.70	0.74	0.41	1.03
iter_info_only	1.20	0.95	0.77	0.75	0.43	1.09

Peak RSS (MB)

Bulk tasks land in the ~600 MB – 2 GB range. Readahead budget is 256 MiB of in-flight prefetched blocks; the rest is assembled chunks held briefly plus interpreter / Zarr overhead. Well within single-machine budgets.

Task	icechunk	local-dir	local-dir-zv3	local-http	local-zip	obstore-file
region_info_and_format	2153	2033	2069	2120	2084	2113
region_filter_format_gq_gt_50	1137	1116	1122	1133	1127	1142
first_variant_chunks	1113	1192	1164	1179	1156	1122
region_variant_position	935	935	935	935	935	935
region_and_sample_subset	737	736	736	737	737	737
iter_info_only / iter_no_fields	735	734	734	735	736	735
first_samples_chunk	634	654	540	644	651	647
first_samples_chunk_slice	653	657	639	649	650	653
filter_info_dp_gt_80_genotypes	626	612	616	610	603	615
filter_info_dp_gt_80	164	124	126	141	153	147

Backend ranking (geomean median elapsed across all 11 tasks)

Rank	Backend	Geomean elapsed (s)
1	icechunk	0.589
2	local-dir-zv3	0.803
3	obstore-file	0.868
4	local-dir	0.877
5	local-zip	1.124
6	local-http	1.299

[jeromekelleher]

vcztools benchmark snapshot — more-perf-runs branch

Snapshot of the full benchmark matrix at commit 4b104123df68c84db357a0cfcc28c95269d81035, run 2026-05-12 on claude-worker1. Each cell is the median of three repeats. Sources:

• performance/results/wide-more-perf-runs.jsonl (252 rows)
• performance/results/long-more-perf-runs.jsonl (156 rows)

Setup

• Hardware: 4-CPU host claude-worker1.
• Wide dataset: 100 000 diploid samples × 496 414 variants, call_genotype chunks (1000, 10000, 2). Uniform 1000-variant chunks across every variant-axis field.
• Long dataset: 10 diploid samples × 28 156 764 variants, call_genotype chunks (1000, 10, 2). Variant-only fields use proportional chunking sized to ~10 MiB uncompressed per chunk, so chunk sizes vary across fields (e.g. variant_position 2 621 000, variant_contig 10 485 000, variant_IMPACT 327 000, variant_allele 163 000).
• Backends: six retrieval backends per task (local-dir, local-dir-zv3, local-zip, local-http, obstore-file, icechunk). Output tasks run on local-dir only.
• Region: all region_* tasks share a 993-variant window on the wide dataset (1:5005712-5024875, 0.2 % of contig 1).

Cells below show elapsed (median of 3) / rate MiB/s — data_rate_mib_s for retrieval tasks, output_rate_mib_s for output tasks. — means the rate is zero or not reported. Sub-millisecond tasks are formatted in ms.

Wide dataset (100 000 samples × 496 414 variants)

Retrieval

task	local-dir	local-dir-zv3	local-zip	local-http	obstore-file	icechunk
iter_no_fields	0.45ms / —	0.39ms / —	0.47ms / —	0.54ms / —	0.37ms / —	0.34ms / —
iter_info_only	2.45s / 9	2.72s / 8	10.05s / 2	4.40s / 5	2.10s / 10	1.88s / 11
region_info_and_format	1.16s / 407	1.15s / 410	1.29s / 368	2.03s / 234	855ms / 554	937ms / 505
first_samples_chunk	3.72s / 2547	3.60s / 2632	4.29s / 2210	5.12s / 1849	3.69s / 2568	3.77s / 2512
first_samples_chunk_slice	3.45s / 2748	3.52s / 2693	3.76s / 2516	4.85s / 1950	3.48s / 2720	3.48s / 2719
first_variant_chunks	942ms / 1012	984ms / 970	1.07s / 890	1.09s / 871	1.02s / 939	1.04s / 913
fmt_fields	736ms / 3888	752ms / 3807	1.41s / 2028	1.20s / 2393	1.17s / 2448	983ms / 2911
fmt_fields_filtered	8.17s / 210	8.58s / 200	8.43s / 204	8.70s / 197	8.45s / 203	8.03s / 214
region_variant_position	9.0ms / —	9.9ms / —	2.9ms / 1	18.0ms / —	11.0ms / —	5.7ms / 1
filter_info_dp_gt_80	838ms / 1	921ms / 1	3.82s / —	1.62s / —	839ms / 1	722ms / 1
filter_info_dp_gt_80_genotypes	4.29s / 44	4.55s / 41	8.93s / 21	10.50s / 18	4.31s / 44	4.20s / 45
region_filter_format_gq_gt_50	714ms / 133	837ms / 113	849ms / 112	733ms / 129	689ms / 137	685ms / 138
region_and_sample_subset	33.3ms / 57	31.8ms / 60	33.6ms / 56	42.7ms / 44	34.5ms / 55	28.8ms / 66

Output (local-dir)

task	elapsed	output_rate (MiB/s)
output_vcf	8.56s	107
output_plink	6.15s	77
output_bed_stream	5.52s	85
output_bgen_l0	8.36s	169
output_bgen_l1	9.05s	7
output_bgen_l6	33.73s	1

Observations — wide

• Bulk genotype reads cluster around 2.5–2.7 GiB/s on the four fast backends (local-dir, local-dir-zv3, obstore-file, icechunk) for first_samples_chunk and first_samples_chunk_slice. local-zip and local-http trail at ~1.8–2.2 GiB/s — local-zip serialises decompression on the single zip handle, and local-http adds per-chunk request overhead.
• fmt_fields is the throughput peak at 3.9 GiB/s on local-dir. Pulling three pre-allocated call fields (call_genotype, call_GQ, call_DP) end-to-end keeps the 4-core blosc pool saturated.
• Tiny-output tasks favour fast lookups. iter_no_fields, region_variant_position, region_and_sample_subset are all sub-50 ms across every backend. local-zip does the variant-position lookup fastest (2.9 ms) because the entire field fits in a single read.
• iter_info_only is the slow scalar-only task at 1.9–10 s. It iterates the whole 496 414-variant axis pulling six INFO scalars — bytes-light but Python-loop-heavy. local-zip is 3–5× slower than the rest because the small reads serialise on the zip handle.
• fmt_fields_filtered is filter-bound, not I/O-bound. All six backends finish in 8.0–8.7 s with data rates 197–214 MiB/s — the FMT-scope GQ>50 predicate dominates, leaving the bulk-read path idle. This matches the known follow-up in performance/summary.md.
• filter_info_dp_gt_80_genotypes separates the backends. Once the filter rejects 80 % of variants and only their genotypes are read, the fast backends do 4.2–4.6 s at 41–45 MiB/s while local-http falls to 10.5 s / 18 MiB/s.
• output_bgen compression is the dominant cost. Level-0 (no compression) sustains 169 MiB/s; level-1 collapses to 7 MiB/s and level-6 to 1 MiB/s. The current default for view-bgen should be chosen with this curve in mind.
• Peak RSS: median 828 MB across all wide tasks, max 2.3 GB — bounded by the 256 MiB readahead byte budget plus per-task materialised arrays.

Long dataset (10 samples × 28 156 764 variants, proportional variant-only chunks)

Retrieval

task	local-dir	local-dir-zv3	local-zip	local-http	obstore-file	icechunk
iter_no_fields	20.1ms / —	17.9ms / —	33.3ms / —	20.2ms / —	19.1ms / —	24.2ms / —
iter_info_only	3.05s / 388	3.64s / 324	3.02s / 391	2.97s / 398	3.29s / 359	2.77s / 426
region_info_and_format	192ms / 17	206ms / 16	790ms / 4	1.19s / 3	202ms / 17	152ms / 22
first_variant_chunks	17.0ms / 6	15.8ms / 6	27.4ms / 3	29.6ms / 3	17.3ms / 6	19.9ms / 5
region_variant_position	43.2ms / 5	42.7ms / 5	36.1ms / 6	53.7ms / 4	43.2ms / 5	18.7ms / 11
filter_info_dp_gt_80	241ms / 133	230ms / 138	542ms / 59	302ms / 105	286ms / 112	309ms / 103
region_filter_format_gq_gt_50	87.1ms / 4	95.9ms / 4	114.5ms / 3	1.12s / —	83.0ms / 5	57.3ms / 7
full_genotypes	30.25s / 18	33.64s / 16	20.63s / 26	54.37s / 10	28.90s / 19	31.58s / 17

Output (local-dir)

task	elapsed	output_rate (MiB/s)
full_output_vcf	4.4 min	17
full_output_plink	2.7 min	4
full_output_bed_stream	58.11s	1

Observations — long

• Backend ordering inverts vs the wide dataset for full_genotypes. local-zip is the fastest backend at 26 MiB/s (20.6 s wall), ahead of obstore-file (19), local-dir (18), icechunk (17), local-dir-zv3 (16), and local-http (10). With only 10 samples, each variant chunk is small enough that zip's single-handle penalty disappears and its compact on-disk layout wins.
• iter_info_only runs at 324–426 MiB/s — the dominant work is the read loop, not the bytes. It iterates 28 M variants reading six scalar INFO fields. Because the proportional-chunked variant fields have pairwise-coprime chunk sizes (e.g. 327 000 and 2 621 000 share GCD 1 000), the stream chunk size collapses to the GCD, which multiplies the number of iterations relative to the homogeneous-chunk case. This is the architectural cost of heterogeneous chunk layouts and the current single biggest opportunity for long-dataset retrieval.
• Region tasks are fast on a sparse field. The shared 0.2 % region resolves to a tiny absolute slice on the 28 M-variant dataset; region_info_and_format is 152–202 ms on the fast backends, climbing to 790 ms / 1.19 s on local-zip / local-http.
• first_variant_chunks is dominated by overhead. Reading 5000 variants × 10 samples is sub-30 ms everywhere; the absolute data volume (~150 KB) is too small to spread across backends.
• Output is encoder-bound and the spread is dramatic. VCF text encoding finishes in 4.4 min at 17 MiB/s; PLINK BED via the full encoder (which also writes .bim / .fam and runs allele recoding) takes 2.7 min at 4 MiB/s; the streaming BedEncoder drains the same 28 M variants in 58 s but its 1 MiB/s reported output rate is anomalous — likely an accounting issue in the bytes_written field for output_bed_stream-class tasks.
• Peak RSS is highest in the long-output pipeline. full_output_plink peaks at 4.1 GB (median 4.1 GB), full_output_bed_stream at 2.9 GB, full_output_vcf at 1.4 GB. Retrieval tasks on long are bounded by 1.1 GB (full_genotypes).

Cross-shape highlights

• Bulk genotype throughput on small samples × long variants (full_genotypes, 10 samples) is ~25× lower in MiB/s than on 100 000 samples × moderate variants (first_samples_chunk). The wide bulk task pulls 2.5 GiB/s; the long one ~18 MiB/s. Wide benefits from larger contiguous decompression buffers per chunk; long pays per-chunk overhead on a thin sample axis.
• iter_info_only is the one task where the dominant cost differs between shapes. On wide it is bytes-light Python looping (~10 MiB/s); on long with proportional chunking it iterates many more stream chunks because the GCD of the per-field chunk sizes is small. Both end up in the 2–4 s band but for different reasons.
• output_bgen_l0 is the fastest single-file writer measured (169 MiB/s on wide). Compression dominates the remaining BGEN cost; output_vcf (107 MiB/s) and output_bed_stream (85 MiB/s) are the next closest.
• local-http is the slowest backend on every retrieval task it runs. Per-chunk HTTP request overhead is unavoidable for small-record tasks (region_*) and significant for bulk tasks (first_samples_chunk 1.85 vs 2.5–2.7 GiB/s elsewhere).

Reproduce

uv run python performance/benchmarks.py run \
    --dataset performance/wide_bench.vcz \
    --shape wide \
    --output performance/results/wide-more-perf-runs.jsonl

uv run python performance/benchmarks.py run \
    --dataset performance/long_bench.vcz \
    --shape long \
    --output performance/results/long-more-perf-runs.jsonl

[jeromekelleher]

These are the latest perf numbers reflecting 0.2.0 release (at #430)

Note that these benchmarks were run on different hardware to the above, so shouldn't be over interpreted.

Environment

Run on host claude-worker1: 4 logical CPUs (4 physical), 7.7 GiB RAM, Linux. Software: vcztools 0.1.3.dev417 at git eccbd33 (branch fixup-perf-suite), zarr 3.1.6, numpy 2.4.4. Each matrix cell is the median of 3 repeats; each microbench cell is the median of 10 repeats.

Wide dataset (496,414 variants × 100,000 diploid samples)

This is the headline workload. Retrieval throughput is reported as data_rate_mib_s (summed retrieved-chunk bytes ÷ elapsed); output tasks report output_rate_mib_s (bytes written ÷ elapsed).

Retrieval — MiB/s (median of 3)

task	local-dir	local-dir-zv3	local-zip	obstore-file	icechunk	local-http
iter_no_fields	0	0	0	0	0	0
iter_info_only	9	10	2	10	12	5
region_variant_position	0	0	1	0	1	0
region_info_and_format	498	463	417	633	648	244
first_variant_chunks	1089	1068	883	1035	1053	976
first_samples_chunk	3014	2996	2172	2913	2867	2084
first_samples_chunk_slice	2825	2906	2106	2941	3049	2622
fmt_fields	3784	3569	2068	3428	3250	2819
fmt_fields_filtered	241	243	235	242	244	240
filter_info_dp_gt_80	1	1	0	1	1	0
filter_info_dp_gt_80_genotypes	48	48	21	50	56	14
region_filter_format_gq_gt_50	171	174	160	163	154	150
region_and_sample_subset	53	61	56	57	76	50

The very low MiB/s rows (iter_no_fields, region_variant_position, filter_info_dp_gt_80, iter_info_only) are not slow — they retrieve very few bytes, so MiB/s is the wrong lens; they finish in well under a second (see elapsed below). The bulk-genotype reads are the meaningful throughput numbers: fmt_fields and the first_samples_chunk* family land at ~2.8–3.8 GiB/s, and first_variant_chunks at ~1.0–1.1 GiB/s, which is close to a 4-core blosc decompression ceiling.

Retrieval — elapsed seconds (median of 3)

task	local-dir	local-dir-zv3	local-zip	obstore-file	icechunk	local-http
iter_no_fields	0.00	0.00	0.00	0.00	0.00	0.00
iter_info_only	2.20	2.17	8.36	2.06	1.71	4.00
region_variant_position	0.01	0.01	0.00	0.01	0.01	0.02
region_info_and_format	0.95	1.02	1.13	0.75	0.73	1.94
first_variant_chunks	0.88	0.89	1.08	0.92	0.91	0.98
first_samples_chunk	3.14	3.16	4.36	3.25	3.30	4.54
first_samples_chunk_slice	3.35	3.26	4.50	3.22	3.11	3.61
fmt_fields	0.76	0.80	1.38	0.83	0.88	1.01
fmt_fields_filtered	7.14	7.06	7.29	7.10	7.03	7.14
filter_info_dp_gt_80	0.86	0.89	4.53	0.75	0.68	1.50
filter_info_dp_gt_80_genotypes	3.94	3.91	9.00	3.78	3.35	13.37
region_filter_format_gq_gt_50	0.55	0.55	0.59	0.58	0.62	0.63
region_and_sample_subset	0.04	0.03	0.03	0.03	0.03	0.04

Output tasks (local-dir backend)

task	output MiB/s	bytes_written (MiB)	elapsed_s	records	peak_rss_mb
output_vcf	219	919.5	4.19	982	2787
output_plink	100	471.9	4.71	19688	1770
output_bed_stream	115	469.4	4.08	19688	1965
output_bed_stream_filter_m2	115	469.4	4.09	19688	2097
output_bgen_l0	299	1408.3	4.71	4912	2939
output_bgen_l1	22	62.6	2.81	4912	2653
output_bgen_l6	5	41.7	9.17	4912	2664
output_bgen_stream_filter_m2	492	1406.7	2.86	4912	3076

VCF text encode runs at ~219 MiB/s; plink and the BED stream sit at ~100–115 MiB/s. BGEN is dominated by zlib: stored/level-0 hits ~300–490 MiB/s, but level 1 drops to ~22 MiB/s and level 6 to ~5 MiB/s — the expected compression-vs-speed tradeoff, which is why level 9 is excluded from the default sweep. Peak RSS for the wide output tasks is ~1.8–3.1 GiB; this is the largest memory consumer in the suite and worth keeping an eye on.

Encoder microbenchmark sweep (1000 variants × 100,000 samples, 190.7 MiB input)

This isolates encoder throughput from storage I/O and sweeps worker threads against the per-block byte target. MiB/s is reported on the output-byte basis (median of 10).

encoder	1 thread	best (threads)	notes
plink	210	838 (4t, 1 MiB)	scales ~4× to 4 threads, then flat — bound by 4 cores
vcf	166	499 (4t)	~3× to 4 threads; VcfWriter ignores block-bytes
bgen (C)	537	1222 (4t/8t)	fastest encoder; ~2.3× to 4 threads, block size ~irrelevant
bgen (python)	305	755 (4t)	~2.5× to 4 threads; ~40% of the C encoder

On the input-byte basis plink reaches ~6.7 GiB/s at 4 threads with the 1 MiB block target. Across all encoders, throughput saturates at 4 threads (this is a 4-core box) and the 8-thread column shows no further gain — often a slight regression from oversubscription. The 40 MiB block-bytes column consistently underperforms for plink and the python BGEN encoder (fewer, larger blocks reduce parallelism), confirming the smaller 1–10 MiB block targets are the right default on this core count.