馃 read_xarray_table(..., batch_size=...) exposes the source RecordBatch size, but XarrayContext.from_dataset() and _from_dataset() currently do not pass that option through.
This matters for dense-grid consumers that use the convenient context API. A quarter-degree (lat=720, lon=1440) time plane has 1,036,800 rows, so the current 65,536-row default emits about 16 source batches per time partition. A larger batch may reduce Python/Arrow crossings and downstream batch-coalescing overhead; smaller batches may be preferable under memory pressure.
Suggested API:
ctx.from_dataset(
"canonical",
ds,
chunks={"time": 1},
batch_size=262_144,
)It should be threaded through heterogeneous dimension groups as well. The default should remain unchanged; this issue is about making the existing reader control available from the primary API and documenting the latency/memory tradeoff.
馃
read_xarray_table(..., batch_size=...)exposes the source RecordBatch size, butXarrayContext.from_dataset()and_from_dataset()currently do not pass that option through.This matters for dense-grid consumers that use the convenient context API. A quarter-degree
(lat=720, lon=1440)time plane has 1,036,800 rows, so the current 65,536-row default emits about 16 source batches per time partition. A larger batch may reduce Python/Arrow crossings and downstream batch-coalescing overhead; smaller batches may be preferable under memory pressure.Suggested API:
It should be threaded through heterogeneous dimension groups as well. The default should remain unchanged; this issue is about making the existing reader control available from the primary API and documenting the latency/memory tradeoff.