Skip to content

Commit fbedbf0

Browse files
dfa1claude
andcommitted
perf(parquet): default import chunk size 131072 -> 65536 (#299)
Commit 962520a raised the importer's default chunk from 65536 to 131072 for a size win, but that predates the FSST + global-dict work. Re-measuring nyc-311 shows the premise reversed: finer chunks now compress better because per-chunk cost competition and FSST training adapt more locally. Sweep on nyc-311 (18.49M rows): 131072=2000.00MB, 65536=1934.23MB, 32768=1992.06MB — 65536 is the sweet spot (below 64k, per-chunk overhead outweighs the gain). Taxi confirms no regression (40.77 -> 40.63 MB). 65536 also realigns the importer default with WriteOptions' own default. Updates the README taxi compression figure to 40.6 MB accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 6b9d101 commit fbedbf0

2 files changed

Lines changed: 8 additions & 2 deletions

File tree

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ same Parquet file (47.6 MB), cascading depth 3, Apple M5:
4242
| Implementation | Output size | vs Parquet |
4343
|----------------|-------------|------------|
4444
| vortex-jni¹ | 47.0 MB | −1.3% |
45-
| **vortex-java** | **40.8 MB** | **−14.2%** |
45+
| **vortex-java** | **40.6 MB** | **−14.6%** |
4646

4747
¹ vortex-jni figure measured against 0.78.0 and not re-run (the JNI API has no
4848
Parquet-import path); the vortex-java and Parquet figures were re-measured 2026-07-24.

parquet/src/main/java/io/github/dfa1/vortex/parquet/ImportOptions.java

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,13 @@ public record ImportOptions(
1212
WriteOptions writeOptions
1313
) {
1414
public static ImportOptions defaults() {
15-
return new ImportOptions(131_072, List.of(), null, WriteOptions.cascading(3));
15+
// 65536 physical rows per chunk, matching WriteOptions' default and the Rust reference's
16+
// 32k-64k granularity. This was briefly raised to 131072 for a size win (962520ac), but the
17+
// FSST + global-dict work (#299) reversed that: finer chunks now compress better on
18+
// real-world data (nyc-311 -66 MB, taxi -0.14 MB) as their per-chunk cost competition and
19+
// FSST training adapt more locally. 32768 is worse than 65536 — per-chunk overhead outweighs
20+
// the finer adaptation below 64k.
21+
return new ImportOptions(65_536, List.of(), null, WriteOptions.cascading(3));
1622
}
1723

1824
/// Restrict import to specific columns. Empty list = all columns.

0 commit comments

Comments
 (0)