Commit fbedbf0
perf(parquet): default import chunk size 131072 -> 65536 (#299)
Commit 962520a raised the importer's default chunk from 65536 to 131072 for a
size win, but that predates the FSST + global-dict work. Re-measuring nyc-311
shows the premise reversed: finer chunks now compress better because per-chunk
cost competition and FSST training adapt more locally.
Sweep on nyc-311 (18.49M rows): 131072=2000.00MB, 65536=1934.23MB, 32768=1992.06MB
— 65536 is the sweet spot (below 64k, per-chunk overhead outweighs the gain).
Taxi confirms no regression (40.77 -> 40.63 MB). 65536 also realigns the importer
default with WriteOptions' own default.
Updates the README taxi compression figure to 40.6 MB accordingly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 6b9d101 commit fbedbf0
2 files changed
Lines changed: 8 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | | - | |
| 45 | + | |
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| |||
Lines changed: 7 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
16 | 22 | | |
17 | 23 | | |
18 | 24 | | |
| |||
0 commit comments