Skip to content

[GLUTEN-3456][VL] Enable columnar table cache by default and extend benchmark coverage#12138

Merged
yaooqinn merged 1 commit into
apache:mainfrom
yaooqinn:users/kentyao/columnar-table-cache-benchmark-sources
May 26, 2026
Merged

[GLUTEN-3456][VL] Enable columnar table cache by default and extend benchmark coverage#12138
yaooqinn merged 1 commit into
apache:mainfrom
yaooqinn:users/kentyao/columnar-table-cache-benchmark-sources

Conversation

@yaooqinn
Copy link
Copy Markdown
Member

@yaooqinn yaooqinn commented May 26, 2026

[GLUTEN-3456][VL] Enable columnar table cache by default and extend benchmark coverage

What changes were proposed in this pull request?

  1. Flip spark.gluten.sql.columnar.tableCache default from false to true.
  2. Extend ColumnarTableCacheBenchmark to cover:
    • 3 sources: parquet (Velox-native columnar), csv, json (row-based fallback per GLUTEN-3456).
    • 2 schema shapes: 5-col numeric mix, and a 16-col x ~200-char wide-string shape (the GLUTEN-3488 hazard).
    • Cases: count / column-pruning / filter for numeric; count for wide-string.
  3. Regenerate ColumnarTableCacheBenchmark-results.txt with the new matrix.

Why are the changes needed?

GLUTEN-3456 raised the concern that for row-based sources (csv/json) the
Velox columnar cache would lose to vanilla Spark because of the R2C/C2R
conversion tax. The extended benchmark shows the opposite is true on
Velox today:

Case disable (ms) enable (ms) speedup
numeric/parquet count 17377 2975 5.84x
numeric/parquet column pruning 20768 3778 5.50x
numeric/parquet filter 22681 4242 5.35x
numeric/csv count 40502 30146 1.34x
numeric/csv column pruning 42245 30667 1.38x
numeric/csv filter 43929 31077 1.41x
numeric/json count 44659 28467 1.57x
numeric/json column pruning 46961 29230 1.61x
numeric/json filter 49106 29061 1.69x
wide-string/parquet count 40888 11863 3.45x
wide-string/csv count 82433 86708 0.95x
wide-string/json count 70729 54856 1.29x

11 / 12 cases improve. The only regression is wide-string/csv count
(-5%), where the Arrow CSV scan + R2C cost on a 16-col x ~200-char shape
slightly outweighs the cache benefit. Given how narrow that corner is,
flipping the default to true is the right trade-off; users hitting
that shape can still set spark.gluten.sql.columnar.tableCache=false.

Hardware: Intel(R) Xeon(R) Platinum 8473C, Linux WSL2, JDK 17. Numbers
are 3-iter Best Time from the regenerated golden file.

How was this patch tested?

  • Re-ran ColumnarTableCacheBenchmark in three modes (vanilla / Gluten
    off / Gluten on) and regenerated the golden file.
  • ./build/mvn -Pbackends-velox -Pspark-3.5 spotless:apply clean.

Generated-by: Claude Opus 4.7

Related issue: #3456

…enchmark coverage

### What changes were proposed in this pull request?

1. Flip `spark.gluten.sql.columnar.tableCache` default from `false` to `true`.
2. Extend `ColumnarTableCacheBenchmark` to cover:
   - 3 sources: `parquet` (Velox-native columnar), `csv`, `json` (row-based fallback per GLUTEN-3456).
   - 2 schema shapes: 5-col numeric mix, and a 16-col x ~200-char wide-string shape (the GLUTEN-3488 hazard).
   - Cases: `count` / column-pruning / filter for numeric; `count` for wide-string.
3. Regenerate `ColumnarTableCacheBenchmark-results.txt` with the new matrix.

### Why are the changes needed?

GLUTEN-3456 raised the concern that for row-based sources (csv/json) the
Velox columnar cache would lose to vanilla Spark because of the R2C/C2R
conversion tax. The extended benchmark shows the opposite is true on
Velox today:

| Case                                  | disable (ms) | enable (ms) | speedup |
|---------------------------------------|-------------:|------------:|--------:|
| numeric/parquet count                 |        17377 |        2975 |   5.84x |
| numeric/parquet column pruning        |        20768 |        3778 |   5.50x |
| numeric/parquet filter                |        22681 |        4242 |   5.35x |
| numeric/csv count                     |        40502 |       30146 |   1.34x |
| numeric/csv column pruning            |        42245 |       30667 |   1.38x |
| numeric/csv filter                    |        43929 |       31077 |   1.41x |
| numeric/json count                    |        44659 |       28467 |   1.57x |
| numeric/json column pruning           |        46961 |       29230 |   1.61x |
| numeric/json filter                   |        49106 |       29061 |   1.69x |
| wide-string/parquet count             |        40888 |       11863 |   3.45x |
| wide-string/csv count                 |        82433 |       86708 |   0.95x |
| wide-string/json count                |        70729 |       54856 |   1.29x |

11 / 12 cases improve. The only regression is `wide-string/csv count`
(-5%), where the Arrow CSV scan + R2C cost on a 16-col x ~200-char shape
slightly outweighs the cache benefit. Given how narrow that corner is,
flipping the default to `true` is the right trade-off; users hitting
that shape can still set `spark.gluten.sql.columnar.tableCache=false`.

Hardware: `Intel(R) Xeon(R) Platinum 8473C`, Linux WSL2, JDK 17. Numbers
are 3-iter Best Time from the regenerated golden file.

### How was this patch tested?

- Re-ran `ColumnarTableCacheBenchmark` in three modes (vanilla / Gluten
  off / Gluten on) and regenerated the golden file.
- `./build/mvn -Pbackends-velox -Pspark-3.5 spotless:apply` clean.

Generated-by: Claude Opus 4.7
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions github-actions Bot added CORE works for Gluten Core VELOX labels May 26, 2026
@yaooqinn yaooqinn merged commit 2563a40 into apache:main May 26, 2026
62 checks passed
@yaooqinn yaooqinn deleted the users/kentyao/columnar-table-cache-benchmark-sources branch May 26, 2026 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core VELOX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants