Skip to content

Release v0.4.2 - ClickBench submission

Latest

Choose a tag to compare

@alberttwong alberttwong released this 09 Oct 19:53
· 13 commits to master since this release
ae09095

Multicore query execution

v0.4.2 adds automatic bounded parallel document scans so eligible queries can use multiple Trino drivers and CPU cores. The planner selects scan splits from matching-row estimates, available CPU capacity and connection-pool limits. Small tables and selective filters retain a single split. Redis Query Engine workers can be tuned independently; parallel scans do not accelerate every query or replace server-side aggregation.

This release also improves cursor prefetch, batched exact BIGINT reads, query metrics and timeout controls. It preserves correctness when Redis reports incomplete results or cursor errors. Query planning improvements include bounded late materialization for TopN, exact large-integer lookups and safe Trino aggregation for high-cardinality groups.

ClickBench submission

A complete Redis Cloud entry has been submitted in ClickHouse/ClickBench PR2443. All 43 unchanged queries ran three times on 99,997,497 rows, and all 129 outputs passed independent full-data correctness validation with verified CSV checksums.

The benchmark pins connector revision d83d504d65cd5b3bd14fd9bd559f5acad6ffedf2; the release includes the same connector code plus the merged benchmark documentation. Q24 and Q34, which previously timed out or exceeded the aggregate-group limit, now pass all three attempts. Best warm times are 2.991580s for Q24 and 62.044117s for Q34.

The run used three r8g.16xlarge Redis hosts and an r7a.4xlarge Trino runner in the same AWS region/VPC/physical AZ, with 1 TB all RAM, 40 shards and OSS Cluster API. It is tagged no-cold because Redis remained running. The submission is pending upstream review; these are not yet accepted leaderboard results.

Verified backup initialization took 545.174150s versus 1036.643924s for separately measured fresh ingestion plus indexing. Restore is setup and is not reported as ClickBench ingestion time. Full results, methodology and teardown evidence remain available. Dedicated benchmark compute has been removed.

Compatibility

Requires Trino 483 and Java 25. See the connector documentation for parallel-scan eligibility and configuration.

Changelog

🔄️ Changes

  • 37baee4 add instrumented ClickBench loader with batch encoding

📝 Documentation

  • c9870d2 confirm benchmark teardown and retained backup
  • 36c8124 record complete validated PR154 ClickBench sweep and backup restore
  • 695f38a retain original acceptance CSVs and validation evidence
  • b4c99da validate PR154 Q24 and Q34 on full ClickBench data

  • ab0616d Improve ClickBench TopN, exact lookups, and aggregation planning
  • 302ddb3 Retain verified benchmark teardown evidence
  • 22b62fb Document completed latest-code full-data ClickBench sweep
  • 606b3cf Document controlled Redis Software benchmark results
  • bbd5cf4 Verify effective workers when switching QPF modes
  • a892883 Default to Redis Software or Cloud and add controlled QPF benchmark
  • da4a648 Record final five-million-row issue acceptance and QPF comparison
  • 217b917 Merge remote-tracking branch 'origin/master' into albert/5m-cloud-acceptance
  • 12ec447 Add automatic bounded parallel Redis scans
  • 1ffbb69 Merge remote-tracking branch 'origin/master' into albert/5m-cloud-acceptance
  • a760085 Use five million rows for further PR validation
  • 65fb4c1 Merge remote-tracking branch 'origin/master' into albert/cursor-wrapper-final-acceptance
  • 2bde226 Retain separate merged-artifact Q3 Q5 and Q6 acceptance
  • 74a7d87 Preserve wrapped cursor errors and verify initial warning cleanup
  • bdd2564 Record exact full-data Q3 and Q5 acceptance
  • 4339176 Reject warned aggregate results and clean up initial cursors
  • 94d3267 Recognize wrapped cursor disconnects and reject incomplete health probes
  • d3a53b1 Prevent unsafe cursor replay and retain Cloud validation
  • 1acd81d Record local Params projection validation
  • 799a51d Fix LOAD field references and preserve Query Engine failures
  • 811b1c6 Batch exact BIGINT reads and overlap cursor prefetch for Q5
  • c9f1b91 Record validated 54x mixed-aggregation improvement after patches
  • f3b1aa3 Label historical ClickBench comparisons and record patched rerun
  • 6502292 Merge master and preserve both benchmark documentation sections
  • f8a227f Expose per-query Redis read and cursor metrics
  • 5665370 Push down precision-safe integer widening in aggregations
  • 920f91e Document Cloud API provisioning blocker and add mode diagnostic
  • 766f2ff Document initial Redis Cloud ClickBench results and comparisons
  • 83d05fc Allow per-query Redis aggregation timeout overrides
  • aa32d4d Keep NUMERIC BIGINT grouping exact in Trino
  • 0747041 Releasing version 0.4.1