Skip to content

feat(tpcds): picodata schema + data-load (query port pending) - #100

Merged
Cianidos merged 2 commits into
mainfrom
feat/tpcds-pico
Jul 20, 2026
Merged

feat(tpcds): picodata schema + data-load (query port pending)#100
Cianidos merged 2 commits into
mainfrom
feat/tpcds-pico

Conversation

@Cianidos

@Cianidos Cianidos commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds Picodata as a load target for TPC-DS. Schema + 24-table bulk load verified end-to-end. The 99-query suite is out of scope for this PR — sbroad (picodata's SQL engine) cannot parse it yet (blocker documented below with examples). pico.sql ships as the mechanical-transform skeleton and the port starting point.

Also ships a cross-driver fix found along the way: the bulk-insert bind-parameter clamp moved centrally so Picodata (and a latent YDB path) stop overflowing on wide tables.

What works (picodata)

./build/stroppy run tpcds/tpcds -d pico \
    -D url=postgres://admin:T0psecret@localhost:1336/admin \
    -e SCALE_FACTOR=1 --no-steps workload
  • drop_schema / create_schema / create_indexes: all 24 tables + indexes, green.
  • load_data: 0/24 table errors, all 24 tables populated.
  • Schema (schema.pico.sql): char(N)varchar(N), datedatetime, a PRIMARY KEY per Tarantool space (engine-required), no FK. Plain DROP TABLE IF EXISTS (picodata has no CASCADE). ANALYZE skipped (sbroad rejects it).
  • tpcds.ts: picodata wired into the dialect maps; drop/analyze branches handled.

What does NOT work — query execution (documented blocker)

sbroad (picodata 26.3) accepts only explicit INNER JOIN ... ON. It has no comma join, no CROSS JOIN, no PRAGMA (so no YDB-style AnsiImplicitCrossJoin), and no session setting for implicit cross joins. TPC-DS query templates use comma joins pervasively, so ~88 of 95 queries fail at parse time.

Concrete examples, run against a live picodata 26.3 container:

SELECT count(*) FROM store_sales, date_dim WHERE ss_sold_date_sk = d_date_sk;
-- ERROR: sbroad: rule parsing error: --> 1:33
--   expected EOI, IndexedByExpr, or DqlOption

SELECT count(*) FROM store_sales CROSS JOIN date_dim WHERE ss_sold_date_sk = d_date_sk;
-- ERROR: sbroad: rule parsing error: --> 1:54
--   expected Identifier

PRAGMA AnsiImplicitCrossJoin;
-- ERROR: sbroad: rule parsing error: --> 1:1
--   expected EOI, ... SetParam, SetTransaction, ...

Dominant query failures (all the parser hitting , in a FROM list): expected EOI ×47, expected OrderBy ×21, expected IndexedByExpr ×21.

Secondary gaps behind the comma-join wall: rollup/grouping sets (7 q), INTERSECT (4 q; EXCEPT works), FULL OUTER JOIN (2 q), rank()/dense_rank()/lag()/lead() (8 q, engine-missing — already skipped+logged in tpcds.ts), correlated subqueries, datetime+int, round(). pico.sql's header lists them.

pico.sql is kept (not stubbed) as the port starting point: date/char/cast fixes and date+N offsets are already baked. Run the workload step only after that port lands; today use --no-steps workload.

Included fix — central batch clamp (independent value)

The 65535 bound-parameter cap (pgwire extended protocol = MySQL prepared Error 1390) was enforced only in the MySQL driver (capBatchByColumns). Moved it into the shared sqldriver.RunBulkInsert so every sql.DB-backed dialect is protected:

  • picodata: date_dim (28 cols), catalog_sales/web_sales (34 cols) previously aborted with extended protocol limited to 65535 parameters and loaded 0 rows. Fixed.
  • ydb plain_bulk: was unclamped (latent; tpcds uses native BulkUpsert so never triggered). Now protected.
  • mysql: behavior identical (central clamp subsumes the per-driver helper, which is removed).

One constant (maxBoundParameters = 65535), one place. Net: less code, wider coverage.

Verification

  • make linter_fix → 0 issues.
  • make tests → all packages ok (race + coverage).
  • Live picodata 26.3 docker, TPC-DS load 0/24 table errors at both SCALE_FACTOR=0.01 and SCALE_FACTOR=1 (SF=1 → 20.45M rows; all 24 tables populated, fact tables at correct SF1 magnitude: inventory 11.7M, store_sales 2.88M, catalog_sales 1.44M, web_sales 719K, customer_demographics 1.92M).

Follow-ups (not in this PR)

  • TPC-DS picodata query port — separate epic. Preferred path: AST comma-join rewrite (parse → lift join predicates from WHERE into ON → nested INNER JOIN → deparse), then targeted rollup/intersect/full-outer rewrites; or upstream sbroad implicit-cross-join support.
  • File picodata upstream issue for comma-join / AnsiImplicitCrossJoin parity with YDB (would also fix tpch-pico queries, which were never actually validated — TestTpchLoadOnPicodata only asserts queries are logged, not that they pass).

Refs #86 (partial: schema + load + wiring landed; query dialect still open — blocked on sbroad comma-join gap).

Cianidos added 2 commits July 21, 2026 00:32
Picodata (sbroad) TPC-DS schema and 24-table bulk load. Query execution
is not yet supported: sbroad has no comma-join, no CROSS JOIN, and no
implicit-cross-join pragma/session setting, and ~88 of 95 TPC-DS queries
use comma joins (template style), so they fail at parse time. pico.sql
ships as the mechanical-transform skeleton (date/char/cast fixes, date+N
baked) and the port starting point; load picodata with `--no-steps workload`.

Schema (schema.pico.sql): char->varchar, date->datetime, a PRIMARY KEY
per Tarantool space (required), no FK. drop_schema uses plain DROP TABLE
IF EXISTS (picodata has no CASCADE); create_schema + create_indexes mirror
the pg layout; ANALYZE is skipped (sbroad rejects it).

Bulk-insert batch-size-by-column-count clamp (the 65535 bound-parameter
cap shared by pgwire and MySQL prepared statements) moved out of the
mysql-only capBatchByColumns into the shared sqldriver.RunBulkInsert.
Fixes picodata wide-table load (date_dim 28 cols, catalog_sales/web_sales
34 cols aborted with "extended protocol limited to 65535 parameters"),
protects a latent ydb plain_bulk gap, and leaves mysql behavior identical.
@Cianidos
Cianidos merged commit d765d31 into main Jul 20, 2026
8 checks passed
Cianidos added a commit that referenced this pull request Jul 22, 2026
TPC-DS ships 95 of 103 queries on Picodata in 5.7.0 (PR #100 ported
them; the line still described the initial load-only plan). Also clear
the [Unreleased] block that duplicated [5.7.0] after the release copy.
@Cianidos
Cianidos deleted the feat/tpcds-pico branch July 29, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant