feat(tpcds): picodata schema + data-load (query port pending) - #100
Merged
Conversation
Picodata (sbroad) TPC-DS schema and 24-table bulk load. Query execution is not yet supported: sbroad has no comma-join, no CROSS JOIN, and no implicit-cross-join pragma/session setting, and ~88 of 95 TPC-DS queries use comma joins (template style), so they fail at parse time. pico.sql ships as the mechanical-transform skeleton (date/char/cast fixes, date+N baked) and the port starting point; load picodata with `--no-steps workload`. Schema (schema.pico.sql): char->varchar, date->datetime, a PRIMARY KEY per Tarantool space (required), no FK. drop_schema uses plain DROP TABLE IF EXISTS (picodata has no CASCADE); create_schema + create_indexes mirror the pg layout; ANALYZE is skipped (sbroad rejects it). Bulk-insert batch-size-by-column-count clamp (the 65535 bound-parameter cap shared by pgwire and MySQL prepared statements) moved out of the mysql-only capBatchByColumns into the shared sqldriver.RunBulkInsert. Fixes picodata wide-table load (date_dim 28 cols, catalog_sales/web_sales 34 cols aborted with "extended protocol limited to 65535 parameters"), protects a latent ydb plain_bulk gap, and leaves mysql behavior identical.
Cianidos
added a commit
that referenced
this pull request
Jul 22, 2026
TPC-DS ships 95 of 103 queries on Picodata in 5.7.0 (PR #100 ported them; the line still described the initial load-only plan). Also clear the [Unreleased] block that duplicated [5.7.0] after the release copy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Picodata as a load target for TPC-DS. Schema + 24-table bulk load verified end-to-end. The 99-query suite is out of scope for this PR — sbroad (picodata's SQL engine) cannot parse it yet (blocker documented below with examples).
pico.sqlships as the mechanical-transform skeleton and the port starting point.Also ships a cross-driver fix found along the way: the bulk-insert bind-parameter clamp moved centrally so Picodata (and a latent YDB path) stop overflowing on wide tables.
What works (picodata)
./build/stroppy run tpcds/tpcds -d pico \ -D url=postgres://admin:T0psecret@localhost:1336/admin \ -e SCALE_FACTOR=1 --no-steps workloaddrop_schema/create_schema/create_indexes: all 24 tables + indexes, green.load_data: 0/24 table errors, all 24 tables populated.schema.pico.sql):char(N)→varchar(N),date→datetime, aPRIMARY KEYper Tarantool space (engine-required), no FK. PlainDROP TABLE IF EXISTS(picodata has noCASCADE).ANALYZEskipped (sbroad rejects it).tpcds.ts: picodata wired into the dialect maps; drop/analyze branches handled.What does NOT work — query execution (documented blocker)
sbroad (picodata 26.3) accepts only explicit
INNER JOIN ... ON. It has no comma join, noCROSS JOIN, noPRAGMA(so no YDB-styleAnsiImplicitCrossJoin), and no session setting for implicit cross joins. TPC-DS query templates use comma joins pervasively, so ~88 of 95 queries fail at parse time.Concrete examples, run against a live picodata 26.3 container:
Dominant query failures (all the parser hitting
,in a FROM list):expected EOI×47,expected OrderBy×21,expected IndexedByExpr×21.Secondary gaps behind the comma-join wall:
rollup/grouping sets(7 q),INTERSECT(4 q;EXCEPTworks),FULL OUTER JOIN(2 q),rank()/dense_rank()/lag()/lead()(8 q, engine-missing — already skipped+logged intpcds.ts), correlated subqueries,datetime+int,round().pico.sql's header lists them.pico.sqlis kept (not stubbed) as the port starting point: date/char/cast fixes and date+N offsets are already baked. Run the workload step only after that port lands; today use--no-steps workload.Included fix — central batch clamp (independent value)
The 65535 bound-parameter cap (pgwire extended protocol = MySQL prepared Error 1390) was enforced only in the MySQL driver (
capBatchByColumns). Moved it into the sharedsqldriver.RunBulkInsertso every sql.DB-backed dialect is protected:date_dim(28 cols),catalog_sales/web_sales(34 cols) previously aborted withextended protocol limited to 65535 parametersand loaded 0 rows. Fixed.One constant (
maxBoundParameters = 65535), one place. Net: less code, wider coverage.Verification
make linter_fix→ 0 issues.make tests→ all packagesok(race + coverage).Follow-ups (not in this PR)
INNER JOIN→ deparse), then targeted rollup/intersect/full-outer rewrites; or upstream sbroad implicit-cross-join support.AnsiImplicitCrossJoinparity with YDB (would also fix tpch-pico queries, which were never actually validated —TestTpchLoadOnPicodataonly asserts queries are logged, not that they pass).Refs #86 (partial: schema + load + wiring landed; query dialect still open — blocked on sbroad comma-join gap).