ClickHouse: add PQS/CERT/CODDTest oracles and lift type system to ADT

[fm4v]

The ClickHouse provider previously only supported the five TLP variants (Where / Distinct / GroupBy / Aggregate / Having) and NoREC, and its schema generator only emitted two column types (Int32 and String). This PR

1. adds three more general-purpose oracles (PQS, CERT, CODDTest) — each tracking its source paper as faithfully as the existing ClickHouse expression generator allows, and
2. lifts the type system to a recursive ADT with a capability layer so the schema generator can emit Nullable(T) and LowCardinality(T) columns, and every dispatch site that previously AssertionError'd on anything outside {Int32, String} is re-routed through the new capabilities.

---

1. New oracles

PQS — Pivoted Query Synthesis (Rigger & Su, OSDI '20)

The classical SQLancer PQS implementation requires every AST node to expose a Java-side getExpectedValue() that mirrors the DBMS' evaluation semantics. ClickHouse's expression AST does not provide that for most of the generated tree, and reproducing ClickHouse's coercion / NULL / arithmetic rules in Java would be open-ended. This implementation delegates rectification to the server: for each randomly-generated predicate the pivot row's values are embedded as literals in a one-row subquery and ClickHouse itself evaluates the predicate. Based on TRUE / FALSE / NULL the predicate is kept, negated, or wrapped in IS NULL. Containment is checked with INTERSECT.

CERT — Cardinality Estimation Restriction Testing (Ba & Rigger, ICSE '24, DOI)

Generates a random query Q, derives a strictly more restrictive Q' from it, and asserts the cardinality restriction monotonicity property EstCard(Q', D) ≤ EstCard(Q, D). The estimate is read from EXPLAIN ESTIMATE and the query is never executed — this oracle tests the estimator, not the runtime. Mutations are one-directional and restrictive: AND-tighten the WHERE, drop an OR operand from an existing disjunction, or promote ALL → DISTINCT. Structural-similarity gate on EXPLAIN PLAN skips runs where the plan diverges enough to make the two estimates incomparable.

EXPLAIN ESTIMATE only meaningfully responds to PK/partition-key/index filters on MergeTree tables. For non-MergeTree engines or ORDER BY tuple() tables it returns an empty result; those attempts are dropped via IgnoreMeException.

CODDTest — Constant-Optimization-Driven Database System Testing (Zhang & Rigger, SIGMOD '25, DOI)

For a query Q with a sub-expression E, builds an auxiliary A that evaluates E in isolation, reads the resulting constant V, then builds a folded query F by substituting V for E in Q. Constant folding is semantics-preserving, so any discrepancy between Q's and F's result sets is a logic bug.

All three flavors from the paper land: constant expression (Section 3.1 case 1), scalar non-correlated subquery (Section 3.1 case 2), and dependent expression with a CASE mapping (Section 3.2). The outer predicate template varies — bare comparison, AND/OR compounds, NOT — so phi passes through richer constant-folding paths than a fixed col op phi.

---

2. Type-system foundation v1 (replaces the flat (ClickHouseDataType, String) model)

Plan: docs/plans/2026-05-16-001-feat-clickhouse-type-system-foundation-plan.md. Brainstorm origin: docs/brainstorms/clickhouse-type-system-foundation-requirements.md.

The previous model — ClickHouseLancerDataType wrapping a ClickHouseDataType enum + textual representation — could not encode parameters (so ClickHouseDataType.of("Decimal(9,2)") silently normalised to Decimal) and could not express wrappers like Nullable or LowCardinality. Every oracle bailed on anything outside {Int32, String} because the constant emitters, cast paths, and oracle filters all defaulted to AssertionError.

This PR introduces:

• ClickHouseType — a sealed recursive ADT with four constructors:
  • Primitive(Kind) for atomic types (Int8…Int256, UInt8…UInt256, Float32, Float64, Bool, String, UUID, Date, Date32, IPv4, IPv6),
  • Nullable(inner) and LowCardinality(inner) with canWrap rules (no nested Nullable, no LC on Float/Bool/UUID/IPv*, etc.),
  • Unknown(raw) as a defensive fallback the reflection parser uses for any type string it does not recognise.
• Capability predicates on every term: isNumeric(), supportsLiteralEmission(), hasNullSemantics() (true iff the outer term is Nullable).
• ClickHouseTypeParser — a hand-written recursive-descent parser for ClickHouse type strings; recognises every primitive plus Nullable(…) and LowCardinality(…) and nested combinations, falls back to Unknown(raw) for everything else. Never throws.
• ClickHouseUnsupportedConstant — sentinel returned by ClickHouseCast.castToInt/castToReal/castToText/isTrue/convertInternal when a coercion is not defined for the input. Propagates through cast pipelines and raises IgnoreMeException on any attempt to compare, evaluate, or consume it. This replaces every default: throw new AssertionError(...) in ClickHouseCast.
• Wrapper-aware ClickHouseLancerDataType — adds getTypeTerm(): ClickHouseType while keeping getType(): ClickHouseDataType returning the root primitive (lossy compatibility for legacy callers; documented).
• Feature flags — --test-nullable-types and --test-lowcardinality-types, both default true, threaded through ClickHouseLancerDataType.getRandom(state) and ClickHouseColumn.createDummy(name, table, state).
• Constant-emission ADT dispatch — ClickHouseExpressionGenerator.generateConstant and ClickHouseSchema.getConstant now dispatch on the ADT term:
  • Primitive(kind) → per-kind emitter,
  • Nullable(inner) → with small probability emit NULL, else recurse,
  • LowCardinality(inner) → transparent, recurse,
  • Unknown → IgnoreMeException.
• CODDTest filter migration — the aggType != Int32 && aggType != String filters at ClickHouseCODDTestOracle.java:175-177,208-210 are rewritten as a isFoldableColumnTerm(ClickHouseType) capability check that accepts Int*/UInt*/Bool/String and any Nullable/LowCardinality wrapper of those. The local baseTypeName / parseType / renderLiteral string parsers are migrated onto ClickHouseTypeParser.
• CERT generator dispatch — generatorExprFor(ClickHouseType) composes generators:
  • Nullable(inner) wraps the inner generator with if(rand() % 10 = 0, NULL, …),
  • LowCardinality(inner) is transparent at INSERT (ClickHouse coerces),
  • Unknown throws IgnoreMeException.
• Table-generator clause validation — PARTITION BY / ORDER BY / SAMPLE BY expression generation now validates the result and retries up to 5 times before dropping the clause:
  • ORDER BY rejects all-constant expressions ("Sorting key cannot contain constants"),
  • PARTITION BY additionally rejects Float-column references ("Floating point partition key is not supported").
• Connection-level settings — ClickHouseProvider adds allow_suspicious_low_cardinality_types=1 to the JDBC URL when the LC flag is on, and ClickHouseTableGenerator adds allow_nullable_key=1 to the MergeTree SETTINGS clause so wrapped columns can appear in ORDER/PARTITION/SAMPLE keys.
• Error catalog additions — ClickHouseErrors gains the v1 type-family patterns surfaced by live smoke runs: ILLEGAL_TYPE_OF_ARGUMENT, LowCardinality conversion/nested-type rejections, SUSPICIOUS_TYPE_FOR_LOW_CARDINALITY, Partition key contains nullable columns, allow_nullable_key, CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN, and "Cannot convert NULL value to non-Nullable type".

TLP three-valued logic is explicitly deferred. The third-partition composition lives in sqlancer.common.oracle.TernaryLogicPartitioningOracleBase (cross-DBMS), and the requirements doc forbids common-module edits. TLP runs with enableNullable=true but the partitions remain 2-valued; that's a v1.0.5 follow-up (see the plan).

---

3. Other changes

• ClickHouseSchema.ClickHouseRowValue's constructor promoted from package-private to public so PQS in oracle/pqs/ can construct it for the base class' diagnostic logging.
• .github/workflows/main.yml:
  • Adds the seven new test classes to the enumerated -Dtest=… list for the clickhouse CI job.
  • Drops the 18 per-DBMS jobs we don't ship changes for (citus, cockroachdb, databend, datafusion, duckdb, hive, spark, hsqldb, mariadb, materialize, mysql, oceanbase, postgres, presto, sqlite, tidb, yugabyte, doris). Keeps misc (project-wide style/PMD/Checkstyle/SpotBugs + misc unit tests + naming convention check) and clickhouse.

---

Verification

Unit tests (89 total, all pass) — seven new test classes covering the ADT, parser, cast extension, generation surface, CODDTest filter, CERT generator, and TableGenerator validators. CI runs them on every push.

mvn -DskipTests=true -Dspotbugs.skip=true verify clean (formatter, Checkstyle, PMD). SpotBugs is broken on JDK 25 in this repo (Unsupported class file major version 69 from the bundled ASM); affects every class, not just new code.

Live SQLancer smoke against ClickHouse 26.5.1.111, 4 threads × 10 minutes, all four PR-introduced oracles (TLP / NoREC / PQS / CODDTest, both type flags ON):

Metric	Value
Total queries	70,299 (~117 q/s average)
Successful statements	55% (typical for SQLancer's adversarial generator)
AssertionError count	0
Threads shut down on errors	0 / 4
Process exit code	0 (clean self-timeout)
Nullable columns observed in CREATE TABLE	✅ Nullable(String), Nullable(Int32)
LowCardinality columns observed	✅ LowCardinality(Int32)

The live smoke also caught three v1-introduced rejections mid-run; all are fixed in this PR:

1. LowCardinality(Int32) rejected by default (SUSPICIOUS_TYPE_FOR_LOW_CARDINALITY) — fixed by adding allow_suspicious_low_cardinality_types=1 to the JDBC URL.
2. PARTITION/ORDER BY referencing a Nullable column rejected (ILLEGAL_COLUMN, allow_nullable_key) — fixed by adding allow_nullable_key=1 to MergeTree SETTINGS.
3. CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN from MATERIALIZED clauses casting NULL → non-Nullable target — added to ClickHouseErrors.

The final post-fix run completed all 600 seconds with zero unhandled errors.

---

Downstream

The ClickHouse repo will pick these up via a Dockerfile pin bump once this lands: ClickHouse/ClickHouse#104988.

[CLAassistant]

All committers have signed the CLA.