Complete psycopg 3.3 client generation with shared async/sync types#16
Merged
Conversation
….Sigs
Pins gen/Deps/Contract.dhall (new) and bumps gen/Deps/Sdk.dhall to gen-sdk
v2.0.0, removing the gen/Deps/package.dhall barrel. Rewires
Structures/CustomKind.dhall and all 11 Interpreters/*.dhall plus all 10
Templates/*.dhall to import Lude/Prelude/Contract/Sdk directly instead of
through the barrel, replacing the local gen/Algebras/{Interpreter,Template}
sig-constructors with gen-sdk's Sdk.Sigs.interpreter / Sdk.Sigs.template
(now deleted). Matches the shape java.gen's current release already uses.
Query.dhall, Result.dhall, and ResultColumns.dhall keep their pre-existing
bare-export shape (like Member.dhall/ParamsMember.dhall) instead of adopting
Sdk.Sigs.interpreter: their `run` functions take 3/4 curried arguments
(config, lookup, [rowClassName], input), which doesn't fit Sigs.interpreter's
fixed 2-argument Config -> Input -> Result shape. This extends the
Member/ParamsMember exception already called out in the migration plan to
these three files, confirmed against their actual call sites in
Project.dhall/Query.dhall.
gen/compile.dhall and gen/Gen.dhall still reference the removed barrel and
are intentionally left broken until the next task rewrites the root entry
point (Config.dhall/compile.dhall -> Interpret.dhall, Gen.dhall ->
package.dhall).
….dhall Task 4 of the Dhall dependency/layout migration: consumes Interpreters/Project.dhall's Sdk.Sigs.interpreter-shaped run and produces the Sdk.Sigs.generator-built entry point, now that the Deps barrel is gone (Tasks 1-3). - gen/compile.dhall -> gen/Interpret.dhall: drops the outer Optional Config unwrap (Sdk.Sigs.generator substitutes defaultConfig for the omitted-block case), keeps the per-field Prelude.Optional.fold defaults and importName derivation; imports Contract/Prelude directly instead of through the deleted barrel. - gen/Gen.dhall -> gen/package.dhall: builds Sdk.Sigs.generator Config Config/default interpret, with Config/default supplying an all-None Config.
…mos/ Pure directory move for gen/ -> src/ (relative imports inside are unaffected). tests/Exhaustive.dhall becomes demos/Exhaustive.dhall and is rewritten to use Sdk.Output.toFileMap, since Sdk.Sigs.generator-built modules no longer expose compileToFileMap.
…tive.dhall to demos/ Tasks 1-5 renamed gen/ to src/, gen/Gen.dhall to src/package.dhall, and tests/Exhaustive.dhall to demos/Exhaustive.dhall. This updates every external reference to the old paths across CI workflows, dev scripts, docs, and the bench harness. - .github/workflows/ci.yml, release.yml, build-contract-shell.sh: path refs. - README.md, AGENTS.md, DESIGN.md: doc path refs; DESIGN.md's section 10 tree diagram and entry-point description rewritten to match the actual post-move src/ layout (Interpret.dhall/package.dhall, no more Algebras/, Sdk.Sigs.generator instead of Sdk.module). - build.bash, mise.toml: dev-script path refs. - bench/generate.sh, bench/as-source.sh: gen/ -> src/ copies; the as-Source/plain-import sha256 substitution pair for gen-sdk is verified via `dhall hash` to be a no-op for v2.0.0 (both modes hash to the same value, unlike the old v0.11.0 pair), so only a comment remains; the lude pair is untouched since lude's pin is unchanged by this migration. - tests/fixture-project/project1.pgn.yaml: the fixture project's own `gen:` key still pointed at the now-deleted gen/Gen.dhall, which would break the harness and the `golden` mise task; updated to src/package.dhall.
tests/_harness.py's GEN_DIR (and its two consumers) still pointed the harness at the repo-root gen/ directory, which Task 5 deleted when it renamed gen/ to src/. Every test using the session-scoped generated_tree fixture, or building its own copytree in test_unsupported_types.py, would have hit FileNotFoundError on the very first pgn run. Renamed the constant to SRC_DIR (was misleading otherwise), pointed it at src/, matched the copied-tree destination directory name to what project1.pgn.yaml's `gen: ../../src/package.dhall` (fixed in the prior commit) actually resolves against, and fixed the two hardcoded yaml literals in test_unsupported_types.py plus conftest.py's docstring. Confirmed via `python3 -m py_compile` and a repo-wide grep for any remaining gen/-as-directory or GEN_DIR references in .py files (none).
Documents the gen-contract v4.0.1 / gen-sdk v2.0.0 migration and gen/->src/ layout restructure under # Upcoming, and updates Config.dhall's doc comment to reference Interpret.dhall instead of the old compile.dhall name.
…ltColumns These were declared but never exported (unlike Member.dhall/ParamsMember.dhall, which do export Run), left over from documenting each file's actual multi-arg call signature during the gen-sdk v2 migration. Flagged by final review as harmless but pointless; removing rather than exporting keeps these three files' tails exactly as they were pre-migration, matching the migration's "keep bare exports verbatim" instruction for this exception group.
This reverts commit 15c2836.
Matches the fixtures/ naming already used by the other generators (e.g. java.gen). Updates path references in build.bash, CI config, build-contract-shell.sh, DESIGN.md, and the file's own usage comment; regenerate_demo_output becomes regenerate_fixture_output. No behavior change. Also drops the stale .superpowers/sdd/ scratch directory (gitignored, untracked).
…/nikita-integration
H2: CONFIRM, 23 Python files and 1,394 total lines. Statements: 11 files, 11 SQL assignments, and 974 lines. The statement-line excess above 900 is provisional; C3 must reduce it to 900 or fewer.
Resolved psycopg: 3.3.4 Canaries: 16/16 passed across async and sync, covering scalar and array enums, enum grids, composite load/dump and arrays, nested composites, and sanitized rows. Strict: filesAnalyzed=1, errors=0, warnings=0, with generic BaseRowFactory inference and no unsafe casts. H1 verdict: CONFIRM. Exact Row, enum, composite, nested composite, and built-in list identities passed with async/sync parity.
H1 CONFIRM. Validated the adapter contract with psycopg 3.3.4. Support scalar and rank <=2 enum values, scalar and rank <=1 composite values, and nested scalar composites. Reject enum rank >2, composite rank >1, and custom-array composite fields. Register shared generated classes dependency-first in async and sync order: Mood -> Point2D -> TagValue -> ZCodecPayload -> ACodecWrapper. Tests: collision 1 passed with enum-only strict 0/0; fresh signatures 1 passed and all four applied paths unchanged; golden regenerated; generated 11 passed; unsupported 15 passed; adapter 2 passed; full live-PostgreSQL suite 36 passed with 0 failed or skipped.
H1 CONFIRM. Validated with psycopg 3.3.4 that adapter-backed enum, composite, array, and nested-composite values keep their exact generated identities in async and sync paths, with no generated decoder or cast layer. H2 CONFIRM. The combined package contains 25 Python files and 1,326 total lines. Its 11 statement files contain 802 lines and exactly 11 SQL assignments. Tests: generated plus collision 13 passed; unsupported 15 passed; adapter 2 passed; full live-PostgreSQL suite 37 passed with 0 failed or skipped; basedpyright strict 0 errors and 0 warnings.
This was referenced Jul 13, 2026
Closed
* Plan
* Prohibit local cross-repo deps
* Update the plan
* Bump gen-contract to v5.0.0 and gen-sdk to v3.0.0
Scalar.Custom now carries a CustomTypeRef (name, pgSchema, pgName, index)
instead of a bare Name, and Value flattens its optional array wrapper into
plain dimensionality/elementIsNullable fields. Updated every site touched by
these two shape changes (Scalar, Value, Member, ParamsMember, CustomType
interpreters) to compile against the new contract while preserving current
behavior exactly; the buildLookup/CustomKind.Lookup mechanism itself is
untouched (that migration is a separate follow-up task).
BLOCKED on verification: pgn v0.9.1 (pinned in mise.toml) hard-rejects the
new contract's major version before loading the generator at all
("Incompatible contract major version: 5. Expected 4."), so mise run test
and mise run golden cannot currently exercise this change end-to-end. See
.superpowers/sdd/task-1-report.md for the full failure output and analysis.
* Another plan
* python.gen: replace PyIdent's hand-rolled keyword equality with Lude.Text.replaceIfOneOf
The prior double-Text/replace + digit-marker trick was justified by a claimed
pgn Text/replace bug across concatenation boundaries that never actually
reproduced (verified against plain dhall). Text/equal still can't be used
(dropped in pgn 0.11.0, absent from the Dhall spec), so this swaps in the
shared library's own sentinel-based equality helper instead, already used
elsewhere in this codebase. Same public interface; golden output unchanged.
* Bump pgn pin to v0.12.0 to unblock gen-contract v5 verification
pgn v0.9.1 hard-rejects gen-contract major version 5 ("Incompatible
contract major version: 5. Expected 4."), which blocked verifying
Task 1's gen-contract v5.0.0 / gen-sdk v3.0.0 bump. pgn v0.12.0 adds
gen-contract v5.0.0 support.
* Update mise.lock for pgn v0.12.0
Generated by mise install after the mise.toml pin bump.
* Discard Text/equal-dependent mapping/namespace validation
pgn 0.11.0 (landed in Task 2) dropped the Text/equal builtin from its
embedded Dhall evaluator, and there is no way to reconstruct a Bool or
other branching decision from comparing two runtime Text values using
only Text/replace. Every piece of this generator that made such a
decision is removed:
- queryNameMappings/customTypeNameMappings (Structures/PythonNameMapping.dhall
and its two call sites in Query.dhall/CustomType.dhall) - the rename-mapping
config feature, whole-cloth discarded per the plan's decision table.
- validateCustomTypeIdentities (Interpreters/Project.dhall) - pushed upstream
instead; pgenie-io/pgenie#75 asks pgn to guarantee unique custom-type
identities at the source.
- Structures/PythonNamespace.dhall's validate and its 4 call sites
(per-query, per-custom-type, project-wide facade/module, and per-type
module-internal namespace audits) - no longer expressible, but
tests/test_generated.py's basedpyright --strict gate (errorCount == 0,
warningCount == 0) already catches the same collisions one layer down,
just later and less precisely attributed.
Also removes PyIdent.dhall's isSnakeIdentifier/isPascalIdentifier helpers
(dead code once the mapping-target validation they served is gone), the
now-stale ADR 0001, and tests/test_python_name_collisions.py. Updates
README/DESIGN/CHANGELOG to describe the new state honestly, and updates
test_unsupported_types.py's synthetic contract-probe helper to the
gen-contract v5.0.0 shape (CustomTypeRef, flattened dimensionality) so it
typechecks again, independent of which task's changes it nominally covers.
Repo-wide `grep -rn "Text/equal" src/` now turns up exactly one remaining
functional use: Interpreters/Project.dhall's buildLookup (line 142) -
that's the next task's scope (replacing it with gen-sdk's Sdk.CustomTypes).
mise run test: 4 failed, 18 passed, 28 errors (up from 39 failed, 7 passed,
28 errors before this task) - every remaining failure traces to that same
buildLookup Text/equal line, confirmed by inspecting each one individually.
mise run golden also still fails at the same line (Exhaustive.dhall has
custom types), so tests/golden is untouched - a null result, not a
confirmation of no diff, pending the next task.
* Replace buildLookup's Text/equal cascade with gen-sdk CustomTypes
Retire the hand-rolled buildLookup custom-kind resolver (the last Text/equal
user in src/) in favor of gen-sdk v3's Sdk.CustomTypes. References now resolve
by gen-contract v5's CustomTypeRef.index via CustomKind.at instead of a
Model.Name -> TypeKind closure keyed on Text/equal.
- CustomKind.Lookup is now a List TypeKind indexed by CustomTypeRef.index,
with an `at` accessor (out-of-range reads as Absent).
- Member/ParamsMember/CustomType resolve refs by index; CustomType.run gains an
explicit self-index parameter for its self-lookup consistency check.
- Project.dhall builds a fixed, never-shrinking kind classification (kindOf)
and, in Skip mode, a one-pass survivor cascade via
supportedCustomTypesReasoned, then masks unsupported indices to Absent so a
reference to a removed type reads exactly as a missing type (Absent on either
failure). Fail mode masks nothing, so its behavior is unchanged.
- The survivor predicate (ownDefinitionSupported) mirrors CustomTypeGen.run's
own-shape failure modes exactly: Domain unsupported, Enum always supported,
Composite supported iff every primitive member is a supported primitive
(oracle: Primitive.run) and every custom member is a bare scalar. This is
stricter than a kind-based array-rank check and is required so no surviving
type later fails to render under Skip (exercised by
test_skip_preserves_nested_registration_chain and
test_skip_unsupported_drops_offending_units_and_cascades).
- Update the contract-probe test helper for the List-shaped lookup and the new
CustomType self-index argument.
- Refresh tests/golden: pgn v0.12.0 now emits Project.customTypes in the
gen-contract v5 topological order (dependencies first), reordering only the
facade/types-init export lists (a_codec_wrapper after z_codec_payload). No
generated type body or statement changes; the generator preserves input order.
* Docs: buildLookup replaced by gen-sdk CustomTypes; no Text/equal left in src
- README/DESIGN: describe the index-based reference resolution and the
single-pass Sdk.CustomTypes survivor cascade; record that no live Text/equal
use remains anywhere in src/ (buildLookup was the last one).
- CHANGELOG (Upcoming): add the v5.0.0/v3.0.0 pin bump + buildLookup retirement
bullet; reconcile the now-superseded "Retained buildLookup" and "bounded fixed
point" wording in the same unreleased section.
- upstream-asks: mark the qualified-reference ask as delivered by gen-contract
v5 and correct the stale buildLookup current-mechanism claim.
* Sweep stale doc status strings after final branch review
The 4-task migration (gen-contract/gen-sdk bump, pgn v0.12.0 pin,
Text/equal-dependent validation removal, buildLookup replacement) left
several stale status strings that no single task owned:
- README.md and DESIGN.md still cited the old pgn 0.9.1 pin and the old
73-passed test count; the suite is now 50 passed, 0 skipped after Task 3
deleted a 692-line test file for a discarded feature.
- CHANGELOG.md's "Upcoming" section had two bullets contradicting each other
on the pgn version and test count.
- docs/upstream-asks.md repeated the stale pgn 0.9.1 pin statement.
- DESIGN.md §9 and docs/upstream-asks.md §2 still described the old
repeated-pass "fixed-point filtering" cascade that Task 4 replaced with
Sdk.CustomTypes' single-pass left-fold; reworded both to match §8's
already-correct description.
- README.md's onUnsupported: Skip description also read as the old
iterative "repeatedly removes... until fixed point" mechanism; reworded
to describe the single pass over topologically-sorted custom types.
Doc-only change; confirmed via mise run test (50 passed) and mise run lint.
* Drop plans
* Fix PR migration checks and collision docs
---------
Co-authored-by: Viacheslav Shvets <slava@shvec.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
This PR completes the generator work started by Nikita Volkov.
The goal is simple: generated code should be close to code a user would write by hand. It should be conventional, readable, and possible to maintain even if the generator is temporarily unavailable.
Most changes below serve that goal. The rest fix concrete bugs, such as duplicate imports, invalid references, and silent Python name collisions.
Inherited work
This branch builds on Nikita's earlier PRs:
#3 - Update to new gen contract and polish established the gen-contract v4 baseline, pgn 0.9.1 pin, surfaced generator warnings, and RSS monitoring around pgn subprocesses.
#5 - Update the deps and conform to the latest architecture spec introduced gen-contract 4.0.1, gen-sdk 2.0.0,
Sdk.Sigs, thegen/tosrc/migration, andsrc/package.dhallas the public entry point.#6 - Modernize and fix the compiler issues updated Lude to 5.2 and simplified the
package.dhall -> ProjectInterpreter.runwiring.#7 - Make sync a config switch introduced configurable sync generation. This PR keeps the idea but changes the resulting package structure.
#8 - Remove rows module moved each Row type next to its SQL and query functions. The final output still has no shared
_rows.pymodule.#9 is the cumulative PR at the tip of the inherited stack. Its
demos/tofixtures/migration and the related build, test, CI, and documentation paths are retained.What changed
One package, async by default
emitSyncis back.Async is always generated as the default API. Enabling
emitSyncadds<package>.syncand adjacent_syncquery functions to the same package.Rows, enums, composites, SQL constants, errors, and registration code are shared. They are not regenerated as a second, incompatible type tree.
This follows the same principle used by Pydantic and similar Python libraries: models have one identity and are not duplicated for sync and async access. A separate package adds complexity without adding useful isolation.
Use psycopg instead of generating codecs
PostgreSQL conversion now uses the adapters built into psycopg 3.3:
StrEnumclasses;register_enumandregister_composite;args_row(Row)constructs the generated Row directly.This removes generated
pg_encodeandpg_decodemethods, mapping-row decoders, encoded SQL, andtyping.castworkarounds.The result is less generated code without losing type safety or supported PostgreSQL behaviour.
Fail before emitting broken code
Custom types are resolved and checked before rendering.
Unsupported shapes, missing types, invalid array ranks, bad mappings, namespace collisions, and reserved Python names now produce explicit generation errors. In
Skipmode, dependent types and queries are removed together instead of leaving broken imports.Explicit mappings are available when a database name needs an intentional Python rename.
Keep the generated output maintainable
Generated Python is Ruff-clean and formatter-stable without post-processing. It uses strict typing, normal dataclasses and enums, SPDX headers, and an explicit regeneration warning.
There is no legacy
_rowscompatibility layer. Generated paths are treated as greenfield code.Reduce Dhall normalization cost
The final performance pass removes unused data from custom-type lookup and stops passing complete configuration and model records through closures that only need a few fields.
This changes no generated Python files.
Verification
The Nikita snapshot is included as historical context. It generated a smaller contract with one selected surface per artifact, so it is not directly comparable with the final output.
The final-contract rows emit the same 25-file package surface. The current row
is the median of five fresh single-artifact golden regenerations on the same
Apple Silicon host. Relative to
55d14db, median wall time fell by 42.3% andpeak RSS by 70.9%, approximately 1.75 GiB.
7b876a3ae2869c55d14db07634be