Merge a warehouse catalog into an authored dictionary - #275
Draft
jat255 wants to merge 2 commits into
Draft
Conversation
jat255
marked this pull request as draft
September 4, 2026 05:49
jat255
added
needs-manual-review
Second of the catalog import stack (kata xp65). Ports catalog.R: selection into labelled relations, the exclusion globs and object cap, the manifest that decides whether a listing is small enough to put in a prompt, catalog search, and the dictionary merge. The rules the R side settled and this keeps: authored prose wins, warehouse types and nullability are authoritative, identifier case normalizes per backend, and an ambiguous relative name is an error rather than a guess. Discovered columns come first in the order the warehouse reported them, and authored columns with no counterpart follow. Every function here is pure over the rows a listing returns, so none of it needs a warehouse. Running the queries that produce those rows belongs to the per-backend readers, which is the next two PRs. Two departures from R, both because Python's types differ. A selection entry is a Selector rather than a TableId, because a selection may stop at a catalog or a schema and a TableId always names a table; that also lets id_type() be one dialect-independent function rather than an injected backend one. And the authored table alias is a declared `authored_name` field on Table rather than an attribute set on the fly, since the model is pydantic. The merge rules were confirmed to bite by letting an authored type win over the warehouse and by leaving glob metacharacters unescaped.
…y both names Two defects in the catalog core. An entry naming a table whose relation the warehouse did not return was dropped, including from the validation list, so asking for a table that is not there became a quietly smaller selection instead of an error the backend reports. R keeps an undiscovered stub and validates it; that is restored, and an excluded entry is still dropped from both. The authored_name field added in the previous commit was never read. Relationship prose says what the author wrote, and the merge re-keys a table to its warehouse label, so a relationship on `orders` stopped reaching the first-touch entry of ANALYTICS.PUBLIC.ORDERS. Rendering now matches either name. Found by roborev job 331.
jat255
force-pushed
the
jat255/xp65-catalog-merge
branch
from
September 5, 2026 04:58
0fa5884 to
730db47
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of the catalog import stack (kata xp65). Stacked on #274. Ports
catalog.R: selection into labelled relations, exclusion globs and the object cap, the manifest that decides whether a listing belongs in the prompt or behind search, catalog search, and the dictionary merge.The rules the R side settled and this keeps: authored prose wins, warehouse types and nullability are authoritative, identifier case normalizes per backend, and an ambiguous relative name is an error rather than a guess. Discovered columns come first in the order the warehouse reported them; authored columns with no counterpart follow.
Everything here is a pure function over the rows a listing returns, so none of it needs a warehouse. Running the queries that produce those rows is the next two PRs, which is the split that makes this testable at all.
Departures from R, both because the Python types differ
A selection entry is a
Selectorrather than aTableId, because a selection may stop at a catalog or a schema and aTableIdalways names a table. That also letsid_type()be one dialect-independent function instead of an injected per-backend one.The authored table alias is a declared
authored_namefield onTablerather than an attribute set on the fly, since the model is pydantic.Two defects found in review, both worth knowing
An entry naming a table the warehouse did not return was dropped, including from the validation list, so asking for a table that is not there became a quietly smaller selection rather than an error. R keeps an undiscovered stub and validates it.
authored_namewas added and then never read. Relationship prose says what the author wrote, and the merge re-keys the table to its warehouse label, so a relationship onordersstopped reaching the entry forANALYTICS.PUBLIC.ORDERS. Rendering now matches either name. Worth a reviewer's eye, because it is the seam where the authored and warehouse namespaces meet and there are likely other places that only know one of the two names.Verification: 630 tests, ruff and pyrefly clean. The merge rules were confirmed to bite by letting an authored type win over the warehouse and by leaving glob metacharacters unescaped.