mdcode: add semantic-model scope and BigQuery Graph push

[libei]

Third change in the mdcode Semantic Model series, on top of the now-merged
IR (#258) and BigQuery Graph generator (#269). Targets main.

What this adds

A semantic-model scope and a push-only SemanticModel layout, so the
existing init / push verbs manage a locally-authored Apache Ossie model
instead of a bespoke command:

kcmd init --semantic-model <project>.<location>.<entryGroup>
kcmd push          # deploys the model's BigQuery Graph
kcmd push --validate-only   # prints the DDL, applies nothing

• The scope is recorded in catalog.yaml (scope: semantic-model.<p>.<l>.<eg>);
  init scaffolds catalog/EntryGroups/<eg> and catalog/EntryLinks/<eg>.
• The model is a single Ossie document per model under
  catalog/EntryGroups/<eg>/<model>.yaml.
• push parses each document to the semantic IR (PR1 loader), lowers it to
  CREATE OR REPLACE PROPERTY GRAPH DDL (PR2 generator), and executes it
  against the project named by the model's GOOGLE custom_extension
deploymentTargets, e.g.
  //bigquery.googleapis.com/projects/<p>/datasets/<d>/propertyGraphs/<g>.
• BigQueryClient.query runs DDL synchronously via jobs.query.

Scope / follow-on

This PR is the BigQuery leg only. Knowledge Catalog resource emit
(entries + entryLinks) for the semantic model is a follow-on; push
currently deploys only the BigQuery Graph and logs that the KC emit is not
yet implemented.

Tests

• deploymentTargets parsing (GOOGLE-vendor filter, BigQuery-URI filter,
  malformed JSON, multiple targets).
• The deploy leg over an inline Ossie document with the BigQuery client
  stubbed: --validate-only (no query), execute against the target
  project, BigQuery-error failure, and no-target failure.
• Scope/layout wiring: the scope round-trips through the manifest, and a
  snapshot over an authored workspace resolves to SemanticModelLayout and
  reads the model document while ignoring sidecars.

tsc --noEmit clean; full mdcode suite green.

Live validation

Validated end-to-end against real BigQuery (sqlgen-testing): authored an
Ossie model with a //bigquery.googleapis.com/.../propertyGraphs/sales_graph
deploymentTarget, ran kcmd init --semantic-model + kcmd push, and
confirmed:

• the target project/dataset/graph are resolved from the URI and the
  CREATE OR REPLACE PROPERTY GRAPH DDL executes via jobs.query;
• a GQL traversal over the deployed edge and the native MEASURE
  (GRAPH_EXPAND) both reproduce a direct-SUM control (per-customer
  revenue correct through the FK edge, no fan-out);
• push is idempotent (second run re-deploys cleanly).

[libei]

Ready for review. @amirhormati requested; @dlychagin-gg PTAL as well (couldn't add you as a formal reviewer — not currently a repo collaborator). Live-validated against real BigQuery; details in the PR description.

[amirhormati]

Structural

1. deploy.ts is the only file in src/libts/ that writes to the console.
   sync.ts, snapshot.ts, source.ts, manifest.ts, semantic/loader.ts and layouts/* all have zero console. calls — the library layer is silent by construction and src/tool/commands.ts owns output. deploy.ts adds eight: lines 157, 187, 191, 195-196, 221, 223-225.

The concrete cost is at deploy.ts:191 — the --validate-only DDL only exists on stdout, so no non-CLI caller can reach it. Widening DeployResult (deploy.ts:42-45) to {success,
details?, ddl: string[], warnings: string[], deployed: number} and printing from commands.push fixes this, and drops the spyOn(console, 'log') boilerplate from all eight deploy tests.

2. Partial deploys are silently discarded on failure.
   deployed is tracked at deploy.ts:150 and incremented at 206, but the early return at deploy.ts:199-204 drops it. These are CREATE OR REPLACE, so a run that replaces graph 1 and then fails on graph 2 has already mutated production, and the operator sees only Failed to
   deploy 'graph2'. Fold the count into details at minimum.

3. The layout globs every EntryGroup, ignoring the scoped one.
   init scaffolds catalog/EntryGroups/ (commands.ts:53), but SemanticModelLayout.init() globs EntryGroups//.yaml (layouts/semantic-model.ts:39-43). Any other group directory in the tree gets deployed too. Compounding it, layouts/semantic-model.ts:52 keys _index on
   bare basename, so EntryGroups/a/sales.yaml and EntryGroups/b/sales.yaml collide silently, last-write-wins. SemanticModelSource already carries entryGroup (sources/semantic-model.ts:27); threading it into the layout constructor fixes both.

4. listEntries() and loadEntry() disagree, and MCP walks straight into it.
   layouts/semantic-model.ts:61-63 returns model handles; layouts/semantic-model.ts:80-83 throws for any of them. commands.push short-circuits at commands.ts:97 so the CLI is safe — but mcp.ts:27 calls listEntries() and mcp.ts:44 calls lookupEntry(name) on each result.
   The MCP server will list the semantic model as an entry and then throw on read. For a push-only layout, returning [] from listEntries() with modelDocuments() as the sole accessor is the honest contract.

5. kcmd pull on a semantic-model scope reports success while doing nothing.
   sources/semantic-model.ts:42-45 is an empty generator, so sync.pull() iterates nothing and returns {success: true}, and commands.ts:83 prints "Successfully updated local snapshot." push got an explicit guard at commands.ts:97; pull (commands.ts:72-90) deserves the
   symmetric one.

Correctness

6. Misleading error when there's no job reference. At deploy.ts:107, the while guard requires jobId. If jobComplete === false and jobId is undefined, the loop never runs, and deploy.ts:116-122 then reports did not complete after 30 polls when zero polls occurred.

7. The poll loop has no backoff. deploy.ts:107-114 depends entirely on the server honoring timeoutMs: 10000 (bigquery.ts:98). If getQueryResults returns promptly for any reason, that's 30 back-to-back requests with no pause.

8. jobs.query is issued without a location. bigquery.ts:88-91 sends only {query, useLegacySql}, and deploy.ts:197 has target.project/target.dataset in hand but passes neither. BigQuery infers location from referenced tables, which should hold for property-graph DDL,
   but the live validation was one US project — worth confirming against a non-US dataset. Note getQueryResults and getJob both already accept location (bigquery.ts:96, 108), so the initial call is the odd one out.

9. Identifier interpolation into DDL. deploy.ts:27 captures [^/]+ for project/dataset/graph, which permits backticks and semicolons, and qualifyGraph at semantic/bigquery.ts:676 does a bare ${parts.join('.')} with no escaping. Low severity — the document is
   authored by whoever already holds the BQ credentials — but it's a parsed-data boundary, and tightening the capture groups to [A-Za-z0-9_-]+ costs nothing and rejects malformed URIs earlier with a better message.

10. A document that parses to zero models is silently skipped. The modelsSeen guard at deploy.ts:211 is global. Given two docs where the second yields no models, the push succeeds and never mentions it. Per-document reporting would be more useful, especially since
    loader warnings only go to stderr (deploy.ts:157).

[libei]

Thanks @amirhormati — all 10 addressed in f14a632. Notes on two where I diverged slightly:

Structural

1. Silent library layer — DeployResult now returns {success, details?, ddl: string[], warnings: string[], deployed: number}; commands.push does all printing. --validate-only DDL is now reachable by non-CLI callers, and the spyOn(console, 'log') boilerplate is gone from the deploy tests.
2. Partial deploys — a mid-run failure now appends (N graph(s) already deployed in this run; CREATE OR REPLACE changes are not rolled back) and deployed is on the result.
3. Layout glob — threaded entryGroup from the source through createLayout into the layout; it now globs EntryGroups/<eg>/*.yaml, which also removes the cross-group basename collision. Added a test with a colliding sales.yaml in a second group that must be ignored.
4. listEntries()/loadEntry() mismatch — listEntries() returns [] for the push-only layout; modelDocuments() is the sole accessor. MCP now lists nothing rather than list-then-throw (no mcp.ts change needed).
5. pull no-op — pull() now short-circuits for the semantic-model scope with an explicit "nothing to pull; KC resource pull not yet implemented" message.

Correctness
6. Fixed — no-jobId now reports query did not complete and returned no job reference to poll.
7. Added POLL_BACKOFF_MS between polls (skipped on the first poll, so the fast path is unchanged).
8. Resolved via getDataset (best-effort; falls back to BQ inference if the dataset can't be read) and threaded to query(), so submit/poll/getJob agree on a region. query() now takes an optional location like the other two. I chose lookup over passing ctx.location because the deploy target's dataset can live in a different project/region than the catalog scope — passing a wrong location would break the currently-working inference path.
9. Tightened the capture groups to [A-Za-z0-9_-]+; added a test that a backtick/; in the graph name fails the match.
10. Worth flagging: the loader enforces semantic_model min 1, so a parsed doc always yields ≥1 model — "zero models" isn't actually reachable; a bad doc throws. Today that throw is uncaught (stack trace to the user). I made two changes for the underlying concern: loader/generator warnings are now surfaced via result.warnings (previously stderr-only), and a malformed doc becomes a clean, document-scoped failure (Model document '<name>': ...) instead of an uncaught exception.

139 tests pass, tsc --noEmit clean.

[amirhormati]

A. The regex comment claims a behavior the code doesn't have. deploy.ts:32-33 says malformed URIs "fail the match and are reported rather than producing broken DDL downstream." They aren't reported — a
non-matching URI is silently skipped at deploy.ts:89-94, and if it was the only target the user gets deploy.ts:230-232: "declares no BigQuery Graph deploymentTarget in a GOOGLE custom_extension."

For a typo'd URI that's actively misleading — the target is declared, it just didn't parse, and the message sends the author looking for a missing extension they already wrote. Now that the regex is strict this
failure mode is more likely, not less. Collecting URIs that start with //bigquery.googleapis.com/ but fail the full match, and naming them in the error, would close the loop the comment already promises.

B. datasetLocation swallows a real failure. deploy.ts:185-186 returns undefined on any non-200, including 404. A typo'd dataset in the URI now silently skips the location pin and fails later inside the DDL with
a murkier error. Since the call is being made anyway, a 404 is a free and precise pre-flight: "dataset demo.slaes not found." Keep the best-effort fallback for 403 (see C), fail fast on 404.

C. New permission dependency, undocumented. deploy.ts:250 means bigquery.datasets.get on the target project is now on the path for every deploy. It degrades gracefully rather than failing, which is the right
call — but the live validation ran as project owner, so this is worth a line in the PR description for anyone deploying under a narrower service account.

D. datasetLocation is called per-target, not per-dataset. deploy.ts:250 sits inside the target loop, so a model with three graphs in one dataset issues three identical datasets.get calls. A Map<string,
string|undefined> keyed on project/dataset is a couple of lines.

E. The backoff makes the exhaustion path untestable. POLL_BACKOFF_MS = 1000 (deploy.ts:108) × MAX_QUERY_POLLS = 30 (deploy.ts:103) means a test for deploy.ts:150-151 would burn 29 real seconds — which is likely
why that branch still has no test while the no-jobId branch next to it does. Threading both constants through DeployOptions would make it instant. Also worth noting a stuck job now takes ~5 min of long-polls
plus ~29 s of sleeps before the CLI gives up.

F. entryExists and listEntries now disagree. layouts/semantic-model.ts:75-77 correctly returns [], but layouts/semantic-model.ts:65-68 still answers from _index — so entryExists('sales') is true for a name
listEntries() won't return. snapshot.createEntry is the caller. Low impact, but it's the same class of inconsistency you just fixed one method over.

Still open from last round

All minor, all your call — but two got slightly easier or slightly worse:

• commands.ts:52 — options.semanticModel.split('.')[2]. This is now a one-word fix, since entryGroup was added to the CatalogSource interface at source.ts:28.
• deploy.ts:275 — the error string still hardcodes catalog/EntryGroups/*/. Slightly worse now: the layout is scoped to a single group (layouts/semantic-model.ts:45-47), so the message names a glob the code
  often doesn't use.
• layout.ts:12 — SEMANTIC_MODEL = 'SemanticModel' casing vs 'standard' / 'documents' on lines 10-11.
• commands.ts:106 — unreachable instanceof check.
• deploy.ts:46 — DeployOptions.force declared, never read.
• commands.ts:54 — EntryLinks scaffolding nothing reads.
• layouts/semantic-model.ts:59 — lastIndexOf('/') vs path.basename.
• snapshot.ts:54 — get layout() public solely for the downcast at commands.ts:105.
• README --semantic-model still undocumented (no README in the changed-file list).

[libei]

Thanks for the thorough second pass, @amirhormati — all of A–F plus the round-2 sweep are in ad2bc6a.

A. Regex comment lied. bigQueryGraphTargets now returns {targets, malformed}: a URI that carries the //bigquery.googleapis.com/ prefix but fails the strict match is collected as malformed. When it's the only target the error now names it ("declares … deploymentTarget(s) that could not be parsed: …") instead of the misleading "no deploymentTarget" message; when it sits alongside valid targets it's surfaced as a warning. Comment rewritten to match. Added a sales_bad_target.yaml fixture and a deploy-level test.

B. datasetLocation swallowed 404. Now fails fast on 404 (dataset <p>.<d> not found) as the precise pre-flight you described, and keeps the best-effort fallback (undefined) for other statuses like 403. Test asserts the DDL job is never submitted on 404, and a separate test asserts 403 → proceed with inferred location.

C. New permission dependency. Documented in the README push section: the semantic-model push reads bigquery.datasets.get to pin the job location and degrades gracefully without it.

D. Per-target datasets.get. Added a Map<project/dataset, location> cache in deployBigQuery; test deploys two graphs in one dataset and asserts a single getDataset call.

E. Untestable exhaustion path. maxQueryPolls / pollBackoffMs are now overridable via DeployOptions (defaulting to the module constants), so the "job never completes" branch is tested instantly. You were exactly right about why it lacked a test.

F. entryExists / listEntries disagreed. entryExists now returns false for the push-only layout, consistent with listEntries().

Round-2 sweep — all done except one: commands.ts:52 (uses manifest.source.entryGroup now), deploy.ts:275 (dropped the hardcoded glob), layout.ts enum value → 'semantic-model', commands.ts:106 (unreachable instanceof removed), deploy.ts:46 (force removed), commands.ts:54 (EntryLinks scaffolding removed), semantic-model.ts:59 (path.basename), README --semantic-model documented.

The one I left: snapshot.ts:54 get layout() — it isn't solely for the downcast; the scope test asserts on snapshot.layout (instanceof + the cast to read modelDocuments()), so a public accessor is still the least-surprising shape here. Happy to swap it for a narrower snapshot.modelDocuments() if you'd prefer the layout kept private — your call.

145 tests pass, tsc --noEmit clean.

[libei]

Proactive hardening from a self-review pass (c1172f4), independent of the review threads above:

• Deterministic default project — an under-qualified dataset source is now qualified with the scope's declared project (threaded into deployBigQuery) rather than the ambient gcloud project, which can silently drift from where the model's tables live. (The deployment target's own project can't be the default: a document is parsed once, before its targets are known, and may declare graphs across projects.)
• Broader malformed-target detection — a host/scheme typo (https://…, or a truncated bigquery.googleapis.co) now trips the malformed-target report instead of being misclassified as "no target declared". The strict URI match is unchanged; only the "looks like a BQ graph target" hint widened (host substring or /propertyGraph(s)/ segment), so a plain Dataplex URI is still left alone.
• Verified query completion — a jobs.query 200 with no response body now fails rather than falling through to a reported success.
• Visible location fallback — a forbidden bigquery.datasets.get pre-flight now records a warning (previously silent) before falling back to BigQuery's location inference.
• Empty-workspace validate — push --validate-only over a workspace with no authored model is a clean no-op; a real push still fails on "nothing to deploy".

+5 tests (150 pass), tsc clean; README documents the source-qualification default.