perf: only alias schema members referenced by the query by shriram-devrev · Pull Request #252 · devrev/meerkat

shriram-devrev · 2026-05-08T04:11:15Z

Summary

ensureTableSchemasAlias was taking ~16.5s for tenants with ~8k fields even when the active query referenced only a handful of them:

"ensureTableSchemasAlias": "16569.6ms"

Every measure and dimension across every table schema was collected into one flat batch and pushed through ensureColumnAliasBatch, which does two DuckDB roundtrips (parseExpressions + serializeExpressions) that scale linearly with the batch size. Most of that work was wasted on members the query never touches.

What changed

ensureTableSchemaAliasSql (meerkat-core) now takes the Query as a required arg and filters descriptors before batching. It walks:

query.measures
query.dimensions
query.order keys
query.filters (recursive and / or tree)

joinPaths is intentionally excluded: left/right are table names, not dotted member refs. The actual join SQL lives in TableSchema.joins[].sql which is already fully qualified and bypassed by the aliaser.

Members not referenced by the query are passed through untouched on the cloned schema — no parse/serialize roundtrip for them.

meerkat-browser and meerkat-node thread the new `query` arg through their thin wrappers.

Expected impact

For a widget referencing ~20 members out of 8k, DuckDB sees 20 SQL strings per batch instead of 8000 — roughly a 400x reduction in parse/serialize work per call.

BREAKING CHANGE

API surface change on `@devrev/meerkat-core`, `@devrev/meerkat-browser`, `@devrev/meerkat-node`:

`EnsureTableSchemaAliasSqlParams` now requires `query: Query`.
`EnsureTableSchemasAliasParams` now requires `query: Query`.
The curried `ensureTableSchemaAlias(connection)` factory returns `(tableSchemas, query) => ...`.

All three packages bumped to `0.0.128`. A companion devrev-web PR updates the 4 call sites.

Notes for reviewers

This PR currently includes the commit from #247 (`fix: qualify struct field access to avoid binder ambiguity post-join`) as a base, because it touches the same `ensureTableSchemaAliasSql` surface. Once #247 merges, this branch will rebase cleanly to only the perf commit. Please focus review on the top commit (`perf: only alias schema members referenced by the query`).

Test plan

`ensureTableSchemaAliasSql.spec.ts` — 16/16 tests pass (11 existing + 5 new query-filtering cases):
- skips members not referenced by the query
- includes members referenced only by filters (including nested `and`/`or`)
- includes members referenced only by `order`
- ignores `joinPaths` entries because they reference table names
- does not call the aliaser when no schema member is referenced
`meerkat-browser` and `meerkat-node` wrapper specs updated to pass `query` and assert it flows through.
Load testing on a tenant with ~8k fields before rolling to prod (tracked in the devrev-web PR).

When a dimension SQL references a struct field like `stage.stage_id`, the aliaser previously skipped it because `column_names.length !== 1`. After joining tables that have a column with the same name as the struct root (e.g. `devusers.stage` alongside `issue.stage`), DuckDB's binder raises an ambiguous column reference error before the query can run. Make the aliaser schema-aware: introspect each table's physical columns via `DESCRIBE` in the browser and node wrappers, then rewrite a multi-part column ref as `<tableName>.<root>.<rest...>` only when the leading identifier is a known column on the current table. Cross-table references (`customers.id`), already-qualified refs, lambda-bound identifiers, and unknown multi-part refs are left untouched. Falls back to the legacy length-1 behavior when schema information is not supplied, preserving backwards compatibility.

ensureTableSchemasAlias was collecting every measure and dimension across every table schema into a single batch and sending them through ensureColumnAliasBatch — two DuckDB roundtrips (parseExpressions and serializeExpressions) that scale linearly with the number of members in the batch. For tenants with ~8k fields, a single call took ~16.5s even when the active query referenced only a handful of them. Make ensureTableSchemaAliasSql take the Query as a required arg and only alias members the query actually touches — walking measures, dimensions, order, and filters (recursive and/or tree). joinPaths is intentionally excluded: its left/right values are table names, not member references, and the actual join SQL lives in TableSchema.joins[].sql which is already fully qualified. Members not referenced by the query are passed through untouched on the cloned schema, so DuckDB never parses them. For a widget referencing ~20 members out of 8k, that is roughly a 400x reduction in parse/serialize work per call. BREAKING CHANGE: - @devrev/meerkat-core: EnsureTableSchemaAliasSqlParams now requires `query: Query`. - @devrev/meerkat-browser: EnsureTableSchemasAliasParams now requires `query: Query`; the curried ensureTableSchemaAlias(connection) factory returns (tableSchemas, query) => .... - @devrev/meerkat-node: mirrored changes. Bumped meerkat-core, meerkat-browser, meerkat-node to 0.0.128. Unit tests cover query-driven filtering, member preservation for unreferenced members, filter-only refs, order-only refs, and joinPaths exclusion. All pre-existing tests updated to pass a query. Issue: ISS-299745

github-actions · 2026-05-08T04:11:43Z

⚠️ Heads-up: This repository will be blocked from any work other than patching.

File meerkat-browser/src/ensure-table-schema-alias/ensure-table-schema-alias.spec.ts is not allowed to be modified in this patch.
The following vulnerability issues are past SLA:

ISS-279504, overdue 29.0d, sev high, owner: balasaravanan.vp@devrev.ai, checked: 2026-05-08 04:04 UTC
ISS-279506, overdue 21.0d, sev medium, owner: balasaravanan.vp@devrev.ai, checked: 2026-05-08 04:04 UTC

Note that there is significant latency in updating this list. Please reach out on #antifragile if you are in a hurry or have an emergency.

semantic-release (dev dep) pins lodash-es@^4.17.21, which npm resolves to 4.17.23 — flagged by Snyk as SNYK-JS-LODASH-15869625 (high) and SNYK-JS-LODASH-15869619 (medium). Latest lodash-es 4.18.1 carries the patches. Adds a root npm override for lodash-es >=4.18.1 so every nested semantic-release dep de-dupes to the patched version. Follows the same pattern as the earlier qs/picomatch/@tootallnate/once overrides from #244. Issues: ISS-279504, ISS-279506

github-actions · 2026-05-08T08:04:01Z

⚠️ Heads-up: This repository will be blocked from any work other than patching.

File meerkat-browser/src/ensure-table-schema-alias/ensure-table-schema-alias.spec.ts is not allowed to be modified in this patch.
The following vulnerability issues are past SLA:

ISS-279504, overdue 29.0d, sev high, owner: balasaravanan.vp@devrev.ai, checked: 2026-05-08 07:50 UTC
ISS-279506, overdue 21.0d, sev medium, owner: balasaravanan.vp@devrev.ai, checked: 2026-05-08 07:50 UTC

Note that there is significant latency in updating this list. Please reach out on #antifragile if you are in a hurry or have an emergency.

…ration tests The two meerkat-node integration tests for ensureTableSchemaAliasSql invoke the core helper directly (rather than through the node wrapper), so they need the new required 'query' arg that drives referenced-member filtering. Reference every fixture member so the aliaser behaves the same as before. Issue: ISS-299745

zaidjan-devrev · 2026-05-08T10:07:19Z

+const isAliasableIdentifier = (identifier: string | undefined): boolean => {
+  if (!identifier) {
+    return false;
+  }
+  if (/\s/.test(identifier)) {
+    return false;
+  }
+  return ALIASABLE_IDENTIFIER_REGEX.test(identifier);
+};


what are we doing here?

zaidjan-devrev · 2026-05-08T10:08:41Z

+  const candidate = filter as {
+    and?: unknown[];
+    or?: unknown[];
+    member?: string;
+  };


is this type correct?

zaidjan-devrev · 2026-05-08T10:09:06Z

+const collectReferencedMembers = (query: Query): Set<string> => {
+  const referenced = new Set<string>();
+
+  query.measures.forEach((m) => referenced.add(m));
+  query.dimensions?.forEach((d) => referenced.add(d));
+
+  if (query.order) {
+    Object.keys(query.order).forEach((m) => referenced.add(m));
+  }
+
+  query.filters?.forEach((filter) => collectFilterMembers(filter, referenced));
+
+  return referenced;
+};


zaidjan-devrev · 2026-05-08T10:09:39Z

+const collectReferencedMembers = (query: Query): Set<string> => {
+  const referenced = new Set<string>();
+
+  query.measures.forEach((m) => referenced.add(m));
+  query.dimensions?.forEach((d) => referenced.add(d));
+
+  if (query.order) {
+    Object.keys(query.order).forEach((m) => referenced.add(m));
+  }
+
+  query.filters?.forEach((filter) => collectFilterMembers(filter, referenced));
+
+  return referenced;
+};
+


we have similar functionality for tableName, getUsedTables, this util can be parallel to that

zaidjan-devrev · 2026-05-08T10:10:48Z

-  scopedIdentifiers: Set<string>
+  scopedIdentifiers: Set<string>,
+  tableName: string,
+  knownTableNames?: Set<string>


is it optional?

zaidjan-devrev · 2026-05-08T10:11:07Z

  scopedIdentifiers: Set<string>,
-  tableName?: string
+  tableName?: string,
+  knownTableNames?: Set<string>


why do we need to maintain a set of knownTables?

Addresses review feedback on #252: - Move the filter/order/measures/dimensions walker out of ensure-table-schema-alias-sql.ts into src/get-used-members/, mirroring the structure of src/get-used-table-schema/. - Reuse traverseMeerkatQueryFilter for the filter traversal so we share a single walker with getUsedTableSchema and get a properly typed path through MeerkatQueryFilter (drops the prior `unknown`-cast workaround). - Drop duplicate tests from ensure-table-schema-alias-sql.spec.ts that are now owned by get-used-members.spec.ts; keep the one integration test that asserts the aliaser filters descriptors end-to-end. - Add brief doc comments to ensureTableSchemaAliasSql explaining why TableSchema.joins[].sql is preserved as-is, and to the struct-field helpers clarifying what `knownTableNames` does (optional for the legacy fallback path). Issue: ISS-299745

shriram-devrev added 2 commits April 22, 2026 16:41

shriram-devrev requested review from itsTalwar, ujaval403, vpbs2 and zaidjan-devrev as code owners May 8, 2026 04:11

ujaval403 previously approved these changes May 8, 2026

View reviewed changes

shriram-devrev dismissed ujaval403’s stale review via 71c6b0f May 8, 2026 08:17

shriram-devrev enabled auto-merge (squash) May 8, 2026 08:42

ujaval403 approved these changes May 8, 2026

View reviewed changes

shriram-devrev merged commit eb4f167 into main May 8, 2026
4 checks passed

zaidjan-devrev reviewed May 8, 2026

View reviewed changes

shriram-devrev mentioned this pull request May 8, 2026

refactor: extract getUsedMembers to mirror getUsedTableSchema #254

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: only alias schema members referenced by the query#252

perf: only alias schema members referenced by the query#252
shriram-devrev merged 4 commits into
mainfrom
perf/query-driven-ensure-table-alias

shriram-devrev commented May 8, 2026

Uh oh!

github-actions Bot commented May 8, 2026

Uh oh!

github-actions Bot commented May 8, 2026

Uh oh!

Uh oh!

zaidjan-devrev May 8, 2026

Uh oh!

zaidjan-devrev May 8, 2026

Uh oh!

zaidjan-devrev May 8, 2026

Uh oh!

zaidjan-devrev May 8, 2026

Uh oh!

zaidjan-devrev May 8, 2026

Uh oh!

zaidjan-devrev May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

shriram-devrev commented May 8, 2026

Summary

What changed

Expected impact

BREAKING CHANGE

Notes for reviewers

Test plan

Related

Uh oh!

github-actions Bot commented May 8, 2026

⚠️ Heads-up: This repository will be blocked from any work other than patching.

Uh oh!

github-actions Bot commented May 8, 2026

⚠️ Heads-up: This repository will be blocked from any work other than patching.

Uh oh!

Uh oh!

zaidjan-devrev May 8, 2026

Choose a reason for hiding this comment

Uh oh!

zaidjan-devrev May 8, 2026

Choose a reason for hiding this comment

Uh oh!

zaidjan-devrev May 8, 2026

Choose a reason for hiding this comment

Uh oh!

zaidjan-devrev May 8, 2026

Choose a reason for hiding this comment

Uh oh!

zaidjan-devrev May 8, 2026

Choose a reason for hiding this comment

Uh oh!

zaidjan-devrev May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants