Skip to content

Speed up GraphQL findNodes via batched extract#2049

Merged
shangyian merged 1 commit intoDataJunction:mainfrom
shangyian:findnodes-measures-load
Apr 24, 2026
Merged

Speed up GraphQL findNodes via batched extract#2049
shangyian merged 1 commit intoDataJunction:mainfrom
shangyian:findnodes-measures-load

Conversation

@shangyian
Copy link
Copy Markdown
Collaborator

@shangyian shangyian commented Apr 23, 2026

Summary

This PR cuts findNodes(nodeTypes: [METRIC]) wall-clock from ~14s to ~1s on a 1000-metric fragment that reads extractedMeasures, metricMetadata, and columns. The resolver now serves the common case from denormalized columns instead of parsing and extracting per-metric on every request.

  • extractedMeasures fast path + batching:
    • When the fragment on extractedMeasures only reads derivedQuery, return NodeRevision.derived_expression (which gets populated by the derive_frozen_measures background task on node create/update) and skip MetricComponentExtractor.extract(). This reduces the number of ANTLR parses and AST walks.
    • It falls through to the full path when components, combiner, or the derivedExpression alias is requested. When the full path is needed, a request-scoped loader batches N per-metric extractions into one session.
  • Drops redundant joinedload(Column.dimension/partition), which are already lazy="joined" on the relationship and were causing cartesian row fanout during hydration.
  • Conditional metricMetadata AST parse. Skip the ANTLR parse + function walk unless the fragment requests expression or incompatibleDruidFunctions.

Also added a backfill scrip that populates derived_expression for metrics created before derive_frozen_measures existed, so they hit the fast path after deploy.

Test Plan

  • PR has an associated issue: #
  • make check passes
  • make test shows 100% unit test coverage

Deployment Plan

@netlify
Copy link
Copy Markdown

netlify Bot commented Apr 23, 2026

Deploy Preview for thriving-cassata-78ae72 canceled.

Name Link
🔨 Latest commit ea79599
🔍 Latest deploy log https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/69eb2e81c28be70008141973

@shangyian shangyian changed the title Findnodes measures load Speed up GraphQL findNodes via batched extract Apr 23, 2026
…st-path resolver for derived expression, conditional metric metadata AST parse
@shangyian shangyian force-pushed the findnodes-measures-load branch from cf40abd to ea79599 Compare April 24, 2026 08:49
@shangyian shangyian marked this pull request as ready for review April 24, 2026 09:10
@shangyian shangyian merged commit 94643cc into DataJunction:main Apr 24, 2026
25 checks passed
@shangyian shangyian deleted the findnodes-measures-load branch April 24, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant