Skip to content

Fix slow /downstreams endpoint for high-fanout nodes#1721

Merged
shangyian merged 2 commits intoDataJunction:mainfrom
shangyian:batch-fetch-fanout-threshold
Jan 27, 2026
Merged

Fix slow /downstreams endpoint for high-fanout nodes#1721
shangyian merged 2 commits intoDataJunction:mainfrom
shangyian:batch-fetch-fanout-threshold

Conversation

@shangyian
Copy link
Copy Markdown
Collaborator

@shangyian shangyian commented Jan 27, 2026

Summary

The /downstreams endpoint was slow when querying nodes with many direct children. This is because for high fanout nodes, we're switching from the recursive CTE approach to BFS. The BFS path was a lot slower than the recursive CTE because it performs N queries per level, whereas the recursive CTE does traversal + eager loading in a single database query.

This PR raises fanout_threshold from 50 to 10000 so that the recursive CTE is used for virtually all cases. The BFS fallback is now only triggered for extremely large graphs.

Test Plan

  • PR has an associated issue: #
  • make check passes
  • make test shows 100% unit test coverage

Deployment Plan

@netlify
Copy link
Copy Markdown

netlify bot commented Jan 27, 2026

Deploy Preview for thriving-cassata-78ae72 canceled.

Name Link
🔨 Latest commit 35e053b
🔍 Latest deploy log https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/697913ec1f377b0008bb5066

@shangyian shangyian changed the title Batch fetch fanout threshold Fix slow /downstreams endpoint for high-fanout nodes Jan 27, 2026
@shangyian shangyian marked this pull request as ready for review January 27, 2026 20:03
@shangyian shangyian merged commit c4506b5 into DataJunction:main Jan 27, 2026
17 checks passed
@shangyian shangyian deleted the batch-fetch-fanout-threshold branch January 27, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant