Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return bulk export result with history #16

Open
wants to merge 4 commits into
base: feature/support_cloudflare_r2
Choose a base branch
from

Conversation

sagargg
Copy link
Member

@sagargg sagargg commented May 15, 2024

  1. Create a lookup table _query_history_lookup in BigQuery to store query history.
  2. Store encoded queries and table last modified timestamps in the query history lookup table.
  3. Create a history entry if the query does not exist in the lookup table. If it exists, return the result from the history.

If the query history exists, the current response time is ~6 seconds, but returning directly from BigQuery takes an additional 5 seconds due to the extra logic we added.

@sagargg sagargg changed the base branch from master to feature/support_cloudflare_r2 May 22, 2024 10:38
Copy link
Member

@shubham-mahajan shubham-mahajan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sagargg sagargg force-pushed the fix/datastore_bulk_export_history branch 2 times, most recently from 047ab90 to 468bd00 Compare May 22, 2024 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants