Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛 Source Shopify: fix STATE messages frequency for the nested sub-streams #36170

Merged
merged 6 commits into from Mar 15, 2024
Merged

🐛 Source Shopify: fix STATE messages frequency for the nested sub-streams #36170

merged 6 commits into from Mar 15, 2024

Conversation

bazarnov
Copy link
Collaborator

What

The nested substreams produce the slices from the parent stream. If we have a single record per slice, the CDK pushes the STATE message too frequently, since the STATE is emitted after each slice.

How

  • added internal buffering to stream_slices() to make the batches out of the parent records, instead of the single parent record processing per slice.
  • the state_checkpoint_interval is set to the page_size for every stream now

@vercel Vercel
Copy link

vercel bot commented Mar 15, 2024 •
edited

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 15, 2024 3:34pm

@octavia-squidington-iii octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Mar 15, 2024
@bazarnov bazarnov changed the title 🐛 Source Shopify: Fix STATE messages frequency for the nested sub-streams 🐛 Source Shopify: Fix STATE messages frequency for the nested sub-streams Mar 15, 2024
@bazarnov bazarnov changed the title 🐛 Source Shopify: Fix STATE messages frequency for the nested sub-streams 🐛 Source Shopify: fix STATE messages frequency for the nested sub-streams Mar 15, 2024
@bazarnov bazarnov self-assigned this Mar 15, 2024
@bazarnov bazarnov marked this pull request as ready for review March 15, 2024 01:00
@octavia-squidington-iv octavia-squidington-iv requested a review from a team March 15, 2024 01:01
@octavia-squidington-iv octavia-squidington-iv requested a review from a team March 15, 2024 11:56
@bazarnov bazarnov requested a review from artem1205 March 15, 2024 15:29
@bazarnov bazarnov merged commit 6efff86 into master Mar 15, 2024
30 checks passed
@bazarnov bazarnov deleted the baz/source/shopify/fix-state-frequency-for-nested-substreams branch March 15, 2024 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@artem1205 artem1205 artem1205 approved these changes

Assignees

@bazarnov bazarnov

Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/shopify team/connectors-python
Projects
GL Roadmap
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@bazarnov @artem1205 @octavia-squidington-iii