-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛 Source Shopify: fix STATE
messages frequency for the nested
sub-streams
#36170
Merged
bazarnov
merged 6 commits into
master
from
baz/source/shopify/fix-state-frequency-for-nested-substreams
Mar 15, 2024
Merged
🐛 Source Shopify: fix STATE
messages frequency for the nested
sub-streams
#36170
bazarnov
merged 6 commits into
master
from
baz/source/shopify/fix-state-frequency-for-nested-substreams
Mar 15, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
octavia-squidington-iii
added
area/connectors
Connector related issues
connectors/source/shopify
labels
Mar 15, 2024
octavia-squidington-iii
added
the
area/documentation
Improvements or additions to documentation
label
Mar 15, 2024
bazarnov
changed the title
🐛 Source Shopify: Fix STATE messages frequency for the
🐛 Source Shopify: Fix Mar 15, 2024
nested
sub-streamsSTATE
messages frequency for the nested
sub-streams
bazarnov
changed the title
🐛 Source Shopify: Fix
🐛 Source Shopify: fix Mar 15, 2024
STATE
messages frequency for the nested
sub-streamsSTATE
messages frequency for the nested
sub-streams
…fix-state-frequency-for-nested-substreams
artem1205
requested changes
Mar 15, 2024
airbyte-integrations/connectors/source-shopify/source_shopify/streams/base_streams.py
Show resolved
Hide resolved
…fix-state-frequency-for-nested-substreams
artem1205
approved these changes
Mar 15, 2024
bazarnov
deleted the
baz/source/shopify/fix-state-frequency-for-nested-substreams
branch
March 15, 2024 16:08
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area/connectors
Connector related issues
area/documentation
Improvements or additions to documentation
connectors/source/shopify
team/connectors-python
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
The
nested
substreams produce theslices
from theparent stream
. If we have asingle
record perslice
, theCDK
pushes theSTATE
message too frequently, since the STATE is emitted after each slice.How
stream_slices()
to make the batches out of theparent records
, instead of thesingle parent record
processing per slice.state_checkpoint_interval
is set to thepage_size
for every stream now