✨ Source Salesforce, Shopify: add maxSecondsBetweenMessages in metadata #36723
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
octavia-squidington-iii
added
area/connectors
| @@ -17,6 +17,7 @@ data: | |||
| githubIssueLabel: source-shopify | |||
| icon: shopify.svg | |||
| license: ELv2 | |||
| maxSecondsBetweenMessages: 1 | |||
There was a problem hiding this comment.
@bazarnov does it make sense in terms of rate limit for shopify (i.e. the rate limit window is one second)? @davinchia mentioned that there would still be a minimum. @davinchia, can you confirm the behavior when the maxSecondsBetweenMessages is lower than 90 minutes?
There was a problem hiding this comment.
@maxi297 It doesn't regard to the BULK Jobs running in avg 150 - 600 sec. I would rather reference the Shopify Documentation, where this info could have been, but there is no mentioned of how long the regular bulk job would take to run. My opinion here is we should rely on the value of 2 hours, and make it possible to adjust the slice size from the source itself, so the source logic follows the timings as well, when it's possible (if we have long-running jobs - we decrease the next slice to try-run faster, and if we have faster runs for bulk jobs, we can increase the slice size to try get bigger data portion next time, thus dynamically balance the slice size - working on it, btw)
so, @strosek it should be at least: 7200 for source-shopify and 86400 for source-salesforce, since the max bulk job time for salesforce is up to 1 day.
There was a problem hiding this comment.
So, we've been told by @davinchia that this value should be the value of the rate limit hence why I asked him to confirm that. Given the naming of the variable, your reasoning sounds good but this was not the ask given to us. @davinchia, please confirm
There was a problem hiding this comment.
@davinchia confirmed @bazarnov point. Can we update shopify value? I'll approve so that once the change is done, you can merge.
| @@ -17,6 +17,7 @@ data: | |||
| githubIssueLabel: source-shopify | |||
| icon: shopify.svg | |||
| license: ELv2 | |||
| maxSecondsBetweenMessages: 1 | |||
There was a problem hiding this comment.
@davinchia confirmed @bazarnov point. Can we update shopify value? I'll approve so that once the change is done, you can merge.
* master: (1562 commits) ✨ source-google-drive: migrate to poetry (airbytehq#36581) enable spotbugs for CDK core and dependencies submodule (airbytehq#36612) ✨ Source Salesforce, Shopify: add maxSecondsBetweenMessages in metadata (airbytehq#36723) java-cdk: re-export airbyte-api dependency (airbytehq#36759) Source Google Sheets: address dependency conflict and update CDK (airbytehq#36515) Airbyte CI: rename incorrectly named pipelines (airbytehq#36722) Source Azure Blob Storage: add integration tests (airbytehq#36542) Salesforce: retry on download_data and create_stream_job (airbytehq#36385) ✨Source Monday: Bumped CDK version dependency (airbytehq#36746) airbyte-ci: burst gradle task cache on new java cdk release (airbytehq#36480) chore(connectors): remove tasks.py and top-level requirements.txt (airbytehq#36738) airbyte-ci: fix pull-request-number option for migrate_to_base_image (airbytehq#36779) 🤖 Bump patch version of Python CDK add backward compatibility for an old close slice logic (airbytehq#36774) Bump Airbyte version from 0.57.0 to 0.57.1 🤖 Bump patch version of Python CDK low-code: Fix cursor pagination instantiation if the stop_condition is a string (airbytehq#36760) fix rabbitmq icon and simplify docs registry code (airbytehq#36767) Update azure-entra-id.md (airbytehq#36758) re-enable rabbitmq on OSS (airbytehq#36749) ...
…ta (airbytehq#36723) Add maxSecondsBetweenMessages for Salesforce and Shopify with bulk job time into consideration.
…ta (airbytehq#36723) Add maxSecondsBetweenMessages for Salesforce and Shopify with bulk job time into consideration.
Certified connectors are required to have
maxSecondsBetweenMessagesset in metadata file.maxSecondsBetweenMessagesis the longest time frame (in seconds) of API requests limits reset for endpoints used by connector. IssueWhat
maxSecondsBetweenMessagesvalue for the following connectors was added:source-salesforce reference
source-shopify reference
How
Files
metadata.yamlfor connectors mentioned above were updated by addingmaxSecondsBetweenMessagesvalue.🚨 User Impact 🚨
No breaking changes