✨ source-stripe: Use concurrent cdk #31327

girarda · 2023-10-11T23:50:49Z

What

Use the concurrent cdk with a configurable concurrency level when running in full refresh
The concurrency level is capped to 3 workers to avoid hitting the rate limit even in test mode

How

Add num_workers field to the spec
Create concurrent streams using StreamFacade.create_from_stream if the read process has no input state
Concurrency level is set to 2 workers by default

🚨 User Impact 🚨

Full refresh syncs will use the concurrent CDK

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.0.1
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:generateScaffolds then checking in your changes
Documentation which references the generator is updated as needed

Updating the Python CDK

Airbyter

Before merging:

Pull Request description explains what problem it is solving
Code change is unit tested
Build and my-py check pass
Smoke test the change on at least one affected connector
- On Github: Run this workflow, passing --use-local-cdk --name=source-<connector> as options
- Locally: airbyte-ci connectors --use-local-cdk --name=source-<connector> test
PR is reviewed and approved

After merging:

Publish the CDK
- The CDK does not follow proper semantic versioning. Choose minor if this the change has significant user impact or is a breaking change. Choose patch otherwise.
- Write a thoughtful changelog message so we know what was updated.
Merge the platform PR that was auto-created for updating the Connector Builder's CDK version
- This step is optional if the change does not affect the connector builder or declarative connectors.

…urrent_stream.py

…nto alex/abstract_stream

…urrent_stream.py Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>

…e into alex/concurrency_stream

airbyte-integrations/connectors/source-stripe/source_stripe/source.py

…urce.py

airbyte-oss-build-runner · 2023-10-25T20:02:57Z

source-stripe test report (commit `938c271ba7`) - ✅

⏲️ Total pipeline duration: 08mn58s

Step	Result
Build source-stripe docker image for platform(s) linux/amd64	✅
Unit tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-stripe	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-stripe test

airbyte-oss-build-runner · 2023-10-26T04:20:31Z

source-stripe test report (commit `4d4ec83279`) - ✅

⏲️ Total pipeline duration: 08mn40s

Step	Result
Build source-stripe docker image for platform(s) linux/amd64	✅
Unit tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-stripe	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-stripe test

girarda · 2023-10-26T04:23:40Z

good question @maxi297 I'll set a recurring reminder to search in the GCP log

airbyte-oss-build-runner · 2023-10-26T04:27:42Z

source-stripe test report (commit `f317ee3167`) - ❌

⏲️ Total pipeline duration: 04mn07s

Step	Result
Build source-stripe docker image for platform(s) linux/amd64	✅
Unit tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-stripe	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-stripe test

airbyte-oss-build-runner · 2023-10-26T04:36:36Z

source-stripe test report (commit `afbf79a17c`) - ✅

⏲️ Total pipeline duration: 05mn48s

Step	Result
Build source-stripe docker image for platform(s) linux/amd64	✅
Unit tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-stripe	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-stripe test

airbyte-oss-build-runner · 2023-10-26T04:50:30Z

source-qualaroo test report (commit `c85e5738cf`) - ✅

⏲️ Total pipeline duration: 33.44s

Step	Result
Build source-qualaroo docker image for platform(s) linux/amd64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-qualaroo	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-qualaroo test

airbyte-oss-build-runner · 2023-10-26T04:53:19Z

source-stripe test report (commit `c85e5738cf`) - ✅

⏲️ Total pipeline duration: 03mn21s

Step	Result
Build source-stripe docker image for platform(s) linux/amd64	✅
Unit tests	✅
Acceptance tests	✅
Check our base image is used	✅
Code format checks	✅
Validate metadata for source-stripe	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-stripe test

girarda and others added 30 commits September 1, 2023 13:13

AbstractStream

27f2ca3

missing update

f0dc3ff

concurrency

79b4268

delete unused files

312555e

update

b5b72f9

Automated Commit - Formatting Changes

b44b3f7

Update airbyte-cdk/python/airbyte_cdk/sources/streams/concurrent/conc…

8366b00

…urrent_stream.py

override supports_incremental for concurrent_stream

9071c7b

Update comment and check

f298c6d

Merge branch 'master' into alex/abstract_stream

3069988

decouple AbstractStream and Stream

4a16d57

merge

3f54c36

use facades

2f0b05c

reset to master

f01ed1f

reset to master

27dd603

reset to master

0465e14

Update concurrent stream

e55c19b

remove duplicate file

65de389

bridge stream_slices

9a8c4ce

comment

834b90e

move read_full_refresh_logic

bb01ca3

Update abstract_stream.py

ba9d018

minor cleanup

d4bd169

Merge branch 'alex/abstract_stream' of github.com:airbytehq/airbyte i…

b6a6ab8

…nto alex/abstract_stream

Remove generate_partitions from the interface

6856165

name is abstractmethod

3dd11fa

remove unused import

21eeb30

add missing __init__.py

b75b3b9

Update airbyte-cdk/python/airbyte_cdk/sources/streams/concurrent/conc…

8abe4e4

…urrent_stream.py Co-authored-by: Catherine Noll <clnoll@users.noreply.github.com>

Merge branch 'alex/concurrency_stream' of github.com:airbytehq/airbyt…

d8a1791

…e into alex/concurrency_stream

girarda commented Oct 25, 2023

View reviewed changes

airbyte-integrations/connectors/source-stripe/source_stripe/source.py Outdated Show resolved Hide resolved

Update airbyte-integrations/connectors/source-stripe/source_stripe/so…

938c271

…urce.py

girarda added 2 commits October 25, 2023 21:09

Update metadata.yaml

564a9aa

Update stripe.md

4d4ec83

vercel bot deployed to Preview October 26, 2023 04:11 View deployment

girarda added checklist-action-run and removed checklist-action-run labels Oct 26, 2023

Automated Commit - Formatting Changes

f317ee3

vercel bot deployed to Preview October 26, 2023 04:23 View deployment

girarda added checklist-action-run and removed checklist-action-run labels Oct 26, 2023

girarda added 2 commits October 25, 2023 21:28

Update stripe.md

745a9f4

Merge branch 'master' into alex/stripe_concurrent_cdk

afbf79a

girarda added checklist-action-run and removed checklist-action-run labels Oct 26, 2023

vercel bot deployed to Preview October 26, 2023 04:35 View deployment

girarda enabled auto-merge (squash) October 26, 2023 04:37

Automated Commit - Formatting Changes

c85e573

octavia-squidington-iii added the connectors/source/qualaroo label Oct 26, 2023

girarda merged commit f554ebb into master Oct 26, 2023
27 of 28 checks passed

girarda deleted the alex/stripe_concurrent_cdk branch October 26, 2023 05:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ source-stripe: Use concurrent cdk #31327

✨ source-stripe: Use concurrent cdk #31327

girarda commented Oct 11, 2023 •

edited

airbyte-oss-build-runner commented Oct 25, 2023

airbyte-oss-build-runner commented Oct 26, 2023

girarda commented Oct 26, 2023

airbyte-oss-build-runner commented Oct 26, 2023

airbyte-oss-build-runner commented Oct 26, 2023

airbyte-oss-build-runner commented Oct 26, 2023

airbyte-oss-build-runner commented Oct 26, 2023

✨ source-stripe: Use concurrent cdk #31327

✨ source-stripe: Use concurrent cdk #31327

Conversation

girarda commented Oct 11, 2023 • edited

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Airbyter

airbyte-oss-build-runner commented Oct 25, 2023

source-stripe test report (commit 938c271ba7) - ✅

airbyte-oss-build-runner commented Oct 26, 2023

source-stripe test report (commit 4d4ec83279) - ✅

girarda commented Oct 26, 2023

airbyte-oss-build-runner commented Oct 26, 2023

source-stripe test report (commit f317ee3167) - ❌

airbyte-oss-build-runner commented Oct 26, 2023

source-stripe test report (commit afbf79a17c) - ✅

airbyte-oss-build-runner commented Oct 26, 2023

source-qualaroo test report (commit c85e5738cf) - ✅

airbyte-oss-build-runner commented Oct 26, 2023

source-stripe test report (commit c85e5738cf) - ✅

girarda commented Oct 11, 2023 •

edited

source-stripe test report (commit `938c271ba7`) - ✅

source-stripe test report (commit `4d4ec83279`) - ✅

source-stripe test report (commit `f317ee3167`) - ❌

source-stripe test report (commit `afbf79a17c`) - ✅

source-qualaroo test report (commit `c85e5738cf`) - ✅

source-stripe test report (commit `c85e5738cf`) - ✅