Skip to content

source-salesforce: performance tuning - #67509

Merged
Maxime Carbonneau-Leclerc (maxi297) merged 8 commits into
masterfrom
dbgold17/source-salesforce-performance-tuning
Oct 28, 2025
Merged

Maxime Carbonneau-Leclerc (maxi297) merged 8 commits into
masterfrom
dbgold17/source-salesforce-performance-tuning

Conversation

@dbgold17

@dbgold17 David Gold (dbgold17) commented Oct 7, 2025 •

Copy link
Copy Markdown
Contributor

What

Per a discussion with Maxime Carbonneau-Leclerc (@maxi297), we believe increasing the maximum number of workers and maximum number of concurrent async jobs will speed up connector performance, especially for larger syncs.

An added benefit of making this change is it should allow us to test out the ability of the new connector performance dashboard to measure the impact.

EDIT (max): I've also added the normalization performance improvements as part of this PR. This should address https://github.com/airbytehq/oncall/issues/9850

Can this PR be safely reverted and rolled back?

  • YES 💚
  • NO ❌

@dbgold17
David Gold (dbgold17) marked this pull request as ready for review October 7, 2025 20:43
@github-actions

github-actions Bot commented Oct 7, 2025

Copy link
Copy Markdown
Contributor

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

  • /format-fix - Fixes most formatting issues.
  • /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
    Example: /update-connector-cdk-version connector=destination-bigquery
  • /bump-version - Bumps connector versions.
    • You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
    • Leaving the changelog arg blank will auto-populate the changelog from the PR title.
  • /run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
  • /build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).
  • /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
  • /poe source example lock - Alias for /poe connector source-example lock.
  • /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
  • /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.

📝 Edit this welcome message.

@dbgold17

David Gold (dbgold17) commented Oct 7, 2025 •

Copy link
Copy Markdown
Contributor Author

/bump-version changelog="Minor tuning to increase sync speed."

Bump Version job started... Check job output.

🔴 Job completed successfully (no changes, this is sus).

@github-actions

github-actions Bot commented Oct 7, 2025 •

Copy link
Copy Markdown
Contributor

source-salesforce Connector Test Results

100 tests   96 ✅  17s ⏱️
  2 suites   4 💤
  2 files     0 ❌

Results for commit d70c8f8.

♻️ This comment has been updated with latest results.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have we run regression test on this? Do we want to pin some versions on connections we know might have a speed boost from this?

@github-actions

github-actions Bot commented Oct 7, 2025 •

Copy link
Copy Markdown
Contributor

Deploy preview for airbyte-docs ready!

✅ Preview
https://airbyte-docs-eh6lad0py-airbyte-growth.vercel.app

Built with commit d70c8f8.
This pull request is being automatically deployed with vercel-action

@maxi297

Copy link
Copy Markdown
Contributor

It seems like regression testing is hitting the same problem Brian Lai (@brianjlai) had. Can we run the regression testing with specific connections? I'll have a look into this issue tomorrow

@maxi297

Copy link
Copy Markdown
Contributor

See https://airbytehq-team.slack.com/archives/C02UJ11D3S6/p1759936631631699 for more details about the regression tests failure. In the meanwhile, can we re-run regression tests with a couple of specific connections?

@dbgold17

Copy link
Copy Markdown
Contributor Author

new approach. I plan to publish a pre-release (dev) version and pin some connections.

Pre-release publish workflow: https://github.com/airbytehq/airbyte/actions/runs/18381278400

Comment thread docs/integrations/sources/salesforce.md
Comment thread docs/integrations/sources/salesforce.md
Comment thread docs/integrations/sources/salesforce.md
Comment thread docs/integrations/sources/salesforce.md
Comment thread docs/integrations/sources/salesforce.md
Comment thread docs/integrations/sources/salesforce.md
Comment thread docs/integrations/sources/salesforce.md
Comment thread docs/integrations/sources/salesforce.md
Comment thread docs/integrations/sources/salesforce.md
Comment thread docs/integrations/sources/salesforce.md

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting for a confirmation from David Gold (@dbgold17) but I would start the rollout next Monday

@dbgold17

Copy link
Copy Markdown
Contributor Author

🚢 it

@maxi297

Maxime Carbonneau-Leclerc (maxi297) commented Oct 26, 2025 •

Copy link
Copy Markdown
Contributor

@maxi297

Copy link
Copy Markdown
Contributor

@maxi297

Copy link
Copy Markdown
Contributor

Regression tests did not run because it can't find connections. In the meanwhile, I've tested pinning versions for a couple of users and everything seemed fine. I'll move forward with the progressive rollout and investigate the regression test issue later today

@maxi297
Maxime Carbonneau-Leclerc (maxi297) merged commit 1e9b6b5 into master Oct 28, 2025
48 of 49 checks passed
@maxi297
Maxime Carbonneau-Leclerc (maxi297) deleted the dbgold17/source-salesforce-performance-tuning branch October 28, 2025 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants