Skip to content

Migrate concurrency setting - #63349

Merged
Rodi Reich Zilberman (rodireich) merged 11 commits into
masterfrom
13672-speed-source-concurrency-setting
Jul 22, 2025
Merged

Rodi Reich Zilberman (rodireich) merged 11 commits into
masterfrom
13672-speed-source-concurrency-setting

Conversation

@rodireich

@rodireich Rodi Reich Zilberman (rodireich) commented Jul 17, 2025 •

Copy link
Copy Markdown
Contributor

What

In order to allow us to utilize all available sockets by default in speed mode we need to stop setting a default concurrency value of 1.

How

New source configs will not set concurrency value by default, letting the connector decide based on mode and available sockets.
Existing connections will migrate where connections that were previously set to any number other than the default 1 will behave as if not set.
The new optional field will allow an admin to define a max value as before.

effective concurrency is

  • In legacy stdio mode:
  1. If the new max_db_connection is set use it.
  2. Otherwise use the legacy concurrency setting (1 by default).
  • In socket mode:
  1. If the new max_db_connection is set use it.
  2. Otherwise is legacy concurrency is set to value other than the default 1 use it.
  3. Otherwise use the number of available sockets.

Users who didn't set anything manually would run with concurrency = number of sockets, which is the optimal value for performance. For example 8 concurrent select queries running over 8 sockets.

Review guide

@vercel

vercel Bot commented Jul 17, 2025 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Jul 22, 2025 9:53pm

@github-actions

Copy link
Copy Markdown
Contributor

👋 Greetings, Contributor!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

  • /format-fix - Fixes most formatting issues.
  • /bump-version - Bumps connector versions.
    • You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
    • Leaving the changelog arg blank will auto-populate the changelog from the PR title.
  • /run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
  • /build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).

📝 Edit this welcome message.

@github-actions

github-actions Bot commented Jul 17, 2025 •

Copy link
Copy Markdown
Contributor

source-mysql Connector Test Results

  8 files    8 suites   28s ⏱️
 51 tests  51 ✅ 0 💤 0 ❌
175 runs  175 ✅ 0 💤 0 ❌

Results for commit d016963.

♻️ This comment has been updated with latest results.

@rodireich
Rodi Reich Zilberman (rodireich) marked this pull request as ready for review July 18, 2025 00:16
@github-actions

github-actions Bot commented Jul 18, 2025 •

Copy link
Copy Markdown
Contributor

destination-dev-null Connector Test Results

44 tests   12 ✅  39s ⏱️
 4 suites  32 💤
 4 files     0 ❌

Results for commit d016963.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jul 18, 2025 •

Copy link
Copy Markdown
Contributor

destination-customer-io Connector Test Results

45 tests   15 ✅  22s ⏱️
 7 suites  30 💤
 7 files     0 ❌

Results for commit d016963.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jul 18, 2025 •

Copy link
Copy Markdown
Contributor

destination-clickhouse Connector Test Results

 15 files   15 suites   3m 18s ⏱️
211 tests 173 ✅ 38 💤 0 ❌
231 runs  193 ✅ 38 💤 0 ❌

Results for commit d016963.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jul 18, 2025 •

Copy link
Copy Markdown
Contributor

destination-mssql Connector Test Results

129 tests   111 ✅  8m 59s ⏱️
 12 suites   18 💤
 12 files      0 ❌

Results for commit d016963.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jul 18, 2025 •

Copy link
Copy Markdown
Contributor

destination-s3 Connector Test Results

677 tests   355 ✅  2h 10m 31s ⏱️
 24 suites  322 💤
 24 files      0 ❌

Results for commit d016963.

♻️ This comment has been updated with latest results.

@rodireich

Rodi Reich Zilberman (rodireich) commented Jul 18, 2025 •

Copy link
Copy Markdown
Contributor Author

/bump-version changelog="Make concurrency level match number of sockets in speed mode"

Bump Version job started... Check job output.

✅ Changes applied successfully. (f65ff00)

@rodireich Rodi Reich Zilberman (rodireich) changed the title migrate concurrency setting Migrate concurrency setting Jul 22, 2025
when (DataChannelMedium.valueOf(dataChannelMedium)) {
STDIO -> maxDBConnections ?: maxConcurrencyLegacy
SOCKET -> {
maxDBConnections ?: maxConcurrencyLegacy.takeIf { it != 1 } ?: socketPaths.size

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very interesting syntax sugar related to that takeIf

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chatgpt and me collaborated on this line 😉

@rodireich

Rodi Reich Zilberman (rodireich) commented Jul 22, 2025 •

Copy link
Copy Markdown
Contributor Author

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (d016963)

@rodireich
Rodi Reich Zilberman (rodireich) deleted the 13672-speed-source-concurrency-setting branch July 22, 2025 22:06
devin-ai-integration Bot added a commit that referenced this pull request May 6, 2026
…ames

- 3.52.1: 2026-05-05 -> 2026-05-06 (PR #77787 merge date)
- 3.50.5: 2025-07-30 -> 2025-07-31 (PR #63377 UTC merge date)
- 3.50.3: 2025-07-18 -> 2025-07-22 (PR #63349 merge date)
- 3.50.1-rc.1: 2025-07-08 -> 2025-07-10 (PR #62862 merge date)
- Step 3 SSL modes: `require`/`verify-ca` -> `required`/`verify_ca` to match the actual mode values in the spec

Co-Authored-By: bot_apk <apk@cognition.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants