Skip to content

add table filtering to mysql - #69228

Merged
sophiecuiy merged 28 commits into
masterfrom
sophie/source-mysql/schema-filtering
Nov 14, 2025
Merged

sophiecuiy merged 28 commits into
masterfrom
sophie/source-mysql/schema-filtering

Conversation

@sophiecuiy

@sophiecuiy sophiecuiy commented Nov 6, 2025 •

Copy link
Copy Markdown
Contributor

What

table filtering in mysql source

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

  • YES 💚
  • NO ❌

Note

Auto-merge may have been disabled. Please check the PR status to confirm.

@github-actions

github-actions Bot commented Nov 6, 2025

Copy link
Copy Markdown
Contributor

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

  • /format-fix - Fixes most formatting issues.
  • /bump-version - Bumps connector versions.
    • You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
    • Leaving the changelog arg blank will auto-populate the changelog from the PR title.
  • /run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
  • /build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).
  • JVM connectors:
    • /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
      Example: /update-connector-cdk-version connector=destination-bigquery
    • /bump-bulk-cdk-version type=patch changelog='foo' - Bump the Bulk CDK's version. type can be major/minor/patch.
  • Python connectors:
    • /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
    • /poe source example lock - Alias for /poe connector source-example lock.
    • /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
    • /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.

📝 Edit this welcome message.

@github-actions

github-actions Bot commented Nov 6, 2025 •

Copy link
Copy Markdown
Contributor

source-mysql Connector Test Results

 10 files   10 suites   31s ⏱️
 90 tests  90 ✅ 0 💤 0 ❌
214 runs  214 ✅ 0 💤 0 ❌

Results for commit a81a739.

♻️ This comment has been updated with latest results.

@sophiecuiy

sophiecuiy commented Nov 11, 2025 •

Copy link
Copy Markdown
Contributor Author

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (5e98385)

@github-actions

github-actions Bot commented Nov 11, 2025 •

Copy link
Copy Markdown
Contributor

Deploy preview for airbyte-docs ready!

✅ Preview
https://airbyte-docs-drdhhscy5-airbyte-growth.vercel.app

Built with commit a81a739.
This pull request is being automatically deployed with vercel-action

@sophiecuiy
sophiecuiy enabled auto-merge (squash) November 12, 2025 22:14
| 3.6.3 | 2024-07-22 | [42024](https://github.com/airbytehq/airbyte/pull/42024) | Fix a NPE bug on resuming from a failed attempt. |
| 3.6.2 | 2024-07-17 | [42087](https://github.com/airbytehq/airbyte/pull/42087) | Adding more error translations for MySql source. |
| 3.6.1 | 2024-07-19 | [42122](https://github.com/airbytehq/airbyte/pull/42122) | Improve wass error message + logging. |
| 3.6.0 | 2024-07-17 | [40208](https://github.com/airbytehq/airbyte/pull/40208) | Start using the new error MySql source error handler that comes with a new error translation layer. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]

| 0.6.12 | 2022-09-13 | [16657](https://github.com/airbytehq/airbyte/pull/16657) | Improve CDC record queueing performance |
| 0.6.11 | 2022-09-08 | [16202](https://github.com/airbytehq/airbyte/pull/16202) | Adds error messaging factory to UI |
| 0.6.10 | 2022-09-08 | [16007](https://github.com/airbytehq/airbyte/pull/16007) | Implement per stream state support. |
| 0.6.9 | 2022-09-03 | [16216](https://github.com/airbytehq/airbyte/pull/16216) | Standardize spec for CDC replication. See upgrade instructions [above](#upgrading-from-0.6.8-and-older-versions-to-0.6.9-and-later-versions). |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD051/link-fragments Link fragments should be valid [Context: "above"]

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

docs/integrations/sources/mysql.md|355 col 145| [Vale.Spelling] Did you really mean 'Debezium'?
docs/integrations/sources/mysql.md|357 col 142| [Vale.Spelling] Did you really mean 'pevious'?
docs/integrations/sources/mysql.md|359 col 163| [Vale.Spelling] Did you really mean 'configs'?
docs/integrations/sources/mysql.md|362 col 188| [Vale.Spelling] Did you really mean 'Debezium'?
docs/integrations/sources/mysql.md|366 col 140| [Vale.Spelling] Did you really mean 'Checkpointable'?
docs/integrations/sources/mysql.md|367 col 138| [Vale.Spelling] Did you really mean 'replication_method'?
docs/integrations/sources/mysql.md|367 col 157| [Vale.Spelling] Did you really mean 'display_type'?
docs/integrations/sources/mysql.md|377 col 147| [Vale.Spelling] Did you really mean 'sessionVariables'?
docs/integrations/sources/mysql.md|380 col 148| [Vale.Spelling] Did you really mean 'Debezium'?
docs/integrations/sources/mysql.md|387 col 138| [Vale.Spelling] Did you really mean 'Debezium'?
docs/integrations/sources/mysql.md|388 col 170| [Vale.Spelling] Did you really mean 'Debezium'?
docs/integrations/sources/mysql.md|389 col 166| [Vale.Spelling] Did you really mean 'checkpointing'?
docs/integrations/sources/mysql.md|391 col 162| [Vale.Spelling] Did you really mean 'datatypes'?
docs/integrations/sources/mysql.md|394 col 153| [Vale.Spelling] Did you really mean 'checkpointing'?
docs/integrations/sources/mysql.md|399 col 142| [Vale.Spelling] Did you really mean 'Debezium'?
docs/integrations/sources/mysql.md|408 col 143| [Vale.Spelling] Did you really mean 'Tinyint'?
docs/integrations/sources/mysql.md|409 col 154| [Vale.Spelling] Did you really mean 'ssl'?
docs/integrations/sources/mysql.md|409 col 166| [Google.WordList] Use 'turn off' or 'off' instead of 'disabled'.
docs/integrations/sources/mysql.md|410 col 147| [Vale.Spelling] Did you really mean 'unencrypted'?
docs/integrations/sources/mysql.md|411 col 186| [Vale.Spelling] Did you really mean 'boolean'?
docs/integrations/sources/mysql.md|415 col 187| [Vale.Spelling] Did you really mean 'configs'?
docs/integrations/sources/mysql.md|421 col 142| [Vale.Spelling] Did you really mean 'debezium'?
docs/integrations/sources/mysql.md|423 col 176| [Vale.Spelling] Did you really mean 'mysql'?
docs/integrations/sources/mysql.md|429 col 198| [Google.WordList] Use 'preceding' instead of 'above'.
docs/integrations/sources/mysql.md|431 col 219| [Vale.Spelling] Did you really mean 'mysql'?
docs/integrations/sources/mysql.md|443 col 142| [Vale.Spelling] Did you really mean 'stacktrace'?
docs/integrations/sources/mysql.md|450 col 172| [Vale.Spelling] Did you really mean 'binlog'?
docs/integrations/sources/mysql.md|453 col 174| [Vale.Spelling] Did you really mean 'param'?
docs/integrations/sources/mysql.md|453 col 180| [Vale.Spelling] Did you really mean 'binlog_row_image'?
docs/integrations/sources/mysql.md|458 col 168| [write-good.Weasel] 'various' is a weasel word!
docs/integrations/sources/mysql.md|465 col 166| [Vale.Spelling] Did you really mean 'boolean'?
docs/integrations/sources/mysql.md|468 col 149| [Google.Units] Put a nonbreaking space between the number and the unit in '5s'.

Comment thread docs/integrations/sources/mysql.md
Comment thread docs/integrations/sources/mysql.md
Comment thread docs/integrations/sources/mysql.md
Comment thread docs/integrations/sources/mysql.md
Comment thread docs/integrations/sources/mysql.md
Comment thread docs/integrations/sources/mysql.md
Comment thread docs/integrations/sources/mysql.md
Comment thread docs/integrations/sources/mysql.md
Comment thread docs/integrations/sources/mysql.md
Comment thread docs/integrations/sources/mysql.md
Comment thread docs/integrations/sources/mysql.md
@sophiecuiy

sophiecuiy commented Nov 13, 2025 •

Copy link
Copy Markdown
Contributor Author

/format-fix

Format-fix job started... Check job output.

❌ Job failed.

@sophiecuiy

sophiecuiy commented Nov 13, 2025 •

Copy link
Copy Markdown
Contributor Author

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (9b03c8a)

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

vale

docs/integrations/sources/mysql.md|465 col 166| [Vale.Spelling] Did you really mean 'boolean'?
docs/integrations/sources/mysql.md|468 col 149| [Google.Units] Put a nonbreaking space between the number and the unit in '5s'.

| 3.1.8 | 2023-11-22 | [32656](https://github.com/airbytehq/airbyte/pull/32656) | Adopt java CDK version 0.5.0. |
| 3.1.7 | 2023-11-08 | [32125](https://github.com/airbytehq/airbyte/pull/32125) | fix compilation warnings |
| 3.1.6 | 2023-11-06 | [32193](https://github.com/airbytehq/airbyte/pull/32193) | Adopt java CDK version 0.4.1. |
| 3.1.5 | 2023-10-31 | [32024](https://github.com/airbytehq/airbyte/pull/32024) | Upgrade to Debezium version 2.4.0. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Debezium'?

| 3.1.6 | 2023-11-06 | [32193](https://github.com/airbytehq/airbyte/pull/32193) | Adopt java CDK version 0.4.1. |
| 3.1.5 | 2023-10-31 | [32024](https://github.com/airbytehq/airbyte/pull/32024) | Upgrade to Debezium version 2.4.0. |
| 3.1.4 | 2023-10-30 | [31960](https://github.com/airbytehq/airbyte/pull/31960) | Adopt java CDK version 0.2.0. |
| 3.1.3 | 2023-10-11 | [31322](https://github.com/airbytehq/airbyte/pull/31322) | Correct pevious release |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'pevious'?

| 3.1.4 | 2023-10-30 | [31960](https://github.com/airbytehq/airbyte/pull/31960) | Adopt java CDK version 0.2.0. |
| 3.1.3 | 2023-10-11 | [31322](https://github.com/airbytehq/airbyte/pull/31322) | Correct pevious release |
| 3.1.2 | 2023-09-29 | [30806](https://github.com/airbytehq/airbyte/pull/30806) | Cap log line length to 32KB to prevent loss of records |
| 3.1.1 | 2023-09-26 | [30744](https://github.com/airbytehq/airbyte/pull/30744) | Update MySQL JDBC connection configs to keep default auto-commit behavior |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'configs'?

| 3.1.1 | 2023-09-26 | [30744](https://github.com/airbytehq/airbyte/pull/30744) | Update MySQL JDBC connection configs to keep default auto-commit behavior |
| 3.1.0 | 2023-09-21 | [30270](https://github.com/airbytehq/airbyte/pull/30270) | Enhanced Standard Sync with initial load via Primary Key with a switch to cursor for incremental syncs |
| 3.0.9 | 2023-09-20 | [30620](https://github.com/airbytehq/airbyte/pull/30620) | Airbyte Certified MySQL Source connector |
| 3.0.8 | 2023-09-14 | [30333](https://github.com/airbytehq/airbyte/pull/30333) | CDC : Update the correct timezone parameter passed to Debezium to `database.connectionTimezone` |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Debezium'?

| 3.0.7 | 2023-09-13 | [30375](https://github.com/airbytehq/airbyte/pull/30375) | Fix a bug causing a failure when DB views are included in sync |
| 3.0.6 | 2023-09-12 | [30308](https://github.com/airbytehq/airbyte/pull/30308) | CDC : Enable compression of schema history blob in state |
| 3.0.5 | 2023-09-12 | [30289](https://github.com/airbytehq/airbyte/pull/30289) | CDC : Introduce logic for compression of schema history blob in state |
| 3.0.4 | 2023-09-06 | [30213](https://github.com/airbytehq/airbyte/pull/30213) | CDC : Checkpointable initial snapshot |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Checkpointable'?

| 0.5.16 | 2022-07-14 | [14574](https://github.com/airbytehq/airbyte/pull/14574) | Removed additionalProperties:false from JDBC source connectors |
| 0.5.15 | 2022-06-23 | [14077](https://github.com/airbytehq/airbyte/pull/14077) | Use the new state management |
| 0.5.13 | 2022-06-21 | [13945](https://github.com/airbytehq/airbyte/pull/13945) | Aligned datatype test |
| 0.5.12 | 2022-06-17 | [13864](https://github.com/airbytehq/airbyte/pull/13864) | Updated stacktrace format for any trace message errors |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'stacktrace'?

| 0.5.6 | 2022-02-21 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats |
| 0.5.5 | 2022-02-18 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Updated timestamp transformation with microseconds |
| 0.5.4 | 2022-02-11 | [10251](https://github.com/airbytehq/airbyte/issues/10251) | bug Source MySQL CDC: sync failed when has Zero-date value in mandatory column |
| 0.5.2 | 2021-12-14 | [6425](https://github.com/airbytehq/airbyte/issues/6425) | MySQL CDC sync fails because starting binlog position not found in DB |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'binlog'?

| 0.5.2 | 2021-12-14 | [6425](https://github.com/airbytehq/airbyte/issues/6425) | MySQL CDC sync fails because starting binlog position not found in DB |
| 0.5.1 | 2021-12-13 | [8582](https://github.com/airbytehq/airbyte/pull/8582) | Update connector fields title/description |
| 0.5.0 | 2021-12-11 | [7970](https://github.com/airbytehq/airbyte/pull/7970) | Support all MySQL types |
| 0.4.13 | 2021-12-03 | [8335](https://github.com/airbytehq/airbyte/pull/8335) | Source-MySql: do not check cdc required param binlog_row_image for standard replication |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'param'?

| 0.5.2 | 2021-12-14 | [6425](https://github.com/airbytehq/airbyte/issues/6425) | MySQL CDC sync fails because starting binlog position not found in DB |
| 0.5.1 | 2021-12-13 | [8582](https://github.com/airbytehq/airbyte/pull/8582) | Update connector fields title/description |
| 0.5.0 | 2021-12-11 | [7970](https://github.com/airbytehq/airbyte/pull/7970) | Support all MySQL types |
| 0.4.13 | 2021-12-03 | [8335](https://github.com/airbytehq/airbyte/pull/8335) | Source-MySql: do not check cdc required param binlog_row_image for standard replication |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'binlog_row_image'?

| 0.4.11 | 2021-11-19 | [8047](https://github.com/airbytehq/airbyte/pull/8047) | Source MySQL: transform binary data base64 format |
| 0.4.10 | 2021-11-15 | [7820](https://github.com/airbytehq/airbyte/pull/7820) | Added basic performance test |
| 0.4.9 | 2021-11-02 | [7559](https://github.com/airbytehq/airbyte/pull/7559) | Correctly process large unsigned short integer values which may fall outside java's `Short` data type capability |
| 0.4.8 | 2021-09-16 | [6093](https://github.com/airbytehq/airbyte/pull/6093) | Improve reliability of processing various data types like decimals, dates, datetime, binary, and text |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[write-good.Weasel] 'various' is a weasel word!

| 0.4.1 | 2021-07-23 | [4956](https://github.com/airbytehq/airbyte/pull/4956) | Fix log link |
| 0.3.7 | 2021-06-09 | [3179](https://github.com/airbytehq/airbyte/pull/3973) | Add AIRBYTE_ENTRYPOINT for Kubernetes support |
| 0.3.6 | 2021-06-09 | [3966](https://github.com/airbytehq/airbyte/pull/3966) | Fix excessive logging for CDC method |
| 0.3.5 | 2021-06-07 | [3890](https://github.com/airbytehq/airbyte/pull/3890) | Fix CDC handle tinyint\(1\) and boolean types |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'boolean'?

Comment thread docs/integrations/sources/mysql.md
# Conflicts:
#	airbyte-integrations/connectors/source-mysql/src/main/kotlin/io/airbyte/integrations/source/mysql/MySqlSourceConfiguration.kt
#	airbyte-integrations/connectors/source-mysql/src/main/kotlin/io/airbyte/integrations/source/mysql/MySqlSourceConfigurationSpecification.kt
#	airbyte-integrations/connectors/source-mysql/src/test/resources/expected-spec.json

@burakku Wenqi Hu (burakku) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Make sure we have tested again after any changes

@sophiecuiy
sophiecuiy disabled auto-merge November 14, 2025 00:47
@sophiecuiy

sophiecuiy commented Nov 14, 2025 •

Copy link
Copy Markdown
Contributor Author

/format-fix

Format-fix job started... Check job output.

🟦 Job completed successfully (no changes).

@github-actions

Copy link
Copy Markdown
Contributor

Note

Detected that there are differences in the Gradle dependencies.

@sophiecuiy
sophiecuiy merged commit 4846a52 into master Nov 14, 2025
37 checks passed
@sophiecuiy
sophiecuiy deleted the sophie/source-mysql/schema-filtering branch November 14, 2025 18:35
sophiecuiy added a commit that referenced this pull request Nov 14, 2025
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants