add table filtering to mysql - #69228
Conversation
…/schema-filtering
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. Helpful Resources
PR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
|
|
|
/format-fix
|
|
Deploy preview for airbyte-docs ready! ✅ Preview Built with commit a81a739. |
| | 3.6.3 | 2024-07-22 | [42024](https://github.com/airbytehq/airbyte/pull/42024) | Fix a NPE bug on resuming from a failed attempt. | | ||
| | 3.6.2 | 2024-07-17 | [42087](https://github.com/airbytehq/airbyte/pull/42087) | Adding more error translations for MySql source. | | ||
| | 3.6.1 | 2024-07-19 | [42122](https://github.com/airbytehq/airbyte/pull/42122) | Improve wass error message + logging. | | ||
| | 3.6.0 | 2024-07-17 | [40208](https://github.com/airbytehq/airbyte/pull/40208) | Start using the new error MySql source error handler that comes with a new error translation layer. | |
There was a problem hiding this comment.
[markdownlint] reported by reviewdog 🐶
MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1]
| | 0.6.12 | 2022-09-13 | [16657](https://github.com/airbytehq/airbyte/pull/16657) | Improve CDC record queueing performance | | ||
| | 0.6.11 | 2022-09-08 | [16202](https://github.com/airbytehq/airbyte/pull/16202) | Adds error messaging factory to UI | | ||
| | 0.6.10 | 2022-09-08 | [16007](https://github.com/airbytehq/airbyte/pull/16007) | Implement per stream state support. | | ||
| | 0.6.9 | 2022-09-03 | [16216](https://github.com/airbytehq/airbyte/pull/16216) | Standardize spec for CDC replication. See upgrade instructions [above](#upgrading-from-0.6.8-and-older-versions-to-0.6.9-and-later-versions). | |
There was a problem hiding this comment.
Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit
vale
docs/integrations/sources/mysql.md|355 col 145| [Vale.Spelling] Did you really mean 'Debezium'?
docs/integrations/sources/mysql.md|357 col 142| [Vale.Spelling] Did you really mean 'pevious'?
docs/integrations/sources/mysql.md|359 col 163| [Vale.Spelling] Did you really mean 'configs'?
docs/integrations/sources/mysql.md|362 col 188| [Vale.Spelling] Did you really mean 'Debezium'?
docs/integrations/sources/mysql.md|366 col 140| [Vale.Spelling] Did you really mean 'Checkpointable'?
docs/integrations/sources/mysql.md|367 col 138| [Vale.Spelling] Did you really mean 'replication_method'?
docs/integrations/sources/mysql.md|367 col 157| [Vale.Spelling] Did you really mean 'display_type'?
docs/integrations/sources/mysql.md|377 col 147| [Vale.Spelling] Did you really mean 'sessionVariables'?
docs/integrations/sources/mysql.md|380 col 148| [Vale.Spelling] Did you really mean 'Debezium'?
docs/integrations/sources/mysql.md|387 col 138| [Vale.Spelling] Did you really mean 'Debezium'?
docs/integrations/sources/mysql.md|388 col 170| [Vale.Spelling] Did you really mean 'Debezium'?
docs/integrations/sources/mysql.md|389 col 166| [Vale.Spelling] Did you really mean 'checkpointing'?
docs/integrations/sources/mysql.md|391 col 162| [Vale.Spelling] Did you really mean 'datatypes'?
docs/integrations/sources/mysql.md|394 col 153| [Vale.Spelling] Did you really mean 'checkpointing'?
docs/integrations/sources/mysql.md|399 col 142| [Vale.Spelling] Did you really mean 'Debezium'?
docs/integrations/sources/mysql.md|408 col 143| [Vale.Spelling] Did you really mean 'Tinyint'?
docs/integrations/sources/mysql.md|409 col 154| [Vale.Spelling] Did you really mean 'ssl'?
docs/integrations/sources/mysql.md|409 col 166| [Google.WordList] Use 'turn off' or 'off' instead of 'disabled'.
docs/integrations/sources/mysql.md|410 col 147| [Vale.Spelling] Did you really mean 'unencrypted'?
docs/integrations/sources/mysql.md|411 col 186| [Vale.Spelling] Did you really mean 'boolean'?
docs/integrations/sources/mysql.md|415 col 187| [Vale.Spelling] Did you really mean 'configs'?
docs/integrations/sources/mysql.md|421 col 142| [Vale.Spelling] Did you really mean 'debezium'?
docs/integrations/sources/mysql.md|423 col 176| [Vale.Spelling] Did you really mean 'mysql'?
docs/integrations/sources/mysql.md|429 col 198| [Google.WordList] Use 'preceding' instead of 'above'.
docs/integrations/sources/mysql.md|431 col 219| [Vale.Spelling] Did you really mean 'mysql'?
docs/integrations/sources/mysql.md|443 col 142| [Vale.Spelling] Did you really mean 'stacktrace'?
docs/integrations/sources/mysql.md|450 col 172| [Vale.Spelling] Did you really mean 'binlog'?
docs/integrations/sources/mysql.md|453 col 174| [Vale.Spelling] Did you really mean 'param'?
docs/integrations/sources/mysql.md|453 col 180| [Vale.Spelling] Did you really mean 'binlog_row_image'?
docs/integrations/sources/mysql.md|458 col 168| [write-good.Weasel] 'various' is a weasel word!
docs/integrations/sources/mysql.md|465 col 166| [Vale.Spelling] Did you really mean 'boolean'?
docs/integrations/sources/mysql.md|468 col 149| [Google.Units] Put a nonbreaking space between the number and the unit in '5s'.
|
/format-fix
|
Updated description for table filter configuration.
|
/format-fix
|
There was a problem hiding this comment.
Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit
vale
docs/integrations/sources/mysql.md|465 col 166| [Vale.Spelling] Did you really mean 'boolean'?
docs/integrations/sources/mysql.md|468 col 149| [Google.Units] Put a nonbreaking space between the number and the unit in '5s'.
| | 3.1.8 | 2023-11-22 | [32656](https://github.com/airbytehq/airbyte/pull/32656) | Adopt java CDK version 0.5.0. | | ||
| | 3.1.7 | 2023-11-08 | [32125](https://github.com/airbytehq/airbyte/pull/32125) | fix compilation warnings | | ||
| | 3.1.6 | 2023-11-06 | [32193](https://github.com/airbytehq/airbyte/pull/32193) | Adopt java CDK version 0.4.1. | | ||
| | 3.1.5 | 2023-10-31 | [32024](https://github.com/airbytehq/airbyte/pull/32024) | Upgrade to Debezium version 2.4.0. | |
There was a problem hiding this comment.
🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Debezium'?
| | 3.1.6 | 2023-11-06 | [32193](https://github.com/airbytehq/airbyte/pull/32193) | Adopt java CDK version 0.4.1. | | ||
| | 3.1.5 | 2023-10-31 | [32024](https://github.com/airbytehq/airbyte/pull/32024) | Upgrade to Debezium version 2.4.0. | | ||
| | 3.1.4 | 2023-10-30 | [31960](https://github.com/airbytehq/airbyte/pull/31960) | Adopt java CDK version 0.2.0. | | ||
| | 3.1.3 | 2023-10-11 | [31322](https://github.com/airbytehq/airbyte/pull/31322) | Correct pevious release | |
There was a problem hiding this comment.
🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'pevious'?
| | 3.1.4 | 2023-10-30 | [31960](https://github.com/airbytehq/airbyte/pull/31960) | Adopt java CDK version 0.2.0. | | ||
| | 3.1.3 | 2023-10-11 | [31322](https://github.com/airbytehq/airbyte/pull/31322) | Correct pevious release | | ||
| | 3.1.2 | 2023-09-29 | [30806](https://github.com/airbytehq/airbyte/pull/30806) | Cap log line length to 32KB to prevent loss of records | | ||
| | 3.1.1 | 2023-09-26 | [30744](https://github.com/airbytehq/airbyte/pull/30744) | Update MySQL JDBC connection configs to keep default auto-commit behavior | |
There was a problem hiding this comment.
🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'configs'?
| | 3.1.1 | 2023-09-26 | [30744](https://github.com/airbytehq/airbyte/pull/30744) | Update MySQL JDBC connection configs to keep default auto-commit behavior | | ||
| | 3.1.0 | 2023-09-21 | [30270](https://github.com/airbytehq/airbyte/pull/30270) | Enhanced Standard Sync with initial load via Primary Key with a switch to cursor for incremental syncs | | ||
| | 3.0.9 | 2023-09-20 | [30620](https://github.com/airbytehq/airbyte/pull/30620) | Airbyte Certified MySQL Source connector | | ||
| | 3.0.8 | 2023-09-14 | [30333](https://github.com/airbytehq/airbyte/pull/30333) | CDC : Update the correct timezone parameter passed to Debezium to `database.connectionTimezone` | |
There was a problem hiding this comment.
🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Debezium'?
| | 3.0.7 | 2023-09-13 | [30375](https://github.com/airbytehq/airbyte/pull/30375) | Fix a bug causing a failure when DB views are included in sync | | ||
| | 3.0.6 | 2023-09-12 | [30308](https://github.com/airbytehq/airbyte/pull/30308) | CDC : Enable compression of schema history blob in state | | ||
| | 3.0.5 | 2023-09-12 | [30289](https://github.com/airbytehq/airbyte/pull/30289) | CDC : Introduce logic for compression of schema history blob in state | | ||
| | 3.0.4 | 2023-09-06 | [30213](https://github.com/airbytehq/airbyte/pull/30213) | CDC : Checkpointable initial snapshot | |
There was a problem hiding this comment.
🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Checkpointable'?
| | 0.5.16 | 2022-07-14 | [14574](https://github.com/airbytehq/airbyte/pull/14574) | Removed additionalProperties:false from JDBC source connectors | | ||
| | 0.5.15 | 2022-06-23 | [14077](https://github.com/airbytehq/airbyte/pull/14077) | Use the new state management | | ||
| | 0.5.13 | 2022-06-21 | [13945](https://github.com/airbytehq/airbyte/pull/13945) | Aligned datatype test | | ||
| | 0.5.12 | 2022-06-17 | [13864](https://github.com/airbytehq/airbyte/pull/13864) | Updated stacktrace format for any trace message errors | |
There was a problem hiding this comment.
🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'stacktrace'?
| | 0.5.6 | 2022-02-21 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats | | ||
| | 0.5.5 | 2022-02-18 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Updated timestamp transformation with microseconds | | ||
| | 0.5.4 | 2022-02-11 | [10251](https://github.com/airbytehq/airbyte/issues/10251) | bug Source MySQL CDC: sync failed when has Zero-date value in mandatory column | | ||
| | 0.5.2 | 2021-12-14 | [6425](https://github.com/airbytehq/airbyte/issues/6425) | MySQL CDC sync fails because starting binlog position not found in DB | |
There was a problem hiding this comment.
🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'binlog'?
| | 0.5.2 | 2021-12-14 | [6425](https://github.com/airbytehq/airbyte/issues/6425) | MySQL CDC sync fails because starting binlog position not found in DB | | ||
| | 0.5.1 | 2021-12-13 | [8582](https://github.com/airbytehq/airbyte/pull/8582) | Update connector fields title/description | | ||
| | 0.5.0 | 2021-12-11 | [7970](https://github.com/airbytehq/airbyte/pull/7970) | Support all MySQL types | | ||
| | 0.4.13 | 2021-12-03 | [8335](https://github.com/airbytehq/airbyte/pull/8335) | Source-MySql: do not check cdc required param binlog_row_image for standard replication | |
There was a problem hiding this comment.
🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'param'?
| | 0.5.2 | 2021-12-14 | [6425](https://github.com/airbytehq/airbyte/issues/6425) | MySQL CDC sync fails because starting binlog position not found in DB | | ||
| | 0.5.1 | 2021-12-13 | [8582](https://github.com/airbytehq/airbyte/pull/8582) | Update connector fields title/description | | ||
| | 0.5.0 | 2021-12-11 | [7970](https://github.com/airbytehq/airbyte/pull/7970) | Support all MySQL types | | ||
| | 0.4.13 | 2021-12-03 | [8335](https://github.com/airbytehq/airbyte/pull/8335) | Source-MySql: do not check cdc required param binlog_row_image for standard replication | |
There was a problem hiding this comment.
🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'binlog_row_image'?
| | 0.4.11 | 2021-11-19 | [8047](https://github.com/airbytehq/airbyte/pull/8047) | Source MySQL: transform binary data base64 format | | ||
| | 0.4.10 | 2021-11-15 | [7820](https://github.com/airbytehq/airbyte/pull/7820) | Added basic performance test | | ||
| | 0.4.9 | 2021-11-02 | [7559](https://github.com/airbytehq/airbyte/pull/7559) | Correctly process large unsigned short integer values which may fall outside java's `Short` data type capability | | ||
| | 0.4.8 | 2021-09-16 | [6093](https://github.com/airbytehq/airbyte/pull/6093) | Improve reliability of processing various data types like decimals, dates, datetime, binary, and text | |
There was a problem hiding this comment.
[write-good.Weasel] 'various' is a weasel word!
| | 0.4.1 | 2021-07-23 | [4956](https://github.com/airbytehq/airbyte/pull/4956) | Fix log link | | ||
| | 0.3.7 | 2021-06-09 | [3179](https://github.com/airbytehq/airbyte/pull/3973) | Add AIRBYTE_ENTRYPOINT for Kubernetes support | | ||
| | 0.3.6 | 2021-06-09 | [3966](https://github.com/airbytehq/airbyte/pull/3966) | Fix excessive logging for CDC method | | ||
| | 0.3.5 | 2021-06-07 | [3890](https://github.com/airbytehq/airbyte/pull/3890) | Fix CDC handle tinyint\(1\) and boolean types | |
There was a problem hiding this comment.
🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'boolean'?
# Conflicts: # airbyte-integrations/connectors/source-mysql/src/main/kotlin/io/airbyte/integrations/source/mysql/MySqlSourceConfiguration.kt # airbyte-integrations/connectors/source-mysql/src/main/kotlin/io/airbyte/integrations/source/mysql/MySqlSourceConfigurationSpecification.kt # airbyte-integrations/connectors/source-mysql/src/test/resources/expected-spec.json
Wenqi Hu (burakku)
left a comment
There was a problem hiding this comment.
Looks good! Make sure we have tested again after any changes
|
/format-fix
|
|
Note Detected that there are differences in the Gradle dependencies. |
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
What
table filtering in mysql source
How
Review guide
User Impact
Can this PR be safely reverted and rolled back?
Note
Auto-merge may have been disabled. Please check the PR status to confirm.