Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Java CDK updates for mysql #36929

Merged
merged 2 commits into from May 3, 2024
Merged

Java CDK updates for mysql #36929

merged 2 commits into from May 3, 2024

Conversation

edgao
Copy link
Contributor

@edgao edgao commented Apr 9, 2024

ported from #34608 to kotlin.

  • TestDatabase
    • general cleanup (nullability, etc)
    • ConfigBuilder.build() now declares ObjectNode as its return type for convenience
  • AbstractJdbcDestination - allow injecting a custom v1v2 migrator
  • JdbcDestinationHandler
    • Make databaseName nullable (see comment for reason)
    • allow overriding the data type for _airbyte_destination_state.updated_at (mysql doesn't have a timestamp_with_timezone type)
    • allow overriding a few DB metadata-related things (mysql doesn't have schemas, so need to pass namespace into the database argument)
    • type name comparison now uses .equals(..., ignoreCase=true)
  • JdbcSqlGenerator
    • add a formatTimestampLiteral method
    • kill the createIndexes method - implementing classes can just return a Sql object with multiple statements
    • use dslContext everywhere instead of static DSL
    • allow overriding renameTable and createV2RawTableFromV1Table (once again, to support mysql quirks)
  • JdbcV1V2Migrator - more open stuff, make databaseName nullable
  • JdbcSqlGeneratorIntegrationTest - nullability, make dslContext protected instead of private
  • JdbcTypingDedupingTest
    • make more things open/protected
    • call convertStreamName on the raw table namespace (this is to match the JdbcSqlGenerator behavior, so that tests can correctly find the raw table)
    • more DSL -> dslContext changes
  • JdbcBufferedConsumerFactory - use ParsedCatalog as source of truth for raw table name
  • JdbcSqlOperations: modify airbyte_meta handling (write explicit value when null)
  • CatalogParser - correctly propagate the original stream name+namespace when there are table name collisions
  • BaseTypingDedupingTest
    • add more tests/assertions
  • misc code cleanup
    • lots of making things nonnullable

Copy link

vercel bot commented Apr 9, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview May 3, 2024 7:38pm

@octavia-squidington-iii octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/destination/mysql labels Apr 9, 2024
@edgao edgao changed the base branch from master to edgao/mysql/cdk_upgrade April 9, 2024 16:36
@edgao edgao force-pushed the edgao/mysql/cdk_upgrade branch 2 times, most recently from 78357f2 to aff3a24 Compare April 11, 2024 21:02
Copy link
Contributor Author

edgao commented Apr 11, 2024

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @edgao and the rest of your teammates on Graphite Graphite

@octavia-squidington-iii octavia-squidington-iii removed the area/connectors Connector related issues label Apr 11, 2024
@edgao edgao force-pushed the dv2/mysql_base_2 branch 2 times, most recently from 2caaff2 to 15fc3ef Compare April 12, 2024 18:36
@edgao edgao mentioned this pull request Apr 12, 2024
Base automatically changed from edgao/mysql/cdk_upgrade to master April 12, 2024 18:41
@edgao edgao force-pushed the dv2/mysql_base_2 branch 3 times, most recently from 08d0fc9 to 01f7542 Compare April 15, 2024 14:46
@edgao edgao force-pushed the dv2/mysql_base_2 branch 7 times, most recently from a8edb04 to 0f59b6c Compare April 19, 2024 15:50
@edgao edgao force-pushed the dv2/mysql_base_2 branch 4 times, most recently from b60791b to 7c96e3d Compare April 23, 2024 22:22
@edgao
Copy link
Contributor Author

edgao commented Apr 23, 2024

made a few minor changes to handle mysql's spec using database instead of schema for the default namespace field, plus merge conflicts (more nullability stuff, and a new @Suppress(deprecation) thing)

@edgao
Copy link
Contributor Author

edgao commented Apr 24, 2024

... and fixed SpecModifyingDestination to correctly proxy the isV2Destination call to the underlying destination object

@edgao
Copy link
Contributor Author

edgao commented May 3, 2024

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/8944047470
✅ Successfully published Java CDK version=0.32.0!

@edgao edgao merged commit 2c0cd6c into master May 3, 2024
28 checks passed
@edgao edgao deleted the dv2/mysql_base_2 branch May 3, 2024 20:03
aasimsani pushed a commit to taxwire/airbyte that referenced this pull request May 6, 2024
clnoll pushed a commit that referenced this pull request May 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants