Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File CDK: migrate S3 configs #29435

Closed
clnoll opened this issue Aug 15, 2023 · 1 comment
Closed

File CDK: migrate S3 configs #29435

clnoll opened this issue Aug 15, 2023 · 1 comment

Comments

@clnoll
Copy link
Contributor

clnoll commented Aug 15, 2023

Proposed solution

Steps:

  1. Modify S3 config to support both:
    • make dataset, path_pattern and provider not required and airbyte_hidden=True to all V3 fields
    • update order so that it does not clash with V4 This does not seem relevant
    • have SourceFilesAbstract.spec add all source_s3.v4.config.Config.schema required to connectionSpecification and consider those fields when reading the config
  2. Have the read_config method emit an AirbyteControlMessage with the new config
  3. Deploy

Expected result:

  • There are no V3 configurations anymore

User impact:

  • Once deployed, if a user wants to edit an already existing config before a new sync has been executed, he will have to re-enter all the configuration hence we should make sure the time between those steps is as little a possible
  • If the user updates from <3.1.X to V4, the migration result is undetermined. Based on @flash1293 in most cases, it should be fine as the spec hasn't changed much between v1.X and now but we have only tested with clearly V3 configs

Acceptance criteria

  • End state:
    • migration is still in place and will migrate configs. Eventually, we could remove it
    • Spec visible to the user is V4 spec. V3 specific fields are hidden
  • Update the breaking change to mention that some fields might be empty at the beginning (if a sync has not been executed yet)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants