Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

馃悶 Fix db config persistence initialization #6220

Merged
merged 2 commits into from
Sep 18, 2021

Conversation

tuliren
Copy link
Contributor

@tuliren tuliren commented Sep 17, 2021

What

  • When the local config directory exists, ConfigSeedProvider will always return the file system persistence as the seed provider and load into the database config persistence, which will roll back unused connector definitions, as reported in Connector versions are rolled back after Airbyte server is upgraded聽#5954.
  • This bug is introduced because:
    • There is no unit test for ConfigSeedProvider.
    • Initially the ConfigPersistence#loadData method is only purposed to initialize the persistence. However, later on it is patched to update the connector definitions as well. The method itself is tested, however, the seed persistence fed into this method is not because of the first bullet point, resulting in this bug.
    • In addition, connector definition is always updated when the image tag is different in updateConnectorDefinitions. This is ok if the seed is always the latest. However, because old configs can be fed into the loadData method, without the version check, the connector will always get rolled back if it is not used. Should there be a version check, we can have a second layer of defense.

How

  • Move both the seed provider logic and the initialization logic to a new initialize method. This method is tested and can guard against stale seed being used to initialize the persistence.
  • A second layer of defense is added in updateConnectorDefinitions that only updates a connector if there is a new version.

Recommended reading order

  1. DatabaseConfigPersistence.java
  2. The rest.

@github-actions github-actions bot added the area/platform issues related to the platform label Sep 17, 2021
@tuliren tuliren linked an issue Sep 17, 2021 that may be closed by this pull request
@tuliren tuliren merged commit 111d90f into master Sep 18, 2021
@tuliren tuliren deleted the liren/fix-db-config-persistence-initialization branch September 18, 2021 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/platform issues related to the platform
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Connector versions are rolled back after Airbyte server is upgraded
3 participants