Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tombstone field to actor_definitions table #9988

Merged
merged 8 commits into from
Feb 4, 2022

Conversation

pmossman
Copy link
Contributor

@pmossman pmossman commented Feb 2, 2022

What

#9912 was opened because connector definition deletions via API were not persisting. I added the ability to delete connector definitions while the database denormalization project was ongoing, and the new 'tombstone' field did not make it into the actor_definitions table.

This PR adds a migration to add the tombstone column, and updates configPersistence to write handle the new column.

How

  • Migration adds 'tombstone' boolean column to actor_definition table
  • Modified YamlSeedConfigPersistence to set the tombstone column when loading
    • Without this, ArchiveHandler tests fail because yaml persistence and db persistence exports are directly compared
  • Added a bunch of .withTombstone(false) to tests in order to get them passing

馃毃 User Impact 馃毃

This should fix connector definition deletions

@github-actions github-actions bot added area/platform issues related to the platform area/server labels Feb 2, 2022
Copy link
Contributor

@subodh1810 subodh1810 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you update the schema of the table in the file
airbyte-db/lib/src/main/resources/configs_database/normalized_tables_schema.txt

Also once you merge this, when you make a release, you will have to upgrade the CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION parameter in all the places to the version introduced.

When this would be released to cloud, again CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION parameter would need to be updated before deploying as part of the upgrade OSS version PR.

This is manual right now

public class V0_35_14_001__AddTombstoneToActorDefinitionTest extends AbstractConfigsDatabaseTest {

@Test
public void test() throws SQLException, IOException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we populate the table before hand with a random record and then run the migration and then assert the value of the new column to make sure the not null constraint is being respected.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, added this!

@@ -77,6 +77,7 @@ Referenced by:
spec | jsonb | | not null |
created_at | timestamp with time zone | | not null | CURRENT_TIMESTAMP
updated_at | timestamp with time zone | | not null | CURRENT_TIMESTAMP
tombstone | boolean | | not null | false
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@subodh1810 added

@pmossman pmossman temporarily deployed to more-secrets February 2, 2022 19:55 Inactive
Copy link
Contributor

@subodh1810 subodh1810 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Remember to update the CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION at all the places and when you deploy this to cloud, there as well

@pmossman pmossman force-pushed the parker/set_tombstone_field_in_persistence branch from eed18bb to 597a7d6 Compare February 4, 2022 19:18
@pmossman pmossman temporarily deployed to more-secrets February 4, 2022 19:19 Inactive
@pmossman pmossman merged commit 76da3cc into master Feb 4, 2022
@pmossman pmossman deleted the parker/set_tombstone_field_in_persistence branch February 4, 2022 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/platform issues related to the platform area/server
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants