DV2 destinations: Build DestinationState / Migration framework #35303
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
This was referenced Feb 14, 2024
edgao
force-pushed
the
edgao/destination_state_table
branch
from
b7b4c3d
to
67f0f17
Compare
|
Current dependencies on/for this PR:
This stack of pull requests is managed by Graphite. |
This was referenced Feb 15, 2024
edgao
force-pushed
the
edgao/destination_state_table
branch
from
67f0f17
to
f9156fd
Compare
edgao
changed the base branch from
master
to
gireesh/02-15-cdk/td-init-state
edgao
force-pushed
the
edgao/destination_state_table
branch
3 times, most recently
from
37ff176
to
34a4fb3
Compare
edgao
changed the base branch from
gireesh/02-15-cdk/td-init-state
to
edgao/typer_deduper_interface_cleanup
edgao
force-pushed
the
edgao/destination_state_table
branch
from
34a4fb3
to
fa52885
Compare
edgao
commented
Feb 21, 2024
| @@ -94,12 +149,15 @@ void emptyDestination() throws Exception { | |||
| initialStates.forEach(initialState -> when(initialState.isFinalTablePresent()).thenReturn(false)); | |||
|
|
|||
| typerDeduper.prepareSchemasAndRawTables(); | |||
| verify(destinationHandler).execute(separately("CREATE SCHEMA overwrite_ns", "CREATE SCHEMA append_ns", "CREATE SCHEMA dedup_ns")); | |||
| verify(destinationHandler).execute(separately("CREATE SCHEMA airbyte_internal", "CREATE SCHEMA overwrite_ns", "CREATE SCHEMA append_ns", "CREATE SCHEMA dedup_ns")); | |||
There was a problem hiding this comment.
previously the tests didn't call create schema airbyte_internal because the StreamIds had null rawNamespace. We're now setting those to airbyte_internal, so these tests need to be updated.
edgao
force-pushed
the
edgao/destination_state_table
branch
2 times, most recently
from
be41e16
to
b579660
Compare
edgao
force-pushed
the
edgao/typer_deduper_interface_cleanup
branch
from
e282bf3
to
71904f4
Compare
edgao
force-pushed
the
edgao/destination_state_table
branch
from
eae10b0
to
3c448f8
Compare
edgao
force-pushed
the
edgao/typer_deduper_interface_cleanup
branch
from
71904f4
to
829841d
Compare
edgao
force-pushed
the
edgao/destination_state_table
branch
2 times, most recently
from
9e3d10f
to
2790a2d
Compare
edgao
force-pushed
the
edgao/typer_deduper_interface_cleanup
branch
2 times, most recently
from
dcb79ff
to
ff34c69
Compare
edgao
force-pushed
the
edgao/destination_state_table
branch
2 times, most recently
from
50500a8
to
1f86d02
Compare
edgao
force-pushed
the
edgao/typer_deduper_interface_cleanup
branch
from
ff34c69
to
d2c07e5
Compare
edgao
force-pushed
the
edgao/destination_state_table
branch
2 times, most recently
from
c0221ad
to
ca6ab2a
Compare
edgao
force-pushed
the
edgao/destination_state_table
branch
4 times, most recently
from
b274ff7
to
75cc1f1
Compare
edgao
force-pushed
the
edgao/typer_deduper_interface_cleanup
branch
from
90b10f3
to
d6199c6
Compare
edgao
force-pushed
the
edgao/destination_state_table
branch
from
75cc1f1
to
73f7fe4
Compare
This was referenced Feb 23, 2024
edgao
changed the title
This was referenced Feb 26, 2024
gisripa
force-pushed
the
edgao/typer_deduper_interface_cleanup
branch
2 times, most recently
from
f0a9727
to
c7e2466
Compare
edgao
force-pushed
the
edgao/typer_deduper_interface_cleanup
branch
from
c7e2466
to
d6199c6
Compare
Base automatically changed from
edgao/typer_deduper_interface_cleanup
to
master
February 28, 2024 22:41
gisripa
force-pushed
the
edgao/destination_state_table
branch
5 times, most recently
from
5022d37
to
d8ce2d2
Compare
gisripa
force-pushed
the
edgao/destination_state_table
branch
from
d8ce2d2
to
d158042
Compare
gisripa
approved these changes
Mar 1, 2024
|
/publish-java-cdk
|
xiaohansong
pushed a commit
that referenced
this pull request
Mar 7, 2024
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com> Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a concept of DestinationState, which is tracked in a table in the destination warehouse. Use this concept to implement a generic
Migrationinterface. This involves several changes:copymethod, but ended up not needing it)MinimumDestinationStateinterface, to enforce that all destination states MUST track a soft-reset parameter.DestinationState extends MinimumDestinationState. This is propagated across all our classes. It also has a fielddestinationState: DestinationState.commitDestinationStates, andgatherInitialStateis now expected to populate thedestinationStatefieldMigrationinterface, with two methods. One operates solely on theDestinationStateobject; the other should query the DB to confirm the migration is necessary and then execute the migration.List<Migration>. These are executed withinprepareSchemasAndRawTables. Also, we callcommitDestinationStatesfrom bothprepareSchemasAndRawTablesandprepareFinalTables.DestinationState/ migration result.TyperDeduperUtilis also updated to handle these migrations correctly. The old migrators have been moved intoexecuteWeirdMigrations.See also #35308 for example implementation of a state+migration. (still WIP)