-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DV2 destinations: Build DestinationState / Migration framework #35303
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
b7b4c3d
to
67f0f17
Compare
Current dependencies on/for this PR:
This stack of pull requests is managed by Graphite. |
67f0f17
to
f9156fd
Compare
37ff176
to
34a4fb3
Compare
34a4fb3
to
fa52885
Compare
@@ -94,12 +149,15 @@ void emptyDestination() throws Exception { | |||
initialStates.forEach(initialState -> when(initialState.isFinalTablePresent()).thenReturn(false)); | |||
|
|||
typerDeduper.prepareSchemasAndRawTables(); | |||
verify(destinationHandler).execute(separately("CREATE SCHEMA overwrite_ns", "CREATE SCHEMA append_ns", "CREATE SCHEMA dedup_ns")); | |||
verify(destinationHandler).execute(separately("CREATE SCHEMA airbyte_internal", "CREATE SCHEMA overwrite_ns", "CREATE SCHEMA append_ns", "CREATE SCHEMA dedup_ns")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
previously the tests didn't call create schema airbyte_internal
because the StreamIds had null rawNamespace. We're now setting those to airbyte_internal
, so these tests need to be updated.
be41e16
to
b579660
Compare
e282bf3
to
71904f4
Compare
eae10b0
to
3c448f8
Compare
71904f4
to
829841d
Compare
9e3d10f
to
2790a2d
Compare
dcb79ff
to
ff34c69
Compare
50500a8
to
1f86d02
Compare
ff34c69
to
d2c07e5
Compare
c0221ad
to
ca6ab2a
Compare
b274ff7
to
75cc1f1
Compare
90b10f3
to
d6199c6
Compare
75cc1f1
to
73f7fe4
Compare
f0a9727
to
c7e2466
Compare
c7e2466
to
d6199c6
Compare
5022d37
to
d8ce2d2
Compare
d8ce2d2
to
d158042
Compare
/publish-java-cdk
|
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com> Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Add a concept of DestinationState, which is tracked in a table in the destination warehouse. Use this concept to implement a generic
Migration
interface. This involves several changes:copy
method, but ended up not needing it)MinimumDestinationState
interface, to enforce that all destination states MUST track a soft-reset parameter.DestinationState extends MinimumDestinationState
. This is propagated across all our classes. It also has a fielddestinationState: DestinationState
.commitDestinationStates
, andgatherInitialState
is now expected to populate thedestinationState
fieldMigration
interface, with two methods. One operates solely on theDestinationState
object; the other should query the DB to confirm the migration is necessary and then execute the migration.List<Migration>
. These are executed withinprepareSchemasAndRawTables
. Also, we callcommitDestinationStates
from bothprepareSchemasAndRawTables
andprepareFinalTables
.DestinationState
/ migration result.TyperDeduperUtil
is also updated to handle these migrations correctly. The old migrators have been moved intoexecuteWeirdMigrations
.See also #35308 for example implementation of a state+migration. (still WIP)