convert destination-snowflake to Kotlin CDK #36910
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
octavia-squidington-iii
added
area/connectors
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @stephane-airbyte and the rest of your teammates on |
stephane-airbyte
force-pushed
the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
from
625e34b
to
d5df838
Compare
stephane-airbyte
force-pushed
the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
9 times, most recently
from
24497d2
to
00c0d0c
Compare
stephane-airbyte
force-pushed
the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
7 times, most recently
from
a44cb0d
to
476ad08
Compare
postamar
reviewed
May 2, 2024
build.gradle
Outdated
| @@ -143,6 +143,10 @@ allprojects { | |||
| // see IntegrationRunner.stopOrphanedThreads | |||
| jvmArgs "--add-opens=java.base/java.lang=ALL-UNNAMED" | |||
|
|
|||
| // This is required for our dangling thread checks at the end of a connector run. | |||
| // see IntegrationRunner.stopOrphanedThreads | |||
| jvmArgs "--add-opens=java.base/java.lang=ALL-UNNAMED" | |||
There was a problem hiding this comment.
this seems redundant considering the changes in the previous PR
There was a problem hiding this comment.
it's needed for tests. We have some tests that use the IntegrationRunner without docker. I moved it to the PR below though
stephane-airbyte
force-pushed
the
stephane/05-01-debuggability_improvements_to_the_cdk
branch
from
997351f
to
43a242e
Compare
stephane-airbyte
force-pushed
the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
from
61cf9eb
to
a3f9263
Compare
stephane-airbyte
force-pushed
the
stephane/05-01-debuggability_improvements_to_the_cdk
branch
from
43a242e
to
f3f7b10
Compare
stephane-airbyte
force-pushed
the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
from
a3f9263
to
000cb1f
Compare
stephane-airbyte
force-pushed
the
stephane/05-01-debuggability_improvements_to_the_cdk
branch
from
f3f7b10
to
06bcd3a
Compare
stephane-airbyte
force-pushed
the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
from
000cb1f
to
5c32fa9
Compare
stephane-airbyte
force-pushed
the
stephane/05-01-debuggability_improvements_to_the_cdk
branch
from
06bcd3a
to
961d513
Compare
stephane-airbyte
force-pushed
the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
2 times, most recently
from
d1ed374
to
97d979d
Compare
stephane-airbyte
force-pushed
the
stephane/05-01-debuggability_improvements_to_the_cdk
branch
from
961d513
to
55bf52e
Compare
stephane-airbyte
force-pushed
the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
2 times, most recently
from
dd9a14a
to
0784c79
Compare
stephane-airbyte
force-pushed
the
stephane/05-01-debuggability_improvements_to_the_cdk
branch
from
55bf52e
to
41beb7a
Compare
stephane-airbyte
force-pushed
the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
2 times, most recently
from
20c12cd
to
b344443
Compare
|
/publish-java-cdk
|
Base automatically changed from
stephane/05-01-debuggability_improvements_to_the_cdk
to
master
May 3, 2024 13:50
stephane-airbyte
force-pushed
the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
from
b344443
to
0a3d4d7
Compare
stephane-airbyte
force-pushed
the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
from
b2790df
to
0a3d4d7
Compare
stephane-airbyte
force-pushed
the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
from
0a3d4d7
to
80bf9c6
Compare
stephane-airbyte
deleted the
stephane/04-08-convert_destination-snowflake_to_kotlin_cdk
branch
aasimsani
pushed a commit
to taxwire/airbyte
that referenced
this pull request
May 6, 2024
not only bringing snowflake to the latest CDK but also: 1) Bringing the `SourceOperation` into production code from the test code. There's really no reason those improvements should stay out of production (and they're present in the source-snowflake) 2) adding `putTimestamp` into the `SourceOperation`, so that snowflake doesn't throw an exception at every call, which implies it also creates a new thread 3) make use of the newly added ability to filter orphan thread on shutdown. We filter all the threads created during calls to `SFStatement.close()` 4) don't always take a lock when deleting destinationStates. We now check if there's any states to delete by doing a `SELECT` (and not taking any table lock) before issuing the `DELETE` (the old behavior was causing test contention, and it's a bad idea in general) 5) only execute `airbyte_internal._airbyte_destination_state`
clnoll
pushed a commit
that referenced
this pull request
May 10, 2024
not only bringing snowflake to the latest CDK but also: 1) Bringing the `SourceOperation` into production code from the test code. There's really no reason those improvements should stay out of production (and they're present in the source-snowflake) 2) adding `putTimestamp` into the `SourceOperation`, so that snowflake doesn't throw an exception at every call, which implies it also creates a new thread 3) make use of the newly added ability to filter orphan thread on shutdown. We filter all the threads created during calls to `SFStatement.close()` 4) don't always take a lock when deleting destinationStates. We now check if there's any states to delete by doing a `SELECT` (and not taking any table lock) before issuing the `DELETE` (the old behavior was causing test contention, and it's a bad idea in general) 5) only execute `airbyte_internal._airbyte_destination_state`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
not only bringing snowflake to the latest CDK but also:
SourceOperationinto production code from the test code. There's really no reason those improvements should stay out of production (and they're present in the source-snowflake)putTimestampinto theSourceOperation, so that snowflake doesn't throw an exception at every call, which implies it also creates a new threadSFStatement.close()SELECT(and not taking any table lock) before issuing theDELETE(the old behavior was causing test contention, and it's a bad idea in general)airbyte_internal._airbyte_destination_state