Destinations: Refreshes: Track stream statuses in async framework #38075
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
This stack of pull requests is managed by Graphite. Learn more about stacking. |
edgao
commented
May 8, 2024
...n/io/airbyte/cdk/integrations/destination/buffered_stream_consumer/BufferedStreamConsumer.kt
Show resolved
Hide resolved
edgao
force-pushed
the
destinations_refreshes_cdk_changes
branch
from
50c1e4b
to
e857ce6
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
fdf57a0
to
4514243
Compare
edgao
force-pushed
the
destinations_refreshes_cdk_changes
branch
from
e857ce6
to
982948e
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
2 times, most recently
from
7eb8203
to
2a3f5d9
Compare
edgao
force-pushed
the
destinations_refreshes_cdk_changes
branch
from
982948e
to
0fd05a7
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
2a3f5d9
to
7753251
Compare
edgao
force-pushed
the
destinations_refreshes_cdk_changes
branch
from
0fd05a7
to
0d4af89
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
7753251
to
757297c
Compare
edgao
force-pushed
the
destinations_refreshes_cdk_changes
branch
from
0d4af89
to
52b9185
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
757297c
to
21d308b
Compare
edgao
force-pushed
the
destinations_refreshes_cdk_changes
branch
from
52b9185
to
ccf601a
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
21d308b
to
34d6f0b
Compare
edgao
force-pushed
the
destinations_refreshes_cdk_changes
branch
from
ccf601a
to
6839b3f
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
34d6f0b
to
cf93b76
Compare
edgao
commented
May 8, 2024
...src/main/kotlin/io/airbyte/cdk/integrations/destination/async/model/PartialAirbyteMessage.kt
Show resolved
Hide resolved
edgao
force-pushed
the
destinations_refreshes_cdk_changes
branch
from
6839b3f
to
d361d45
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
cf93b76
to
5878c44
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
5878c44
to
de8e132
Compare
edgao
changed the base branch from
destinations_refreshes_cdk_changes
to
edgao/better_default_namespace_tracking
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
2 times, most recently
from
136594e
to
3c20271
Compare
edgao
commented
May 9, 2024
| import io.airbyte.protocol.models.v0.StreamDescriptor | ||
|
|
||
| /** | ||
| * Interface allowing destination to specify clean up logic that must be executed after all | ||
| * record-related logic has finished. | ||
| * | ||
| * The map of StreamSyncSummaries MUST be non-null, but MAY be empty. Streams not present in the map |
There was a problem hiding this comment.
This was only needed for BufferedStreamConsumer, which should no longer be relevant to DV2 destinations...
edgao
force-pushed
the
edgao/catalog_parser_improvements
branch
from
e29c5d9
to
1f11d82
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
221663a
to
80bab4a
Compare
edgao
force-pushed
the
edgao/catalog_parser_improvements
branch
from
1f11d82
to
afef1a6
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
80bab4a
to
96c5070
Compare
gisripa
reviewed
Jun 3, 2024
| @@ -3,17 +3,9 @@ | |||
| */ | |||
| package io.airbyte.cdk.integrations.destination | |||
|
|
|||
| import java.util.* | |||
| import io.airbyte.protocol.models.v0.AirbyteStreamStatusTraceMessage.AirbyteStreamStatus | |||
There was a problem hiding this comment.
for later: we need to clean up this v0 protocol from dependency before we introduce protocol versioning. most places we use classes from v0 and some non-v0
gisripa
reviewed
Jun 5, 2024
| AirbyteMessage.Type.STATE -> { | ||
| stateManager.trackState(message, sizeInBytes.toLong()) | ||
| } | ||
| else -> {} |
gisripa
reviewed
Jun 5, 2024
...ain/kotlin/io/airbyte/cdk/integrations/destination/async/deser/AirbyteMessageDeserializer.kt
Outdated
Show resolved
Hide resolved
...src/main/kotlin/io/airbyte/cdk/integrations/destination/async/model/PartialAirbyteMessage.kt
Show resolved
Hide resolved
edgao
force-pushed
the
edgao/catalog_parser_improvements
branch
from
afef1a6
to
60fe677
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
4 times, most recently
from
9c697a3
to
1be53b3
Compare
gisripa
force-pushed
the
edgao/catalog_parser_improvements
branch
from
60fe677
to
7910b55
Compare
2 tasks
gisripa
approved these changes
Jun 7, 2024
edgao
force-pushed
the
edgao/catalog_parser_improvements
branch
from
7910b55
to
1638b6d
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
1be53b3
to
600a84f
Compare
gisripa
force-pushed
the
edgao/catalog_parser_improvements
branch
from
1638b6d
to
7910b55
Compare
edgao
force-pushed
the
edgao/catalog_parser_improvements
branch
from
7910b55
to
feba86e
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
600a84f
to
30466fd
Compare
edgao
force-pushed
the
edgao/async_tracking_stream_statuses
branch
from
30466fd
to
c469716
Compare
|
/publish-java-cdk
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes https://github.com/airbytehq/airbyte-internal-issues/issues/7608; closes #7621
overwriteFinalTableif the status was not COMPLETEthis PR has "safe" default behavior, i.e. if platform doesn't send a stream status message then we assume success. This is compatible with older platform versions. See #38067 for how we switch to the backwards-incompatible behavior (which is required for no data downtime).