Skip to content

Destinations CDK: Add interfaces for operations by responsibility - #38107

Merged
Gireesh Sreepathi (gisripa) merged 1 commit into
masterfrom
cdk-ops-refactor
May 17, 2024
Merged

Destinations CDK: Add interfaces for operations by responsibility#38107
Gireesh Sreepathi (gisripa) merged 1 commit into
masterfrom
cdk-ops-refactor

Conversation

@gisripa

@gisripa Gireesh Sreepathi (gisripa) commented May 9, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

New interfaces to plug into the existing AsyncStreamConsumer.

image

Review guide

Minor cleanup of Existing code.

  • remove TyperDeduperTransaction.kt and unified the methods to TyperDeduperUtil.kt
  • convert TyperDeduperUtil from class to object

New code.

  • SyncOperation, StreamOperation, StorageOperation (see sequence diagram).
  • Default implementations:
    • DefaultSyncOperation - Sync level orchestration logic
    • AbstractStreamOperation - Stream specific orchestration logic
      • StagingStreamOperation - Concrete implementation accepts SerializableBuffer for all staging destinations
      • StandardStreamOperation - Pass through implementation which using Stream<PartialAirbyteMessage> as the records data type
    • StorageOperation - See reference implementation BigQueryStorageOperation in Destination BigQuery: Adapt to newer interface for Sync operations #38132

User Impact

Can this PR be safely reverted and rolled back?

  • YES 💚
  • NO ❌

@vercel

vercel Bot commented May 9, 2024 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview May 17, 2024 8:08pm

@octavia-squidington-iii Octavia Squidington III (octavia-squidington-iii) added the CDK Connector Development Kit label May 9, 2024
@gisripa Graphite App

Gireesh Sreepathi (gisripa) commented May 9, 2024 •
edited by edgao
Loading

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join Gireesh Sreepathi (@gisripa) and the rest of your teammates on Graphite Graphite

@gisripa Gireesh Sreepathi (gisripa) mentioned this pull request May 10, 2024
Merged
1 of 2 tasks
@gisripa
Gireesh Sreepathi (gisripa) force-pushed the cdk-ops-refactor branch 2 times, most recently from 726b1fe to 75b2191 Compare May 10, 2024 18:47
@gisripa
Gireesh Sreepathi (gisripa) changed the base branch from master to edgao/catalog_parser_improvements May 10, 2024 23:06
This was referenced May 10, 2024
Merged
Merged
Closed
Merged
Merged
Merged
@gisripa
Gireesh Sreepathi (gisripa) changed the base branch from edgao/catalog_parser_improvements to cdk_generation_id May 12, 2024 19:17
This was referenced May 12, 2024
Merged
Merged
@gisripa
Gireesh Sreepathi (gisripa) changed the base branch from cdk_generation_id to gireesh/05-10-Bigquery_cdk_signature_changes May 12, 2024 22:52
@gisripa
Gireesh Sreepathi (gisripa) force-pushed the gireesh/05-10-Bigquery_cdk_signature_changes branch from cf090a7 to 9c0dfc5 Compare May 13, 2024 03:13
@gisripa
Gireesh Sreepathi (gisripa) force-pushed the gireesh/05-10-Bigquery_cdk_signature_changes branch from 9c0dfc5 to 4328f0a Compare May 13, 2024 03:27
@gisripa
Gireesh Sreepathi (gisripa) force-pushed the gireesh/05-10-Bigquery_cdk_signature_changes branch from 4328f0a to 3de0d51 Compare May 13, 2024 16:36
@gisripa
Gireesh Sreepathi (gisripa) force-pushed the gireesh/05-10-Bigquery_cdk_signature_changes branch from 3de0d51 to 6bf8684 Compare May 13, 2024 17:43
Base automatically changed from gireesh/05-10-Bigquery_cdk_signature_changes to master May 13, 2024 18:09
@edgao
Edward Gao (edgao) force-pushed the cdk-ops-refactor branch 2 times, most recently from f40bd48 to e7f388f Compare May 15, 2024 22:28
@gisripa
Gireesh Sreepathi (gisripa) force-pushed the cdk-ops-refactor branch 3 times, most recently from 2f3380f to 4d2fbca Compare May 16, 2024 02:52
@gisripa Gireesh Sreepathi (gisripa) mentioned this pull request May 16, 2024
Merged
2 tasks
@edgao
Edward Gao (edgao) force-pushed the cdk-ops-refactor branch 3 times, most recently from 23be58e to 168f866 Compare May 16, 2024 18:22
@gisripa
Gireesh Sreepathi (gisripa) force-pushed the cdk-ops-refactor branch 3 times, most recently from ee99124 to 130b089 Compare May 16, 2024 19:46
@gisripa Gireesh Sreepathi (gisripa) changed the title Connector operations by responsibility Destinations CDK: Add interfaces for operations by responsibility May 16, 2024
@gisripa
Gireesh Sreepathi (gisripa) force-pushed the cdk-ops-refactor branch 2 times, most recently from 692bc34 to fb905b5 Compare May 16, 2024 23:41
@edgao
Edward Gao (edgao) force-pushed the cdk-ops-refactor branch 3 times, most recently from ae5741a to c8281d5 Compare May 17, 2024 18:47
@gisripa
Gireesh Sreepathi (gisripa) marked this pull request as ready for review May 17, 2024 19:15
@gisripa
Gireesh Sreepathi (gisripa) deleted the cdk-ops-refactor branch May 17, 2024 20:40
@edgao Edward Gao (edgao) mentioned this pull request May 17, 2024
Merged
2 tasks
@gisripa Gireesh Sreepathi (gisripa) mentioned this pull request May 31, 2024
Merged
1 of 2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@edgao Edward Gao (edgao) edgao approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CDK Connector Development Kit

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gisripa @edgao @octavia-squidington-iii