Async Destination V0: Async Staging Consumer Factory. #26371
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davinchia
requested review from
cgardens and
ryankfu
and removed request for
a team
Affected Connector ReportNOTE
|
| Connector | Version | Changelog | Publish |
|---|
- See "Actionable Items" below for how to resolve warnings and errors.
❌ Destinations (23)
| Connector | Version | Changelog | Publish |
|---|---|---|---|
destination-azure-blob-storage |
0.2.0 |
✅ | ✅ |
destination-clickhouse |
0.2.3 |
✅ | ✅ |
destination-clickhouse-strict-encrypt |
0.2.3 |
🔵 (ignored) |
🔵 (ignored) |
destination-databricks |
1.0.2 |
✅ | ✅ |
destination-dynamodb |
0.1.7 |
✅ | ✅ |
destination-exasol |
0.1.1 |
✅ | ✅ |
destination-gcs |
0.3.0 |
✅ | ✅ |
destination-mariadb-columnstore |
0.1.7 |
✅ | ✅ |
destination-mssql |
0.1.23 |
✅ | ✅ |
destination-mssql-strict-encrypt |
0.1.23 |
🔵 (ignored) |
🔵 (ignored) |
destination-mysql |
0.1.20 |
✅ | ✅ |
destination-mysql-strict-encrypt |
❌ 0.1.21(mismatch: 0.1.20) |
🔵 (ignored) |
🔵 (ignored) |
destination-oracle |
0.1.19 |
✅ | ✅ |
destination-oracle-strict-encrypt |
0.1.19 |
🔵 (ignored) |
🔵 (ignored) |
destination-postgres |
0.3.27 |
✅ | ✅ |
destination-postgres-strict-encrypt |
0.3.27 |
🔵 (ignored) |
🔵 (ignored) |
destination-redshift |
0.4.7 |
✅ | ✅ |
destination-rockset |
0.1.4 |
✅ | ✅ |
destination-snowflake |
1.0.4 |
✅ | ✅ |
destination-starburst-galaxy |
0.0.1 |
✅ | ✅ |
destination-teradata |
0.1.1 |
✅ | ✅ |
destination-tidb |
0.1.1 |
✅ | ✅ |
destination-yugabytedb |
0.1.1 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Other Modules (0)
Actionable Items
(click to expand)
| Category | Status | Actionable Item |
|---|---|---|
| Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
| ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
| Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
| ❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
| Publish | ⚠ not in seed |
The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled. |
cgardens
reviewed
May 22, 2023
| // write this to a file - serilizable buffer? | ||
| // where do we create all the write configs? | ||
| log.info("Starting staging flush.."); | ||
| CsvSerializedBuffer writer = null; |
There was a problem hiding this comment.
is this CSV general or snowflake specific?
There was a problem hiding this comment.
good point. this might be snowflake specific. i'll follow up. however, shouldn't block you from moving ahead now.
leaving comment for myself
There was a problem hiding this comment.
The CsvSerializedBuffer is also used for Redshift, so general
| * this writing, we avoid doing so to simplify the migration to async flushing. | ||
| */ | ||
| @Slf4j | ||
| public class SerialFlush { |
There was a problem hiding this comment.
where would this be used?
There was a problem hiding this comment.
StagingConsumerFactory.create = the default consumer constructor
There was a problem hiding this comment.
felt this at least highlighted the difference between the flush functions vs the current state of getYFunction returning anonymous functions everywhere
cgardens
approved these changes
May 22, 2023
nguyenaiden
pushed a commit
that referenced
this pull request
May 25, 2023
Follow up to #26366 . Pull in the async consumer changes into the Consumer Factory. Also take the chance to split out the StagingConsumerFactory with the goal of clarifying the various general, serial and async functions. Instead of one massive factory file, split into - GeneralStagingFunction.java - AsyncFlush.java - SerialFlush.java representing the general buckets of code we have today. I'm sure we can do smarter things here. This is the bare minimum to unblock us + 'leave things better than we found them'.
marcosmarxm
pushed a commit
to natalia-miinto/airbyte
that referenced
this pull request
Jun 8, 2023
Follow up to airbytehq#26366 . Pull in the async consumer changes into the Consumer Factory. Also take the chance to split out the StagingConsumerFactory with the goal of clarifying the various general, serial and async functions. Instead of one massive factory file, split into - GeneralStagingFunction.java - AsyncFlush.java - SerialFlush.java representing the general buckets of code we have today. I'm sure we can do smarter things here. This is the bare minimum to unblock us + 'leave things better than we found them'.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Follow up to #26366 .
Pull in the async consumer changes into the Consumer Factory. Also take the chance to split out the StagingConsumerFactory with the goal of clarifying the various general, serial and async functions.
How
Instead of one massive factory file, split into
representing the general buckets of code we have today.
I'm sure we can do smarter things here. This is the bare minimum to unblock us + 'leave things better than we found them'.
Recommended reading order
All files.
🚨 User Impact 🚨
Are there any breaking changes? What is the end result perceived by the user?
For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.
If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.
Pre-merge Actions
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
./gradlew :airbyte-integrations:connectors:<name>:integrationTest.0.0.1Dockerfilehas version0.0.1README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog with an entry for the initial version. See changelog exampledocs/integrations/README.mdAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
Updating a connector
Community member or Airbyter
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
Connector Generator
-scaffoldin their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplatesthen checking in your changes