Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Iceberg support #68

Merged
merged 46 commits into from Apr 23, 2023
Merged

Add Iceberg support #68

merged 46 commits into from Apr 23, 2023

Conversation

nextdude-mdsol
Copy link
Contributor

@nextdude-mdsol nextdude-mdsol commented Jan 26, 2023

  • Add support for iceberg table format (both as a source and sink).
  • Added support for kinesis firehose sink.
  • Swap kinesis sink to AWS v2 SDK version in flink-connector-aws-kinesis-streams
  • Standardize kinesis sink configuration across firehose and streams sink connectors
  • Refactor SourceConfig and SinkConfig to inherit from a common SourceOrSinkConfig trait to enable common configuration processing across source and sink configs.
  • Refactor SinkConfig.get*Sink() methods to return Unit instead of DataStream[E] and rename them to add*Sink(). This simplifies their usage since their return value was never used and not all sinks return DataStreams (like Cassandra and Iceberg) as easily as some other Flink sink apis.
  • Add EmbeddedRowType trait to support converting FlinkEvent instances to Row objects for use with Row type sinks, like Iceberg
  • Created a TableStreamJob job class to support jobs that sink to Row type sinks, like Iceberg.
  • Upgrade to flink 1.16.1

@nextdude nextdude marked this pull request as ready for review April 4, 2023 12:12
@nextdude nextdude self-requested a review as a code owner April 4, 2023 12:12
Copy link
Collaborator

@nextdude nextdude left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nextdude nextdude merged commit 0461083 into main Apr 23, 2023
3 checks passed
@nextdude nextdude deleted the iceberg-sink branch April 23, 2023 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants