Skip to content

[INLONG-7546][Sort] Fix dirty data not archived for iceberg connector#7547

Merged
dockerzhang merged 1 commit intoapache:masterfrom
lordcheng10:fix_dirtysink_not_open
Mar 8, 2023
Merged

[INLONG-7546][Sort] Fix dirty data not archived for iceberg connector#7547
dockerzhang merged 1 commit intoapache:masterfrom
lordcheng10:fix_dirtysink_not_open

Conversation

@lordcheng10
Copy link
Copy Markdown
Contributor

@lordcheng10 lordcheng10 commented Mar 7, 2023

Prepare a Pull Request

Motivation

The S3DirtySink#open method is not called in IcebergSingleStreamWriter and IcebergMultipleStreamWriter, resulting in dirty data that may never be flushed and archived:

this.scheduledFuture = this.scheduler.scheduleWithFixedDelay(() -> {
if (!closed && !flushing) {
flush();
}
}, s3Options.getBatchIntervalMs(), s3Options.getBatchIntervalMs(), TimeUnit.MILLISECONDS);
}

Modifications

In the open method of IcebergSingleStreamWriter and IcebergMultipleStreamWriter, call the DirtySink#open method

@dockerzhang dockerzhang merged commit c136fa9 into apache:master Mar 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][Sort] Fix dirty data not archived for iceberg connector

4 participants