Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][OssFile Connector] Make Oss implement source factory and sink factory #6062

Merged

Conversation

EricJoy2048
Copy link
Member

Purpose of this pull request

  1. This pr is make ossfile connector implement source factory and sink factory
  2. Make OssFile Connector support Multi table reading and multi table writing.

Does this PR introduce any user-facing change?

Yes. For specific information, please refer to the changes in the document.

How was this patch tested?

I tested in my local and all test case is success.
image

Because there is no available OSS environment, I disabled the test cases.

Check list

@EricJoy2048 EricJoy2048 force-pushed the 162_oss_support_multiple_table_read branch from d014a2f to b066ebd Compare December 23, 2023 02:08
@@ -29,6 +29,9 @@

<properties>
<hadoop3.version>3.1.4</hadoop3.version>
<hadoop-aliyun.version>3.1.4</hadoop-aliyun.version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whether it needs to appear in example?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whether it needs to appear in example?

Thanks, I will roll back it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on #5545 , we need to remove the binary and automatically generate it

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on #5545 , we need to remove the binary and automatically generate it

I asked other open source projects, but this is not mandatory.

docs/en/connector-v2/sink/OssFile.md Outdated Show resolved Hide resolved
docs/en/connector-v2/source/OssFile.md Outdated Show resolved Hide resolved
public class LocalFileSinkFactory
implements TableSinkFactory<
SeaTunnelRow, FileSinkState, FileCommitInfo, FileAggregatedCommitInfo> {
public class LocalFileSinkFactory extends BaseMultipleTableFinkSinkFactory {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to oss?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to oss?

While adding multi-table support for oss, I found that BaseMultipleTableFinkSinkFactory could be added to make the code cleaner, so I needed to modify the LocalFileSinkFactory implementation

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to oss?

While adding multi-table support for oss, I found that BaseMultipleTableFinkSinkFactory could be added to make the code cleaner, so I needed to modify the LocalFileSinkFactory implementation

seatunnel-dist/pom.xml Show resolved Hide resolved
EricJoy2048 and others added 4 commits December 25, 2023 10:48
Co-authored-by: hailin0 <wanghailin@apache.org>
Co-authored-by: hailin0 <wanghailin@apache.org>
…e/src/test/resources/excel/fake_to_oss_excel.conf

Co-authored-by: hailin0 <wanghailin@apache.org>
@EricJoy2048 EricJoy2048 force-pushed the 162_oss_support_multiple_table_read branch from 7c5d7de to e823425 Compare December 25, 2023 02:59
Copy link
Member

@Carl-Zhou-CN Carl-Zhou-CN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@TaoZex TaoZex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Member

@ruanwenjun ruanwenjun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@EricJoy2048 EricJoy2048 merged commit 1a8e9b4 into apache:dev Dec 27, 2023
7 checks passed
alextinng pushed a commit to alextinng/seatunnel that referenced this pull request Jan 4, 2024
alextinng pushed a commit to alextinng/seatunnel that referenced this pull request Jan 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants