[Improve] [Connector-V2] File Connector add lzo compression way. #3782

lightzhao · 2022-12-21T11:23:40Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist

.../src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseFileSinkConfig.java

TyrantLucifer · 2022-12-22T09:56:18Z

IMO, using the enum to limit the method of compress is better. Orc and parquet file format also have other compress method

lightzhao · 2022-12-22T10:46:37Z

IMO, using the enum to limit the method of compress is better. Orc and parquet file format also have other compress method

Add compression enum completed.

TyrantLucifer · 2022-12-22T11:28:31Z

.../src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseFileSinkConfig.java

@@ -50,7 +50,7 @@ public class BaseFileSinkConfig implements DelimiterConfig, CompressConfig, Seri

    public BaseFileSinkConfig(@NonNull Config config) {
        if (config.hasPath(BaseSinkConfig.COMPRESS_CODEC.key())) {
-            if ("lzo".equals(config.getString(BaseSinkConfig.COMPRESS_CODEC.key()))) {
+            if (CompressFormat.LZO.getCompressCodec().equals(config.getString(BaseSinkConfig.COMPRESS_CODEC.key()))) {


switch is better

done, thanks.

TyrantLucifer · 2022-12-22T11:28:50Z

.../main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/writer/TextWriteStrategy.java

@@ -115,7 +116,7 @@ private FSDataOutputStream getOrCreateOutputStream(@NonNull String filePath) {
        FSDataOutputStream fsDataOutputStream = beingWrittenOutputStream.get(filePath);
        if (fsDataOutputStream == null) {
            try {
-                if ("lzo".equals(compressCodec)) {
+                if (CompressFormat.LZO.getCompressCodec().equals(compressCodec)) {


switch is better

done, thanks.

Hisoka-X · 2022-12-24T13:32:20Z

Please add license header.

lightzhao · 2022-12-25T02:25:15Z

Please add license header.

done,thanks.

TyrantLucifer · 2022-12-25T04:05:45Z

Please fix CI problem.

lightzhao · 2022-12-26T03:37:48Z

Please fix CI problem.

done, thanks.

TyrantLucifer · 2022-12-27T07:42:15Z

Waiting CI completed.

TyrantLucifer · 2022-12-27T08:22:02Z

Could you please offer test screenshot images to verify this pull request is work? The following points need to be verified:

The file name is correct
Data can be read normally in hive or spark

lightzhao · 2022-12-27T08:48:53Z

Could you please offer test screenshot images to verify this pull request is work? The following points need to be verified:

The file name is correct

Data can be read normally in hive or spark

hdfs file：

hive query：

TyrantLucifer · 2022-12-27T11:12:05Z

Could you please offer test screenshot images to verify this pull request is work? The following points need to be verified:

The file name is correct

Data can be read normally in hive or spark

hdfs file： hive query：

*.lzo.txt might be better? We can use the suffix to Identify the file type quickly, BTW could you please test csv file type?

lightzhao · 2022-12-27T11:32:33Z

csv file type

The traditional method is to end with a compressed code, but it doesn't use the ending method you mentioned. csv is just a separation symbol problem, which has been tested

TyrantLucifer · 2022-12-27T12:38:32Z

csv file type

The traditional method is to end with a compressed code, but it doesn't use the ending method you mentioned. csv is just a separation symbol problem, which has been tested

cc @EricJoy2048

lightzhao · 2022-12-28T06:51:40Z

@TyrantLucifer plesase review and merge , thanks.

EricJoy2048 · 2022-12-28T07:02:48Z

Could you please offer test screenshot images to verify this pull request is work? The following points need to be verified:

The file name is correct

Data can be read normally in hive or spark

hdfs file： hive query：

*.lzo.txt might be better? We can use the suffix to Identify the file type quickly, BTW could you please test csv file type?

I agree with you, *.lzo.txt is better.

lightzhao · 2022-12-28T07:16:07Z

done, complete file name ends with *. lzo.txt.

lightzhao · 2022-12-28T11:00:05Z

@TyrantLucifer @Hisoka-X please review .thanks.

lightzhao · 2022-12-30T02:41:21Z

@EricJoy2048 plesase review .

…apache#3782) * add lzo compression way. * format code style * add CompressFormat enum. * Judge with switch statement. * fix CompressFormat enum valueOf adapt * Repair NPE when compression is not set. * add license header. * fix CI problem. * CompressFormat enum add default value NONE. * Supplement lzo compressed documents. * Supplement lzo compressed documents. * modify file name ends with *.lzo.txt Co-authored-by: zhaoliang01 <zhaoliang01@58.com>

add lzo compression way.

a81139e

Hisoka-X added improve First-time contributor First-time contributor connectors-v2 labels Dec 22, 2022

Hisoka-X changed the title ~~add lzo compression way.~~ [Improve] [Connector-V2] File Connector add lzo compression way. Dec 22, 2022

Hisoka-X requested a review from TyrantLucifer December 22, 2022 03:13

Hisoka-X reviewed Dec 22, 2022

View reviewed changes

.../src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseFileSinkConfig.java Outdated Show resolved Hide resolved

format code style

ee07d84

add CompressFormat enum.

380e56e

TyrantLucifer reviewed Dec 22, 2022

View reviewed changes

zhaoliang01 added 3 commits December 23, 2022 10:36

Judge with switch statement.

f999f47

fix CompressFormat enum valueOf adapt

b36b394

Repair NPE when compression is not set.

fbd72c8

lightzhao requested review from Hisoka-X and TyrantLucifer and removed request for Hisoka-X and TyrantLucifer December 24, 2022 12:27

add license header.

45e3679

Hisoka-X added the Waiting for code update label Dec 26, 2022

fix CI problem.

f88c762

lightzhao requested review from Hisoka-X and removed request for Hisoka-X December 27, 2022 02:37

github-actions bot added approved reviewed labels Dec 27, 2022

lightzhao requested a review from TyrantLucifer December 27, 2022 07:35

TyrantLucifer added the Waiting for code update label Dec 28, 2022

modify file name ends with *.lzo.txt

bf941a2

lightzhao dismissed Hisoka-X’s stale review via bf941a2 December 28, 2022 07:14

github-actions bot removed approved reviewed labels Dec 28, 2022

lightzhao requested review from Hisoka-X and removed request for TyrantLucifer December 28, 2022 10:59

Hisoka-X requested a review from EricJoy2048 December 29, 2022 02:03

Hisoka-X approved these changes Dec 29, 2022

View reviewed changes

github-actions bot added approved reviewed labels Dec 29, 2022

TyrantLucifer approved these changes Dec 30, 2022

View reviewed changes

TyrantLucifer merged commit 8875d02 into apache:dev Dec 30, 2022

lightzhao deleted the add-lzo-compression branch February 22, 2023 05:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Improve] [Connector-V2] File Connector add lzo compression way. #3782

[Improve] [Connector-V2] File Connector add lzo compression way. #3782

lightzhao commented Dec 21, 2022

TyrantLucifer commented Dec 22, 2022

lightzhao commented Dec 22, 2022

TyrantLucifer Dec 22, 2022

lightzhao Dec 23, 2022

TyrantLucifer Dec 22, 2022

lightzhao Dec 23, 2022

Hisoka-X commented Dec 24, 2022

lightzhao commented Dec 25, 2022

TyrantLucifer commented Dec 25, 2022

lightzhao commented Dec 26, 2022

TyrantLucifer commented Dec 27, 2022

TyrantLucifer commented Dec 27, 2022 •

edited

lightzhao commented Dec 27, 2022

TyrantLucifer commented Dec 27, 2022 •

edited

lightzhao commented Dec 27, 2022

TyrantLucifer commented Dec 27, 2022

lightzhao commented Dec 28, 2022

EricJoy2048 commented Dec 28, 2022

lightzhao commented Dec 28, 2022

lightzhao commented Dec 28, 2022

lightzhao commented Dec 30, 2022

[Improve] [Connector-V2] File Connector add lzo compression way. #3782

[Improve] [Connector-V2] File Connector add lzo compression way. #3782

Conversation

lightzhao commented Dec 21, 2022

Purpose of this pull request

Check list

TyrantLucifer commented Dec 22, 2022

lightzhao commented Dec 22, 2022

TyrantLucifer Dec 22, 2022

Choose a reason for hiding this comment

lightzhao Dec 23, 2022

Choose a reason for hiding this comment

TyrantLucifer Dec 22, 2022

Choose a reason for hiding this comment

lightzhao Dec 23, 2022

Choose a reason for hiding this comment

Hisoka-X commented Dec 24, 2022

lightzhao commented Dec 25, 2022

TyrantLucifer commented Dec 25, 2022

lightzhao commented Dec 26, 2022

TyrantLucifer commented Dec 27, 2022

TyrantLucifer commented Dec 27, 2022 • edited

lightzhao commented Dec 27, 2022

TyrantLucifer commented Dec 27, 2022 • edited

lightzhao commented Dec 27, 2022

TyrantLucifer commented Dec 27, 2022

lightzhao commented Dec 28, 2022

EricJoy2048 commented Dec 28, 2022

lightzhao commented Dec 28, 2022

lightzhao commented Dec 28, 2022

lightzhao commented Dec 30, 2022

TyrantLucifer commented Dec 27, 2022 •

edited

TyrantLucifer commented Dec 27, 2022 •

edited