[Feature][Connector-V2] Add Huawei Cloud OBS connector #4578

kim-up · 2023-04-14T10:47:24Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

TyrantLucifer · 2023-04-14T10:51:07Z

Good pull request, let's waiting CI

kim-up · 2023-04-17T03:56:00Z

The dependency of esdk-obs-java can not download.
https://mvnrepository.com/artifact/com.huawei.storage/esdk-obs-java/3.19.7.3

@TyrantLucifer How should this be handled?

TyrantLucifer · 2023-04-17T09:22:11Z

The dependency of esdk-obs-java can not download. https://mvnrepository.com/artifact/com.huawei.storage/esdk-obs-java/3.19.7.3

@TyrantLucifer How should this be handled?

You can find solution in oss-jindo module's pom file.

EricJoy2048 · 2023-04-17T09:30:34Z

docs/en/connector-v2/sink/ObsFile.md

@@ -0,0 +1,260 @@
+# ObsFile


After discussion in the community, we have optimized the format requirements of the document, which you can refer to：https://github.com/apache/incubator-seatunnel/issues/4544

hailin0 · 2023-04-17T16:10:14Z

Please add e2e testcase and make junit @disabled

kim-up · 2023-04-20T04:13:25Z

@TyrantLucifer @EricJoy2048 @hailin0 Is there anything else I need to do ?

kim-up · 2023-04-22T03:32:58Z

@TyrantLucifer @hailin0 @EricJoy2048 @jbonofre Asking for Help!. Are there any end-to-end test examples for file storage cloud services such as obs, oss, and s3? Because it involves sensitive information such as ak and sk. And cloud services do not necessarily have relevant docker images that can be used for Mock testing.

TyrantLucifer · 2023-04-22T06:33:35Z

Hi @kim-up , you can add a e2e test case and run it in your local env, before submit codes you should add annotation to disabled it and remove sensitive information from config file. Finally offer the snapshot image in this pull request. You can refer to selectdb-e2e and file-local-e2e. The idea is to allow novices to quickly find test cases for the connector and run through the code framework you provide by directly changing some key information

kim-up · 2023-04-23T07:55:17Z

Hi @kim-up , you can add a e2e test case and run it in your local env, before submit codes you should add annotation to disabled it and remove sensitive information from config file. Finally offer the snapshot image in this pull request. You can refer to selectdb-e2e and file-local-e2e. The idea is to allow novices to quickly find test cases for the connector and run through the code framework you provide by directly changing some key information

Get it.

…tor-seatunnel into connector-file-obs

kim-up · 2023-04-24T09:02:02Z

Docs & e2e have add. Start the workflow? @TyrantLucifer @EricJoy2048 @hailin0

TyrantLucifer · 2023-04-24T14:57:43Z

Docs & e2e have add. Start the workflow? @TyrantLucifer @EricJoy2048 @hailin0

Approved, let's waiting CI/CD, BTW could you please offer some snapshots of this connector?

kim-up · 2023-04-25T04:11:46Z

Docs & e2e have add. Start the workflow? @TyrantLucifer @EricJoy2048 @hailin0

Approved, let's waiting CI/CD, BTW could you please offer some snapshots of this connector?

What is some snapshots ? Docker image ? Is there any related document that can refer to...

kim-up · 2023-04-26T06:28:13Z

Local Test Images

Csv Test

fake_to_obs_csv

env {
  execution.parallelism = 1
  checkpoint.interval = 5000
  job.mode = "BATCH"
}

source {
  FakeSource {
    result_table_name = "fake"
    schema = {
      fields {
        c_map = "map<string, string>"
        c_array = "array<int>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_bytes = bytes
        c_date = date
        c_decimal = "decimal(38, 18)"
        c_timestamp = timestamp
        c_row = {
          c_map = "map<string, string>"
          c_array = "array<int>"
          c_string = string
          c_boolean = boolean
          c_tinyint = tinyint
          c_smallint = smallint
          c_int = int
          c_bigint = bigint
          c_float = float
          c_double = double
          c_bytes = bytes
          c_date = date
          c_decimal = "decimal(38, 18)"
          c_timestamp = timestamp
        }
      }
    }
  }
}

sink {
  ObsFile {
    path="/seatunnel/csv"
    bucket = "obs://********"
    access_key = "********"
    access_secret = "********"
    endpoint = "obs.cn-east-3.myhuaweicloud.com"
    partition_dir_expression="${k0}=${v0}"
    is_partition_field_write_in_file=true
    file_name_expression="${transactionId}_${now}"
    file_format_type="csv"
    filename_time_format="yyyy.MM.dd"
    is_enable_transaction=true
  }
}

Result

obs_csv_projection_to_assert

env {
  execution.parallelism = 1
  checkpoint.interval = 5000
  job.mode = "BATCH"
}

source {
  ObsFile {
    path="/seatunnel/csv"
    bucket = "obs://******"
    access_key = "******"
    access_secret = "******"
    endpoint = "obs.cn-east-3.myhuaweicloud.com"
    result_table_name = "fake"
    file_format_type = csv
    delimiter = ","
    read_columns = [c_string, c_boolean]
    skip_header_row_number = 1
    schema = {
      fields {
          c_map = "map<string, string>"
          c_array = "array<int>"
          c_string = string
          c_boolean = boolean
          c_tinyint = tinyint
          c_smallint = smallint
          c_int = int
          c_bigint = bigint
          c_float = float
          c_double = double
          c_bytes = bytes
          c_date = date
          c_decimal = "decimal(38, 18)"
          c_timestamp = timestamp
          c_row = {
            c_map = "map<string, string>"
            c_array = "array<int>"
            c_string = string
            c_boolean = boolean
            c_tinyint = tinyint
            c_smallint = smallint
            c_int = int
            c_bigint = bigint
            c_float = float
            c_double = double
            c_bytes = bytes
            c_date = date
            c_decimal = "decimal(38, 18)"
            c_timestamp = timestamp
          }
      }
    }
  }
}


sink {
  Assert {
    rules {
      row_rules = [
        {
          rule_type = MAX_ROW
          rule_value = 5
        }
      ],
      field_rules = [
        {
          field_name = c_string
          field_type = string
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        },
        {
          field_name = c_boolean
          field_type = boolean
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        }
      ]
    }
  }
}

Result

kim-up · 2023-04-26T06:40:10Z

Local Test Images

Excel Test

fake_to_obs_excel

env {
  execution.parallelism = 1
  checkpoint.interval = 5000
  job.mode = "BATCH"
}

source {
  FakeSource {
    result_table_name = "fake"
    schema = {
      fields {
        c_map = "map<string, string>"
        c_array = "array<int>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_bytes = bytes
        c_date = date
        c_decimal = "decimal(38, 18)"
        c_timestamp = timestamp
        c_row = {
          c_map = "map<string, string>"
          c_array = "array<int>"
          c_string = string
          c_boolean = boolean
          c_tinyint = tinyint
          c_smallint = smallint
          c_int = int
          c_bigint = bigint
          c_float = float
          c_double = double
          c_bytes = bytes
          c_date = date
          c_decimal = "decimal(38, 18)"
          c_timestamp = timestamp
        }
      }
    }
  }
}

sink {
  ObsFile {
    path="/seatunnel/excel"
    bucket = "obs://******"
    access_key = "******"
    access_secret = "******"
    endpoint = "obs.cn-east-3.myhuaweicloud.com"
    partition_dir_expression="${k0}=${v0}"
    is_partition_field_write_in_file=true
    file_name_expression="${transactionId}_${now}"
    file_format_type="excel"
    filename_time_format="yyyy.MM.dd"
    is_enable_transaction=true
  }
}

Result

obs_excel_projection_to_assert

env {
 execution.parallelism = 1
  checkpoint.interval = 5000
  job.mode = "BATCH"
}

source {
  ObsFile {
    path="/seatunnel/excel"
    bucket = "obs://******"
    access_key = "******"
    access_secret = "******"
    endpoint = "obs.cn-east-3.myhuaweicloud.com"
    result_table_name = "fake"
    file_format_type = excel
    delimiter = ;
    read_columns = [c_string, c_boolean]
    skip_header_row_number = 1
    schema = {
      fields {
          c_map = "map<string, string>"
          c_array = "array<int>"
          c_string = string
          c_boolean = boolean
          c_tinyint = tinyint
          c_smallint = smallint
          c_int = int
          c_bigint = bigint
          c_float = float
          c_double = double
          c_bytes = bytes
          c_date = date
          c_decimal = "decimal(38, 18)"
          c_timestamp = timestamp
          c_row = {
            c_map = "map<string, string>"
            c_array = "array<int>"
            c_string = string
            c_boolean = boolean
            c_tinyint = tinyint
            c_smallint = smallint
            c_int = int
            c_bigint = bigint
            c_float = float
            c_double = double
            c_bytes = bytes
            c_date = date
            c_decimal = "decimal(38, 18)"
            c_timestamp = timestamp
          }
      }
    }
  }
}


sink {
  Assert {
    rules {
      row_rules = [
        {
          rule_type = MAX_ROW
          rule_value = 5
        }
      ],
      field_rules = [
        {
          field_name = c_string
          field_type = string
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        },
        {
          field_name = c_boolean
          field_type = boolean
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        }
      ]
    }
  }
}

Result

kim-up · 2023-04-26T06:46:22Z

Local Test Images

Json Test

fake_to_obs_file_json

env {
  execution.parallelism = 1
  checkpoint.interval = 5000
  job.mode = "BATCH"
}

source {
  FakeSource {
    schema = {
      fields {
        c_map = "map<string, string>"
        c_array = "array<int>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_bytes = bytes
        c_date = date
        c_decimal = "decimal(38, 18)"
        c_timestamp = timestamp
        c_row = {
          c_map = "map<string, string>"
          c_array = "array<int>"
          c_string = string
          c_boolean = boolean
          c_tinyint = tinyint
          c_smallint = smallint
          c_int = int
          c_bigint = bigint
          c_float = float
          c_double = double
          c_bytes = bytes
          c_date = date
          c_decimal = "decimal(38, 18)"
          c_timestamp = timestamp
        }
      }
    }
    result_table_name = "fake"
  }
}

sink {
  ObsFile {
    path = "/seatunnel/json"
    bucket = "obs://******"
    access_key = "******"
    access_secret = "******"
    endpoint = "obs.cn-east-3.myhuaweicloud.com"
    row_delimiter = "\n"
    partition_dir_expression = "${k0}=${v0}"
    is_partition_field_write_in_file = true
    file_name_expression = "${transactionId}_${now}"
    file_format_type = "json"
    filename_time_format = "yyyy.MM.dd"
    is_enable_transaction = true
  }
}

Result

obs_file_json_to_assert

env {
  execution.parallelism = 1
  checkpoint.interval = 5000
  job.mode = "BATCH"
}

source {
  ObsFile {
    path = "/seatunnel/json"
    bucket = "obs://******"
    access_key = "******"
    access_secret = "******"
    endpoint = "obs.cn-east-3.myhuaweicloud.com"
    file_format_type = "json"
    schema = {
      fields {
        c_map = "map<string, string>"
        c_array = "array<int>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_bytes = bytes
        c_date = date
        c_decimal = "decimal(38, 18)"
        c_timestamp = timestamp
        c_row = {
          c_map = "map<string, string>"
          c_array = "array<int>"
          c_string = string
          c_boolean = boolean
          c_tinyint = tinyint
          c_smallint = smallint
          c_int = int
          c_bigint = bigint
          c_float = float
          c_double = double
          c_bytes = bytes
          c_date = date
          c_decimal = "decimal(38, 18)"
          c_timestamp = timestamp
        }
      }
    }
    result_table_name = "fake"
  }
}

sink {
  Assert {
    rules {
      row_rules = [
        {
          rule_type = MAX_ROW
          rule_value = 5
        }
      ],
      field_rules = [
        {
          field_name = c_string
          field_type = string
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        },
        {
          field_name = c_boolean
          field_type = boolean
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        },
        {
          field_name = c_double
          field_type = double
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        }
      ]
    }
  }
}

Result

kim-up · 2023-04-26T06:53:01Z

Local Test Images

Orc Test

fake_to_obs_file_orc

env {
  execution.parallelism = 1
  checkpoint.interval = 5000
  job.mode = "BATCH"
}

source {
  FakeSource {
    schema = {
      fields {
        c_map = "map<string, string>"
        c_array = "array<int>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_bytes = bytes
        c_date = date
        c_decimal = "decimal(38, 18)"
        c_timestamp = timestamp
        c_row = {
          c_map = "map<string, string>"
          c_array = "array<int>"
          c_string = string
          c_boolean = boolean
          c_tinyint = tinyint
          c_smallint = smallint
          c_int = int
          c_bigint = bigint
          c_float = float
          c_double = double
          c_bytes = bytes
          c_date = date
          c_decimal = "decimal(38, 18)"
          c_timestamp = timestamp
        }
      }
    }
    result_table_name = "fake"
  }
}

sink {
  ObsFile {
    path = "/seatunnel/orc"
    bucket = "obs://*******"
    access_key = "*******"
    access_secret = "*******"
    endpoint = "obs.cn-east-3.myhuaweicloud.com"
    row_delimiter = "\n"
    partition_dir_expression = "${k0}=${v0}"
    is_partition_field_write_in_file = true
    file_name_expression = "${transactionId}_${now}"
    file_format_type = "orc"
    filename_time_format = "yyyy.MM.dd"
    is_enable_transaction = true
    compress_codec = "zlib"
  }
}

Result

obs_file_orc_projection_to_assert

env {
  execution.parallelism = 1
  checkpoint.interval = 5000
  job.mode = "BATCH"
}

source {
  ObsFile {
    path = "/seatunnel/orc"
    bucket = "obs://******"
    access_key = "******"
    access_secret = "******"
    endpoint = "obs.cn-east-3.myhuaweicloud.com"
    file_format_type = "orc"
    read_columns = [c_string, c_boolean, c_double]
    result_table_name = "fake"
  }
}

sink {
  Assert {
    rules {
      row_rules = [
        {
          rule_type = MAX_ROW
          rule_value = 5
        }
      ],
      field_rules = [
        {
          field_name = c_string
          field_type = string
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        },
        {
          field_name = c_boolean
          field_type = boolean
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        },
        {
          field_name = c_double
          field_type = double
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        }
      ]
    }
  }
}

Result

kim-up · 2023-04-26T07:01:42Z

Local Test Images

Parquet Test

fake_to_obs_file_parquet

env {
  execution.parallelism = 1
  checkpoint.interval = 5000
  job.mode = "BATCH"
}

source {
  FakeSource {
    schema = {
      fields {
        c_map = "map<string, string>"
        c_array = "array<int>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_bytes = bytes
        c_date = date
        c_decimal = "decimal(38, 18)"
        c_timestamp = timestamp
        c_row = {
          c_map = "map<string, string>"
          c_array = "array<int>"
          c_string = string
          c_boolean = boolean
          c_tinyint = tinyint
          c_smallint = smallint
          c_int = int
          c_bigint = bigint
          c_float = float
          c_double = double
          c_bytes = bytes
          c_date = date
          c_decimal = "decimal(38, 18)"
          c_timestamp = timestamp
        }
      }
    }
    result_table_name = "fake"
  }
}

sink {
  ObsFile {
    path = "/seatunnel/parquet"
    bucket = "obs://******"
    access_key = "******"
    access_secret = "******"
    endpoint = "obs.cn-east-3.myhuaweicloud.com"
    row_delimiter = "\n"
    partition_dir_expression = "${k0}=${v0}"
    is_partition_field_write_in_file = true
    file_name_expression = "${transactionId}_${now}"
    file_format_type = "parquet"
    filename_time_format = "yyyy.MM.dd"
    is_enable_transaction = true
    compress_codec = "gzip"
  }
}

Result

obs_file_parquet_projection_to_assert

env {
  execution.parallelism = 1
  checkpoint.interval = 5000
  job.mode = "BATCH"
}

source {
  ObsFile {
    path = "/seatunnel/parquet"
    bucket = "obs://******"
    access_key = "******"
    access_secret = "******"
    endpoint = "obs.cn-east-3.myhuaweicloud.com"
    file_format_type = "parquet"
    read_columns = [c_string, c_boolean, c_double]
    result_table_name = "fake"
  }
}

sink {
  Assert {
    rules {
      row_rules = [
        {
          rule_type = MAX_ROW
          rule_value = 5
        }
      ],
      field_rules = [
        {
          field_name = c_string
          field_type = string
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        },
        {
          field_name = c_boolean
          field_type = boolean
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        },
        {
          field_name = c_double
          field_type = double
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        }
      ]
    }
  }
}

Result

kim-up · 2023-04-26T07:15:07Z

Local Test Images

Text Test

fake_to_obs_file_text

env {
  execution.parallelism = 1
  checkpoint.interval = 5000
  job.mode = "BATCH"
}

source {
  FakeSource {
    schema = {
      fields {
        c_map = "map<string, string>"
        c_array = "array<int>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_bytes = bytes
        c_date = date
        c_decimal = "decimal(38, 18)"
        c_timestamp = timestamp
        c_row = {
          c_map = "map<string, string>"
          c_array = "array<int>"
          c_string = string
          c_boolean = boolean
          c_tinyint = tinyint
          c_smallint = smallint
          c_int = int
          c_bigint = bigint
          c_float = float
          c_double = double
          c_bytes = bytes
          c_date = date
          c_decimal = "decimal(38, 18)"
          c_timestamp = timestamp
        }
      }
    }
    result_table_name = "fake"
  }
}

sink {
  ObsFile {
    path = "/seatunnel/text"
    bucket = "obs://******"
    access_key = "******"
    access_secret = "******"
    endpoint = "obs.cn-east-3.myhuaweicloud.com"
    row_delimiter = "\n"
    partition_dir_expression = "${k0}=${v0}"
    is_partition_field_write_in_file = true
    file_name_expression = "${transactionId}_${now}"
    file_format_type = "text"
    filename_time_format = "yyyy.MM.dd"
    is_enable_transaction = true
  }
}

Result

obs_file_text_projection_to_assert

env {
  execution.parallelism = 1
  checkpoint.interval = 5000
  job.mode = "BATCH"
}

source {
  ObsFile {
    path = "/seatunnel/text"
    bucket = "obs://******"
    access_key = "******"
    access_secret = "******"
    endpoint = "obs.cn-east-3.myhuaweicloud.com"
    file_format_type = "text"
    read_columns = [c_string, c_boolean, c_double]
    schema = {
      fields {
        c_map = "map<string, string>"
        c_array = "array<int>"
        c_string = string
        c_boolean = boolean
        c_tinyint = tinyint
        c_smallint = smallint
        c_int = int
        c_bigint = bigint
        c_float = float
        c_double = double
        c_bytes = bytes
        c_date = date
        c_decimal = "decimal(38, 18)"
        c_timestamp = timestamp
        c_row = {
          c_map = "map<string, string>"
          c_array = "array<int>"
          c_string = string
          c_boolean = boolean
          c_tinyint = tinyint
          c_smallint = smallint
          c_int = int
          c_bigint = bigint
          c_float = float
          c_double = double
          c_bytes = bytes
          c_date = date
          c_decimal = "decimal(38, 18)"
          c_timestamp = timestamp
        }
      }
    }
    result_table_name = "fake"
  }
}

sink {
  Assert {
    rules {
      row_rules = [
        {
          rule_type = MAX_ROW
          rule_value = 5
        }
      ],
      field_rules = [
        {
          field_name = c_string
          field_type = string
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        },
        {
          field_name = c_boolean
          field_type = boolean
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        },
        {
          field_name = c_double
          field_type = double
          field_value = [
            {
              rule_type = NOT_NULL
            }
          ]
        }
      ]
    }
  }
}

Result

kim-up · 2023-04-26T07:22:15Z

@TyrantLucifer Did you mean the test screenshot provided above?

…tor-seatunnel into connector-file-obs # Conflicts: # release-note.md

ic4y · 2023-08-06T12:42:27Z

@kim-up Please use mvn spotless:apply to format the code.

kim-up · 2023-08-08T10:04:56Z

@kim-up Please use mvn spotless:apply to format the code.

Done. Fixed by merging dev .

EricJoy2048 · 2024-06-14T10:07:27Z

Hi, @kim-up
You submitted a great PR, but it hasn't been updated in a while, can you update it?

kim-up added 3 commits April 14, 2023 18:32

[Feature][Connector-V2] Add huawei cloud obs connector apache#4577

cf01a17

[Fix][Merge] merge from dev

a0d2923

[Fix][plugin-mapping] update plugin-mapping.properties

2c6746a

kim-up changed the title ~~[Feature][Connector-V2] Add Huawei Cloud OBS connector #4577~~ [Feature][Connector-V2] Add Huawei Cloud OBS connector Apr 14, 2023

kim-up added 3 commits April 15, 2023 11:20

[Connector-V2][Obs] Add license for pom.xml

66d6eca

[Connector-V2][Obs] Update dist pom.xml

b19a433

[Fix][Merge] merge and fix conflicts

15a8f2a

[Fix][dependency] add huawei cloud repository

f6174c0

EricJoy2048 reviewed Apr 17, 2023

View reviewed changes

[Doc][obs] Unified format

35a25c3

kim-up requested a review from EricJoy2048 April 18, 2023 12:24

Merge branch 'dev' into connector-file-obs

d4da29a

kim-up added 6 commits April 23, 2023 16:49

Merge branch 'apache:dev' into connector-file-obs

43564f5

[Connector-v2][obs] Add e2e

993ea6a

Merge branch 'dev' into connector-file-obs

9eb8464

[Connector-v2][obs] Update e2e

c49aca9

Merge branch 'connector-file-obs' of https://github.com/kim-up/incuba…

e01d92a

…tor-seatunnel into connector-file-obs

Merge branch 'dev' into connector-file-obs

dca5836

EricJoy2048 and others added 7 commits April 28, 2023 13:53

Merge branch 'dev' into connector-file-obs

38f2465

[Connector-V2][file-obs] Merge dev and fix conflicts

5a6e9d9

[connector-V2][file-obs] Merge dev

8b45ef9

Merge branch 'dev' into connector-file-obs

17f27e3

Merge branch 'connector-file-obs' of https://github.com/kim-up/incuba…

6b5158a

…tor-seatunnel into connector-file-obs # Conflicts: # release-note.md

[Connector-V2][file] fix pom

802ca97

[Connector-V2][file] fix pom

05aa579

Merge remote-tracking branch 'origin' into connector-file-obs

479aefd

Merge branch 'dev' into connector-file-obs

cecda1a

hailin0 added 2 commits June 15, 2024 10:46

Merge branch 'dev' into pr/4578

3683454

fix

f33c57b

EricJoy2048 approved these changes Jun 15, 2024

View reviewed changes

github-actions bot added approved reviewed labels Jun 15, 2024

hailin0 approved these changes Jun 15, 2024

View reviewed changes

hailin0 merged commit d266f4d into apache:dev Jun 15, 2024
5 of 6 checks passed

hailin0 mentioned this pull request Jun 15, 2024

[Feature] [Connector-V2] Support connector-file-obs #6000

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2] Add Huawei Cloud OBS connector #4578

[Feature][Connector-V2] Add Huawei Cloud OBS connector #4578

kim-up commented Apr 14, 2023

TyrantLucifer commented Apr 14, 2023

kim-up commented Apr 17, 2023

TyrantLucifer commented Apr 17, 2023

EricJoy2048 Apr 17, 2023

kim-up Apr 17, 2023

hailin0 commented Apr 17, 2023

kim-up commented Apr 20, 2023

kim-up commented Apr 22, 2023

TyrantLucifer commented Apr 22, 2023

kim-up commented Apr 23, 2023

kim-up commented Apr 24, 2023

TyrantLucifer commented Apr 24, 2023

kim-up commented Apr 25, 2023 •

edited

Loading

kim-up commented Apr 26, 2023 •

edited

Loading

kim-up commented Apr 26, 2023

kim-up commented Apr 26, 2023

kim-up commented Apr 26, 2023

kim-up commented Apr 26, 2023

kim-up commented Apr 26, 2023

kim-up commented Apr 26, 2023

ic4y commented Aug 6, 2023

kim-up commented Aug 8, 2023

EricJoy2048 commented Jun 14, 2024

[Feature][Connector-V2] Add Huawei Cloud OBS connector #4578

[Feature][Connector-V2] Add Huawei Cloud OBS connector #4578

Conversation

kim-up commented Apr 14, 2023

Purpose of this pull request

Check list

TyrantLucifer commented Apr 14, 2023

kim-up commented Apr 17, 2023

TyrantLucifer commented Apr 17, 2023

EricJoy2048 Apr 17, 2023

Choose a reason for hiding this comment

kim-up Apr 17, 2023

Choose a reason for hiding this comment

hailin0 commented Apr 17, 2023

kim-up commented Apr 20, 2023

kim-up commented Apr 22, 2023

TyrantLucifer commented Apr 22, 2023

kim-up commented Apr 23, 2023

kim-up commented Apr 24, 2023

TyrantLucifer commented Apr 24, 2023

kim-up commented Apr 25, 2023 • edited Loading

kim-up commented Apr 26, 2023 • edited Loading

Local Test Images

Csv Test

fake_to_obs_csv

Result

obs_csv_projection_to_assert

Result

kim-up commented Apr 26, 2023

Local Test Images

Excel Test

fake_to_obs_excel

Result

obs_excel_projection_to_assert

Result

kim-up commented Apr 26, 2023

Local Test Images

Json Test

fake_to_obs_file_json

Result

obs_file_json_to_assert

Result

kim-up commented Apr 26, 2023

Local Test Images

Orc Test

fake_to_obs_file_orc

Result

obs_file_orc_projection_to_assert

Result

kim-up commented Apr 26, 2023

Local Test Images

Parquet Test

fake_to_obs_file_parquet

Result

obs_file_parquet_projection_to_assert

Result

kim-up commented Apr 26, 2023

Local Test Images

Text Test

fake_to_obs_file_text

Result

obs_file_text_projection_to_assert

Result

kim-up commented Apr 26, 2023

ic4y commented Aug 6, 2023

kim-up commented Aug 8, 2023

EricJoy2048 commented Jun 14, 2024

kim-up commented Apr 25, 2023 •

edited

Loading

kim-up commented Apr 26, 2023 •

edited

Loading