diff --git a/docs/en/connector-v2/Error-Quick-Reference-Manual.md b/docs/en/connector-v2/Error-Quick-Reference-Manual.md index b2a8bb520af..99d7027c3c9 100644 --- a/docs/en/connector-v2/Error-Quick-Reference-Manual.md +++ b/docs/en/connector-v2/Error-Quick-Reference-Manual.md @@ -66,4 +66,12 @@ This document records some common error codes and corresponding solutions of Sea | RABBITMQ-06 | messages could not be acknowledged with basicReject | When users encounter this error code, it means that job has some problems, please check it whether is work well | | RABBITMQ-07 | parse uri failed | When users encounter this error code, it means that rabbitmq connect uri incorrect, please check it | | RABBITMQ-08 | initialize ssl context failed | When users encounter this error code, it means that rabbitmq has some problems, please check it whether is work | -| RABBITMQ-09 | setup ssl factory failed | When users encounter this error code, it means that rabbitmq has some problems, please check it whether is work | \ No newline at end of file +| RABBITMQ-09 | setup ssl factory failed | When users encounter this error code, it means that rabbitmq has some problems, please check it whether is work | + +## Socket Connector Error Codes + +| code | description | solution | +|-----------|----------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------| +| SOCKET-01 | Cannot connect to socket server | When the user encounters this error code, it means that the connection address may not match, please check | +| SOCKET-02 | Failed to send message to socket server | When the user encounters this error code, it means that there is a problem sending data and retry is not enabled, please check | +| SOCKET-03 | Unable to write; interrupted while doing another attempt | When the user encounters this error code, it means that the data writing is interrupted abnormally, please check | diff --git a/docs/en/connector-v2/source/Jira.md b/docs/en/connector-v2/source/Jira.md new file mode 100644 index 00000000000..0f170b695d8 --- /dev/null +++ b/docs/en/connector-v2/source/Jira.md @@ -0,0 +1,160 @@ +# Jira + +> Jira source connector + +## Description + +Used to read data from Jira. + +## Key features + +- [x] [batch](../../concept/connector-v2-features.md) +- [ ] [stream](../../concept/connector-v2-features.md) +- [ ] [exactly-once](../../concept/connector-v2-features.md) +- [x] [schema projection](../../concept/connector-v2-features.md) +- [ ] [parallelism](../../concept/connector-v2-features.md) +- [ ] [support user-defined split](../../concept/connector-v2-features.md) + +## Options + +| name | type | required | default value | +| --------------------------- | ------ | -------- | ------------- | +| url | String | Yes | - | +| email | String | Yes | - | +| api_token | String | Yes | - | +| method | String | No | get | +| schema.fields | Config | No | - | +| format | String | No | json | +| params | Map | No | - | +| body | String | No | - | +| poll_interval_ms | int | No | - | +| retry | int | No | - | +| retry_backoff_multiplier_ms | int | No | 100 | +| retry_backoff_max_ms | int | No | 10000 | +| common-options | config | No | - | + +### url [String] + +http request url + +### email [String] + +Jira Email + +### api_token [String] + +Jira API Token + +https://id.atlassian.com/manage-profile/security/api-tokens + +### method [String] + +http request method, only supports GET, POST method + +### params [Map] + +http params + +### body [String] + +http body + +### poll_interval_ms [int] + +request http api interval(millis) in stream mode + +### retry [int] + +The max retry times if request http return to `IOException` + +### retry_backoff_multiplier_ms [int] + +The retry-backoff times(millis) multiplier if request http failed + +### retry_backoff_max_ms [int] + +The maximum retry-backoff times(millis) if request http failed + +### format [String] + +the format of upstream data, now only support `json` `text`, default `json`. + +when you assign format is `json`, you should also assign schema option, for example: + +upstream data is the following: + +```json + +{"code": 200, "data": "get success", "success": true} + +``` + +you should assign schema as the following: + +```hocon + +schema { + fields { + code = int + data = string + success = boolean + } +} + +``` + +connector will generate data as the following: + +| code | data | success | +|------|-------------|---------| +| 200 | get success | true | + +when you assign format is `text`, connector will do nothing for upstream data, for example: + +upstream data is the following: + +```json + +{"code": 200, "data": "get success", "success": true} + +``` + +connector will generate data as the following: + +| content | +|---------| +| {"code": 200, "data": "get success", "success": true} | + +### schema [Config] + +#### fields [Config] + +the schema fields of upstream data + +### common options + +Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details + +## Example + +```hocon +Jira { + url = "https://liugddx.atlassian.net/rest/api/3/search" + email = "test@test.com" + api_token = "xxx" + schema { + fields { + expand = string + startAt = bigint + maxResults = int + total = int + } + } +} +``` + +## Changelog + +### next version + +- Add Jira Source Connector diff --git a/docs/en/faq.md b/docs/en/faq.md index af46ae20f26..47714c4b76d 100644 --- a/docs/en/faq.md +++ b/docs/en/faq.md @@ -345,7 +345,7 @@ Just configure hdfs-site.xml properly, refer to: https://www.cnblogs.com/suanec/ ## I want to learn the source code of SeaTunnel, where should I start? -SeaTunnel has a completely abstract and structured code implementation, and many people have chosen SeaTunnel As a way to learn Spark, you can learn the source code from the main program entry: [Seatunnel.java](https://github.com/apache/incubator-seatunnel/blob/72b57b22688f17376fe7e5cf522b4bdd3f62cce0/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/Seatunnel.java) +SeaTunnel has a completely abstract and structured code implementation, and many people have chosen SeaTunnel As a way to learn Spark, you can learn the source code from the main program entry: Seatunnel.java ## When SeaTunnel developers develop their own plugins, do they need to understand the SeaTunnel code? Should these code integrated into the SeaTunnel project? diff --git a/plugin-mapping.properties b/plugin-mapping.properties index 77a62baa1ba..2962c75b51d 100644 --- a/plugin-mapping.properties +++ b/plugin-mapping.properties @@ -152,6 +152,7 @@ seatunnel.source.Lemlist = connector-http-lemlist seatunnel.source.Klaviyo = connector-http-klaviyo seatunnel.sink.Slack = connector-slack seatunnel.source.OneSignal = connector-http-onesignal +seatunnel.source.Jira = connector-http-jira seatunnel.source.Gitlab = connector-http-gitlab seatunnel.sink.RabbitMQ = connector-rabbitmq seatunnel.source.RabbitMQ = connector-rabbitmq \ No newline at end of file diff --git a/pom.xml b/pom.xml index 726327b142c..832e199eca6 100644 --- a/pom.xml +++ b/pom.xml @@ -113,7 +113,7 @@ - 2.1.3-SNAPSHOT + 2.3.1-SNAPSHOT 2.1.1 UTF-8 1.8 diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/ConfigValidator.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/ConfigValidator.java index b6aa75a7362..4041dc56da6 100644 --- a/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/ConfigValidator.java +++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/ConfigValidator.java @@ -22,10 +22,9 @@ import org.apache.seatunnel.api.configuration.Option; import org.apache.seatunnel.api.configuration.ReadonlyConfig; -import java.util.HashSet; +import java.util.ArrayList; import java.util.List; import java.util.Objects; -import java.util.Set; public class ConfigValidator { private final ReadonlyConfig config; @@ -65,8 +64,8 @@ void validate(RequiredOption requiredOption) { throw new UnsupportedOperationException(String.format("This type option(%s) of validation is not supported", requiredOption.getClass())); } - private Set> getAbsentOptions(Set> requiredOption) { - Set> absent = new HashSet<>(); + private List> getAbsentOptions(List> requiredOption) { + List> absent = new ArrayList<>(); for (Option option : requiredOption) { if (!hasOption(option)) { absent.add(option); @@ -76,7 +75,7 @@ private Set> getAbsentOptions(Set> requiredOption) { } void validate(RequiredOption.AbsolutelyRequiredOptions requiredOption) { - Set> absentOptions = getAbsentOptions(requiredOption.getRequiredOption()); + List> absentOptions = getAbsentOptions(requiredOption.getRequiredOption()); if (absentOptions.size() == 0) { return; } @@ -88,9 +87,9 @@ boolean hasOption(Option option) { } boolean validate(RequiredOption.BundledRequiredOptions bundledRequiredOptions) { - Set> bundledOptions = bundledRequiredOptions.getRequiredOption(); - Set> present = new HashSet<>(); - Set> absent = new HashSet<>(); + List> bundledOptions = bundledRequiredOptions.getRequiredOption(); + List> present = new ArrayList<>(); + List> absent = new ArrayList<>(); for (Option option : bundledOptions) { if (hasOption(option)) { present.add(option); @@ -109,8 +108,8 @@ boolean validate(RequiredOption.BundledRequiredOptions bundledRequiredOptions) { } void validate(RequiredOption.ExclusiveRequiredOptions exclusiveRequiredOptions) { - Set presentBundledRequiredOptions = new HashSet<>(); - Set> presentOptions = new HashSet<>(); + List presentBundledRequiredOptions = new ArrayList<>(); + List> presentOptions = new ArrayList<>(); for (RequiredOption.BundledRequiredOptions bundledOptions : exclusiveRequiredOptions.getExclusiveBundledOptions()) { if (validate(bundledOptions)) { presentBundledRequiredOptions.add(bundledOptions); @@ -142,7 +141,7 @@ void validate(RequiredOption.ConditionalRequiredOptions conditionalRequiredOptio if (!match) { return; } - Set> absentOptions = getAbsentOptions(conditionalRequiredOptions.getRequiredOption()); + List> absentOptions = getAbsentOptions(conditionalRequiredOptions.getRequiredOption()); if (absentOptions.size() == 0) { return; } diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/OptionRule.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/OptionRule.java index 967cfdc6586..5b07e00e34f 100644 --- a/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/OptionRule.java +++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/OptionRule.java @@ -21,7 +21,6 @@ import java.util.ArrayList; import java.util.Arrays; -import java.util.HashSet; import java.util.List; import java.util.Objects; @@ -185,7 +184,7 @@ public Builder conditional(Expression expression, Option... requiredOptions) verifyRequiredOptionDefaultValue(o); } this.requiredOptions.add(RequiredOption.ConditionalRequiredOptions.of(expression, - new HashSet<>(Arrays.asList(requiredOptions)))); + new ArrayList<>(Arrays.asList(requiredOptions)))); return this; } diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/OptionUtil.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/OptionUtil.java index 8027087c32d..2e16daeba9b 100644 --- a/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/OptionUtil.java +++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/OptionUtil.java @@ -26,16 +26,14 @@ import java.lang.reflect.Type; import java.util.ArrayList; import java.util.Collections; -import java.util.HashSet; import java.util.List; -import java.util.Set; public class OptionUtil { private OptionUtil() { } - public static String getOptionKeys(Set> options) { + public static String getOptionKeys(List> options) { StringBuilder builder = new StringBuilder(); boolean flag = false; for (Option option : options) { @@ -50,17 +48,18 @@ public static String getOptionKeys(Set> options) { return builder.toString(); } - public static String getOptionKeys(Set> options, Set bundledOptions) { - Set>> optionSets = new HashSet<>(); + public static String getOptionKeys(List> options, + List bundledOptions) { + List>> optionList = new ArrayList<>(); for (Option option : options) { - optionSets.add(Collections.singleton(option)); + optionList.add(Collections.singletonList(option)); } for (RequiredOption.BundledRequiredOptions bundledOption : bundledOptions) { - optionSets.add(bundledOption.getRequiredOption()); + optionList.add(bundledOption.getRequiredOption()); } boolean flag = false; StringBuilder builder = new StringBuilder(); - for (Set> optionSet : optionSets) { + for (List> optionSet : optionList) { if (flag) { builder.append(", "); } @@ -80,7 +79,8 @@ public static List> getOptions(Class clazz) throws InstantiationExc field.setAccessible(true); OptionMark option = field.getAnnotation(OptionMark.class); if (option != null) { - options.add(new Option<>(!StringUtils.isNotBlank(option.name()) ? formatUnderScoreCase(field.getName()) : option.name(), + options.add(new Option<>( + !StringUtils.isNotBlank(option.name()) ? formatUnderScoreCase(field.getName()) : option.name(), new TypeReference() { @Override public Type getType() { diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/RequiredOption.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/RequiredOption.java index be28d60b446..85186aea2ba 100644 --- a/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/RequiredOption.java +++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/RequiredOption.java @@ -23,10 +23,10 @@ import lombok.Getter; +import java.util.ArrayList; import java.util.Arrays; -import java.util.HashSet; +import java.util.List; import java.util.Objects; -import java.util.Set; public interface RequiredOption { @@ -35,20 +35,20 @@ public interface RequiredOption { */ @Getter class ExclusiveRequiredOptions implements RequiredOption { - private final Set exclusiveBundledOptions; - private final Set> exclusiveOptions; + private final List exclusiveBundledOptions; + private final List> exclusiveOptions; - ExclusiveRequiredOptions(Set exclusiveBundledOptions, Set> exclusiveOptions) { + ExclusiveRequiredOptions(List exclusiveBundledOptions, List> exclusiveOptions) { this.exclusiveBundledOptions = exclusiveBundledOptions; this.exclusiveOptions = exclusiveOptions; } public static ExclusiveRequiredOptions of(Option... exclusiveOptions) { - return ExclusiveRequiredOptions.of(new HashSet<>(), exclusiveOptions); + return ExclusiveRequiredOptions.of(new ArrayList<>(), exclusiveOptions); } - public static ExclusiveRequiredOptions of(Set exclusiveBundledOptions, Option... exclusiveOptions) { - return new ExclusiveRequiredOptions(exclusiveBundledOptions, new HashSet<>(Arrays.asList(exclusiveOptions))); + public static ExclusiveRequiredOptions of(List exclusiveBundledOptions, Option... exclusiveOptions) { + return new ExclusiveRequiredOptions(exclusiveBundledOptions, new ArrayList<>(Arrays.asList(exclusiveOptions))); } @Override @@ -79,14 +79,14 @@ public String toString() { */ class AbsolutelyRequiredOptions implements RequiredOption { @Getter - private final Set> requiredOption; + private final List> requiredOption; - AbsolutelyRequiredOptions(Set> requiredOption) { + AbsolutelyRequiredOptions(List> requiredOption) { this.requiredOption = requiredOption; } public static AbsolutelyRequiredOptions of(Option... requiredOption) { - return new AbsolutelyRequiredOptions(new HashSet<>(Arrays.asList(requiredOption))); + return new AbsolutelyRequiredOptions(new ArrayList<>(Arrays.asList(requiredOption))); } @Override @@ -114,18 +114,18 @@ public String toString() { class ConditionalRequiredOptions implements RequiredOption { private final Expression expression; - private final Set> requiredOption; + private final List> requiredOption; - ConditionalRequiredOptions(Expression expression, Set> requiredOption) { + ConditionalRequiredOptions(Expression expression, List> requiredOption) { this.expression = expression; this.requiredOption = requiredOption; } - public static ConditionalRequiredOptions of(Expression expression, Set> requiredOption) { + public static ConditionalRequiredOptions of(Expression expression, List> requiredOption) { return new ConditionalRequiredOptions(expression, requiredOption); } - public static ConditionalRequiredOptions of(Condition condition, Set> requiredOption) { + public static ConditionalRequiredOptions of(Condition condition, List> requiredOption) { return new ConditionalRequiredOptions(Expression.of(condition), requiredOption); } @@ -133,7 +133,7 @@ public Expression getExpression() { return expression; } - public Set> getRequiredOption() { + public List> getRequiredOption() { return requiredOption; } @@ -164,21 +164,21 @@ public String toString() { * These options are bundled, must be present or absent together. */ class BundledRequiredOptions implements RequiredOption { - private final Set> requiredOption; + private final List> requiredOption; - BundledRequiredOptions(Set> requiredOption) { + BundledRequiredOptions(List> requiredOption) { this.requiredOption = requiredOption; } public static BundledRequiredOptions of(Option... requiredOption) { - return new BundledRequiredOptions(new HashSet<>(Arrays.asList(requiredOption))); + return new BundledRequiredOptions(new ArrayList<>(Arrays.asList(requiredOption))); } - public static BundledRequiredOptions of(Set> requiredOption) { + public static BundledRequiredOptions of(List> requiredOption) { return new BundledRequiredOptions(requiredOption); } - public Set> getRequiredOption() { + public List> getRequiredOption() { return requiredOption; } diff --git a/seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/TableSchema.java b/seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/TableSchema.java index 61d99ef5262..0d076943e76 100644 --- a/seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/TableSchema.java +++ b/seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/TableSchema.java @@ -18,6 +18,7 @@ package org.apache.seatunnel.api.table.catalog; import org.apache.seatunnel.api.table.type.SeaTunnelDataType; +import org.apache.seatunnel.api.table.type.SeaTunnelRowType; import java.io.Serializable; import java.util.ArrayList; @@ -49,6 +50,18 @@ public List getColumns() { return columns; } + public SeaTunnelRowType toPhysicalRowDataType() { + SeaTunnelDataType[] fieldTypes = columns.stream() + .filter(Column::isPhysical) + .map(Column::getDataType) + .toArray(SeaTunnelDataType[]::new); + String[] fields = columns.stream() + .filter(Column::isPhysical) + .map(Column::getName) + .toArray(String[]::new); + return new SeaTunnelRowType(fields, fieldTypes); + } + public static final class Builder { private final List columns = new ArrayList<>(); diff --git a/seatunnel-api/src/test/java/org/apache/seatunnel/api/configuration/util/ConfigValidatorTest.java b/seatunnel-api/src/test/java/org/apache/seatunnel/api/configuration/util/ConfigValidatorTest.java index 1242b8b2e12..8e172ff0495 100644 --- a/seatunnel-api/src/test/java/org/apache/seatunnel/api/configuration/util/ConfigValidatorTest.java +++ b/seatunnel-api/src/test/java/org/apache/seatunnel/api/configuration/util/ConfigValidatorTest.java @@ -34,10 +34,10 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.api.function.Executable; +import java.util.ArrayList; import java.util.HashMap; -import java.util.HashSet; +import java.util.List; import java.util.Map; -import java.util.Set; public class ConfigValidatorTest { public static final Option KEY_USERNAME = @@ -78,7 +78,7 @@ public void testAbsolutelyRequiredOption() { // absent config.put(TEST_PORTS.key(), "[9090]"); - assertEquals("ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - There are unconfigured options, the options('password', 'username') are required.", + assertEquals("ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - There are unconfigured options, the options('username', 'password') are required.", assertThrows(OptionValidationException.class, executable).getMessage()); config.put(KEY_USERNAME.key(), "asuka"); @@ -102,7 +102,7 @@ public void testBundledRequiredOptions() { // case2: some present config.put(KEY_USERNAME.key(), "asuka"); - assertEquals("ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - These options('password', 'username') are bundled, must be present or absent together." + + assertEquals("ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - These options('username', 'password') are bundled, must be present or absent together." + " The options present are: 'username'. The options absent are 'password'.", assertThrows(OptionValidationException.class, executable).getMessage()); @@ -137,7 +137,7 @@ public void testSimpleExclusiveRequiredOptions() { @Test public void testComplexExclusiveRequiredOptions() { - Set exclusiveBundledOptions = new HashSet<>(); + List exclusiveBundledOptions = new ArrayList<>(); exclusiveBundledOptions.add(RequiredOption.BundledRequiredOptions.of(KEY_USERNAME, KEY_PASSWORD)); OptionRule rule = OptionRule.builder() .exclusive(RequiredOption.ExclusiveRequiredOptions.of(exclusiveBundledOptions, KEY_BEARER_TOKEN, KEY_KERBEROS_TICKET)) @@ -147,13 +147,13 @@ public void testComplexExclusiveRequiredOptions() { Executable executable = () -> validate(config, rule); // all absent - assertEquals("ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - There are unconfigured options, these options(['kerberos-ticket'], ['password', 'username'], ['bearer-token']) are mutually exclusive," + + assertEquals("ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - There are unconfigured options, these options(['bearer-token'], ['kerberos-ticket'], ['username', 'password']) are mutually exclusive," + " allowing only one set(\"[] for a set\") of options to be configured.", assertThrows(OptionValidationException.class, executable).getMessage()); // bundled option some present config.put(KEY_USERNAME.key(), "asuka"); - assertEquals("ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - These options('password', 'username') are bundled, must be present or absent together." + + assertEquals("ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - These options('username', 'password') are bundled, must be present or absent together." + " The options present are: 'username'. The options absent are 'password'.", assertThrows(OptionValidationException.class, executable).getMessage()); @@ -163,13 +163,13 @@ public void testComplexExclusiveRequiredOptions() { // tow set options present config.put(KEY_BEARER_TOKEN.key(), "ashulin"); - assertEquals("ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - These options(['password', 'username'], ['bearer-token']) are mutually exclusive," + + assertEquals("ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - These options(['bearer-token'], ['username', 'password']) are mutually exclusive," + " allowing only one set(\"[] for a set\") of options to be configured.", assertThrows(OptionValidationException.class, executable).getMessage()); // three set options present config.put(KEY_KERBEROS_TICKET.key(), "zongwen"); - assertEquals("ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - These options(['kerberos-ticket'], ['password', 'username'], ['bearer-token']) are mutually exclusive," + + assertEquals("ErrorCode:[API-02], ErrorDescription:[Option item validate failed] - These options(['bearer-token'], ['kerberos-ticket'], ['username', 'password']) are mutually exclusive," + " allowing only one set(\"[] for a set\") of options to be configured.", assertThrows(OptionValidationException.class, executable).getMessage()); } diff --git a/seatunnel-common/src/main/java/org/apache/seatunnel/common/utils/FileUtils.java b/seatunnel-common/src/main/java/org/apache/seatunnel/common/utils/FileUtils.java index a4738b6cce7..58f95e6626e 100644 --- a/seatunnel-common/src/main/java/org/apache/seatunnel/common/utils/FileUtils.java +++ b/seatunnel-common/src/main/java/org/apache/seatunnel/common/utils/FileUtils.java @@ -25,14 +25,33 @@ import java.io.FileOutputStream; import java.io.IOException; import java.io.PrintStream; +import java.net.MalformedURLException; +import java.net.URL; +import java.nio.file.FileVisitOption; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.util.Arrays; +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.Stream; @Slf4j public class FileUtils { + public static List searchJarFiles(@NonNull Path directory) throws IOException { + try (Stream paths = Files.walk(directory, FileVisitOption.FOLLOW_LINKS)) { + return paths.filter(path -> path.toString().endsWith(".jar")) + .map(path -> { + try { + return path.toUri().toURL(); + } catch (MalformedURLException e) { + throw new SeaTunnelException(e); + } + }).collect(Collectors.toList()); + } + } + public static String readFileToStr(Path path) { try { byte[] bytes = Files.readAllBytes(path); @@ -87,7 +106,7 @@ public static void createNewFile(String filePath) { * return the line number of file * * @param filePath The file need be read - * @return + * @return The file line number */ public static Long getFileLineNumber(@NonNull String filePath) { try { @@ -101,7 +120,7 @@ public static Long getFileLineNumber(@NonNull String filePath) { * return the line number of all files in the dirPath * * @param dirPath dirPath - * @return + * @return The file line number of dirPath */ public static Long getFileLineNumberFromDir(@NonNull String dirPath) { File file = new File(dirPath); @@ -135,7 +154,6 @@ public static void createNewDir(@NonNull String dirPath) { * clear dir and the sub dir * * @param filePath filePath - * @return */ public static void deleteFile(@NonNull String filePath) { File file = new File(filePath); diff --git a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/config/JdbcSourceConfigFactory.java b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/config/JdbcSourceConfigFactory.java index ccbb6de23fa..0f98969ce6f 100644 --- a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/config/JdbcSourceConfigFactory.java +++ b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/config/JdbcSourceConfigFactory.java @@ -17,11 +17,13 @@ package org.apache.seatunnel.connectors.cdc.base.config; +import org.apache.seatunnel.api.configuration.ReadonlyConfig; import org.apache.seatunnel.connectors.cdc.base.option.JdbcSourceOptions; import org.apache.seatunnel.connectors.cdc.base.option.SourceOptions; import java.time.Duration; import java.util.Arrays; +import java.util.Collections; import java.util.List; import java.util.Properties; @@ -177,6 +179,26 @@ public JdbcSourceConfigFactory startupOptions(StartupConfig startupConfig) { return this; } + public JdbcSourceConfigFactory fromReadonlyConfig(ReadonlyConfig config) { + this.port = config.get(JdbcSourceOptions.PORT); + this.hostname = config.get(JdbcSourceOptions.HOSTNAME); + this.password = config.get(JdbcSourceOptions.PASSWORD); + // TODO: support multi-table + this.databaseList = Collections.singletonList(config.get(JdbcSourceOptions.DATABASE_NAME)); + this.tableList = Collections.singletonList(config.get(JdbcSourceOptions.TABLE_NAME)); + this.distributionFactorUpper = config.get(JdbcSourceOptions.CHUNK_KEY_EVEN_DISTRIBUTION_FACTOR_UPPER_BOUND); + this.distributionFactorLower = config.get(JdbcSourceOptions.CHUNK_KEY_EVEN_DISTRIBUTION_FACTOR_LOWER_BOUND); + this.splitSize = config.get(SourceOptions.SNAPSHOT_SPLIT_SIZE); + this.fetchSize = config.get(SourceOptions.SNAPSHOT_FETCH_SIZE); + this.serverTimeZone = config.get(JdbcSourceOptions.SERVER_TIME_ZONE); + this.connectTimeout = config.get(JdbcSourceOptions.CONNECT_TIMEOUT); + this.connectMaxRetries = config.get(JdbcSourceOptions.CONNECT_MAX_RETRIES); + this.connectionPoolSize = config.get(JdbcSourceOptions.CONNECTION_POOL_SIZE); + this.dbzProperties = new Properties(); + config.getOptional(SourceOptions.DEBEZIUM_PROPERTIES).ifPresent(map -> dbzProperties.putAll(map)); + return this; + } + @Override public abstract JdbcSourceConfig create(int subtask); } diff --git a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/JdbcSourceOptions.java b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/JdbcSourceOptions.java index 3052b40fb84..94249705182 100644 --- a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/JdbcSourceOptions.java +++ b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/JdbcSourceOptions.java @@ -103,4 +103,26 @@ public class JdbcSourceOptions extends SourceOptions { .defaultValue(3) .withDescription( "The max retry times that the connector should retry to build database server connection."); + + public static final Option CHUNK_KEY_EVEN_DISTRIBUTION_FACTOR_UPPER_BOUND = + Options.key("chunk-key.even-distribution.factor.upper-bound") + .doubleType() + .defaultValue(1000.0d) + .withDescription( + "The upper bound of chunk key distribution factor. The distribution factor is used to determine whether the" + + " table is evenly distribution or not." + + " The table chunks would use evenly calculation optimization when the data distribution is even," + + " and the query for splitting would happen when it is uneven." + + " The distribution factor could be calculated by (MAX(id) - MIN(id) + 1) / rowCount."); + + public static final Option CHUNK_KEY_EVEN_DISTRIBUTION_FACTOR_LOWER_BOUND = + Options.key("chunk-key.even-distribution.factor.lower-bound") + .doubleType() + .defaultValue(0.05d) + .withDescription( + "The lower bound of chunk key distribution factor. The distribution factor is used to determine whether the" + + " table is evenly distribution or not." + + " The table chunks would use evenly calculation optimization when the data distribution is even," + + " and the query for splitting would happen when it is uneven." + + " The distribution factor could be calculated by (MAX(id) - MIN(id) + 1) / rowCount."); } diff --git a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/SourceOptions.java b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/SourceOptions.java index 16f12079cbe..3558f11bd6f 100644 --- a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/SourceOptions.java +++ b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/base/option/SourceOptions.java @@ -21,6 +21,8 @@ import org.apache.seatunnel.api.configuration.Options; import org.apache.seatunnel.api.configuration.util.OptionRule; +import java.util.Map; + @SuppressWarnings("MagicNumber") public class SourceOptions { @@ -83,10 +85,16 @@ public class SourceOptions { .noDefaultValue() .withDescription("Optional offsets used in case of \"specific\" stop mode"); + public static final Option> DEBEZIUM_PROPERTIES = Options.key("debezium") + .mapType() + .noDefaultValue() + .withDescription("Decides if the table options contains Debezium client properties that start with prefix 'debezium'."); + public static final OptionRule.Builder BASE_RULE = OptionRule.builder() .optional(SNAPSHOT_SPLIT_SIZE, SNAPSHOT_FETCH_SIZE) .optional(INCREMENTAL_PARALLELISM) .optional(STARTUP_MODE, STOP_MODE) + .optional(DEBEZIUM_PROPERTIES) .conditional(STARTUP_MODE, StartupMode.TIMESTAMP, STARTUP_TIMESTAMP) .conditional(STARTUP_MODE, StartupMode.SPECIFIC, STARTUP_SPECIFIC_OFFSET_FILE, STARTUP_SPECIFIC_OFFSET_POS) .conditional(STOP_MODE, StopMode.TIMESTAMP, STOP_TIMESTAMP) diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/Command.java b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/DebeziumDeserializationConverter.java similarity index 69% rename from seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/Command.java rename to seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/DebeziumDeserializationConverter.java index cca16c30f70..859ef27cfde 100644 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/Command.java +++ b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/DebeziumDeserializationConverter.java @@ -15,22 +15,16 @@ * limitations under the License. */ -package org.apache.seatunnel.core.base.command; +package org.apache.seatunnel.connectors.cdc.debezium; -import org.apache.seatunnel.apis.base.command.CommandArgs; -import org.apache.seatunnel.core.base.exception.CommandException; +import org.apache.kafka.connect.data.Schema; + +import java.io.Serializable; /** - * Command interface. - * - * @param args type + * Runtime converter that converts objects of Debezium into objects of internal data structures. */ @FunctionalInterface -public interface Command { - - /** - * Execute command - */ - void execute() throws CommandException; - +public interface DebeziumDeserializationConverter extends Serializable { + Object convert(Object dbzObj, Schema schema) throws Exception; } diff --git a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/DebeziumDeserializationConverterFactory.java b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/DebeziumDeserializationConverterFactory.java new file mode 100644 index 00000000000..c1dfe19a1a0 --- /dev/null +++ b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/DebeziumDeserializationConverterFactory.java @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.cdc.debezium; + +import org.apache.seatunnel.api.table.type.SeaTunnelDataType; + +import java.io.Serializable; +import java.time.ZoneId; +import java.util.Optional; + +/** + * Factory to create {@link DebeziumDeserializationConverter} according to {@link SeaTunnelDataType}. It's + * usually used to create a user-defined {@link DebeziumDeserializationConverter} which has a higher + * resolve order than default converter. + */ +public interface DebeziumDeserializationConverterFactory extends Serializable { + + /** + * A user-defined converter factory which always fallback to default converters. + */ + DebeziumDeserializationConverterFactory DEFAULT = + (logicalType, serverTimeZone) -> Optional.empty(); + + /** + * Returns an optional {@link DebeziumDeserializationConverter}. Returns {@link Optional#empty()} + * if fallback to default converter. + * + * @param type the SeaTunnel datatype to be converted from objects of Debezium + * @param serverTimeZone TimeZone used to convert data with timestamp type + */ + Optional createUserDefinedConverter( + SeaTunnelDataType type, ZoneId serverTimeZone); +} diff --git a/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySinkState.java b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/MetadataConverter.java similarity index 74% rename from seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySinkState.java rename to seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/MetadataConverter.java index b5ad101c326..519aa4efefe 100644 --- a/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySinkState.java +++ b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/MetadataConverter.java @@ -15,9 +15,16 @@ * limitations under the License. */ -package org.apache.seatunnel.connectors.seatunnel.sentry.sink; +package org.apache.seatunnel.connectors.cdc.debezium; + +import org.apache.kafka.connect.source.SourceRecord; import java.io.Serializable; -public class SentrySinkState implements Serializable { +/** + * {@link SourceRecord} metadata info converter. + */ +@FunctionalInterface +public interface MetadataConverter extends Serializable { + Object read(SourceRecord record); } diff --git a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/row/SeaTunnelRowDebeziumDeserializationConverters.java b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/row/SeaTunnelRowDebeziumDeserializationConverters.java new file mode 100644 index 00000000000..52920672720 --- /dev/null +++ b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/row/SeaTunnelRowDebeziumDeserializationConverters.java @@ -0,0 +1,538 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.cdc.debezium.row; + +import org.apache.seatunnel.api.table.type.SeaTunnelDataType; +import org.apache.seatunnel.api.table.type.SeaTunnelRow; +import org.apache.seatunnel.api.table.type.SeaTunnelRowType; +import org.apache.seatunnel.connectors.cdc.debezium.DebeziumDeserializationConverter; +import org.apache.seatunnel.connectors.cdc.debezium.DebeziumDeserializationConverterFactory; +import org.apache.seatunnel.connectors.cdc.debezium.MetadataConverter; +import org.apache.seatunnel.connectors.cdc.debezium.utils.TemporalConversions; + +import io.debezium.data.SpecialValueDecimal; +import io.debezium.data.VariableScaleDecimal; +import io.debezium.time.MicroTime; +import io.debezium.time.MicroTimestamp; +import io.debezium.time.NanoTime; +import io.debezium.time.NanoTimestamp; +import io.debezium.time.Timestamp; +import org.apache.kafka.connect.data.Decimal; +import org.apache.kafka.connect.data.Field; +import org.apache.kafka.connect.data.Schema; +import org.apache.kafka.connect.data.Struct; +import org.apache.kafka.connect.source.SourceRecord; + +import java.io.Serializable; +import java.math.BigDecimal; +import java.nio.ByteBuffer; +import java.time.Instant; +import java.time.LocalDate; +import java.time.LocalDateTime; +import java.time.LocalTime; +import java.time.ZoneId; +import java.util.Arrays; +import java.util.Optional; + +/** + * Deserialization schema from Debezium object to {@link SeaTunnelRow} + */ +public class SeaTunnelRowDebeziumDeserializationConverters implements Serializable { + private static final long serialVersionUID = -897499476343410567L; + protected final DebeziumDeserializationConverter[] physicalConverters; + protected final MetadataConverter[] metadataConverters; + protected final String[] fieldNames; + + public SeaTunnelRowDebeziumDeserializationConverters( + SeaTunnelRowType physicalDataType, + MetadataConverter[] metadataConverters, + ZoneId serverTimeZone, + DebeziumDeserializationConverterFactory userDefinedConverterFactory) { + this.metadataConverters = metadataConverters; + + this.physicalConverters = + Arrays.stream(physicalDataType.getFieldTypes()) + .map(type -> createConverter(type, serverTimeZone, userDefinedConverterFactory)) + .toArray(DebeziumDeserializationConverter[]::new); + this.fieldNames = physicalDataType.getFieldNames(); + } + + public SeaTunnelRow convert(SourceRecord record, Struct struct, Schema schema) throws Exception { + int arity = physicalConverters.length + metadataConverters.length; + SeaTunnelRow row = new SeaTunnelRow(arity); + // physical column + for (int i = 0; i < physicalConverters.length; i++) { + String fieldName = fieldNames[i]; + Object fieldValue = struct.get(fieldName); + Field field = schema.field(fieldName); + if (field == null) { + row.setField(i, null); + } else { + Schema fieldSchema = field.schema(); + Object convertedField = SeaTunnelRowDebeziumDeserializationConverters.convertField(physicalConverters[i], fieldValue, fieldSchema); + row.setField(i, convertedField); + } + } + // metadata column + for (int i = 0; i < metadataConverters.length; i++) { + row.setField(i + physicalConverters.length, metadataConverters[i].read(record)); + } + return row; + } + + // ------------------------------------------------------------------------------------- + // Runtime Converters + // ------------------------------------------------------------------------------------- + + /** + * Creates a runtime converter which is null safe. + */ + private static DebeziumDeserializationConverter createConverter(SeaTunnelDataType type, + ZoneId serverTimeZone, + DebeziumDeserializationConverterFactory userDefinedConverterFactory) { + return wrapIntoNullableConverter(createNotNullConverter(type, serverTimeZone, userDefinedConverterFactory)); + } + + // -------------------------------------------------------------------------------- + // IMPORTANT! We use anonymous classes instead of lambdas for a reason here. It is + // necessary because the maven shade plugin cannot relocate classes in + // SerializedLambdas (MSHADE-260). + // -------------------------------------------------------------------------------- + + /** + * Creates a runtime converter which assuming input object is not null. + */ + private static DebeziumDeserializationConverter createNotNullConverter(SeaTunnelDataType type, + ZoneId serverTimeZone, + DebeziumDeserializationConverterFactory userDefinedConverterFactory) { + + // user defined converter has a higher resolve order + Optional converter = + userDefinedConverterFactory.createUserDefinedConverter(type, serverTimeZone); + if (converter.isPresent()) { + return converter.get(); + } + + // if no matched user defined converter, fallback to the default converter + switch (type.getSqlType()) { + case NULL: + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) throws Exception { + return null; + } + }; + case BOOLEAN: + return wrapNumericConverter(convertToBoolean()); + case TINYINT: + return wrapNumericConverter(convertToByte()); + case SMALLINT: + return wrapNumericConverter(convertToShort()); + case INT: + return wrapNumericConverter(convertToInt()); + case BIGINT: + return wrapNumericConverter(convertToLong()); + case DATE: + return convertToDate(); + case TIME: + return convertToTime(); + case TIMESTAMP: + return convertToTimestamp(serverTimeZone); + case FLOAT: + return wrapNumericConverter(convertToFloat()); + case DOUBLE: + return wrapNumericConverter(convertToDouble()); + case STRING: + return convertToString(); + case BYTES: + return convertToBinary(); + case DECIMAL: + return wrapNumericConverter(createDecimalConverter()); + case ROW: + return createRowConverter((SeaTunnelRowType) type, serverTimeZone, userDefinedConverterFactory); + case ARRAY: + case MAP: + default: + throw new UnsupportedOperationException("Unsupported type: " + type); + } + } + + private static DebeziumDeserializationConverter convertToBoolean() { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) { + if (dbzObj instanceof Boolean) { + return dbzObj; + } else if (dbzObj instanceof Byte) { + return (byte) dbzObj != 0; + } else if (dbzObj instanceof Short) { + return (short) dbzObj != 0; + } else if (dbzObj instanceof BigDecimal) { + return ((BigDecimal) dbzObj).shortValue() != 0; + } else { + return Boolean.parseBoolean(dbzObj.toString()); + } + } + }; + } + + private static DebeziumDeserializationConverter convertToByte() { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) { + if (dbzObj instanceof Byte) { + return dbzObj; + } else if (dbzObj instanceof BigDecimal) { + return ((BigDecimal) dbzObj).byteValue(); + } else { + return Byte.parseByte(dbzObj.toString()); + } + } + }; + } + + private static DebeziumDeserializationConverter convertToShort() { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) { + if (dbzObj instanceof Byte) { + return dbzObj; + } else if (dbzObj instanceof Short) { + return dbzObj; + } else if (dbzObj instanceof BigDecimal) { + return ((BigDecimal) dbzObj).shortValue(); + } else { + return Short.parseShort(dbzObj.toString()); + } + } + }; + } + + private static DebeziumDeserializationConverter convertToInt() { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) { + if (dbzObj instanceof Integer) { + return dbzObj; + } else if (dbzObj instanceof Long) { + return ((Long) dbzObj).intValue(); + } else if (dbzObj instanceof BigDecimal) { + return ((BigDecimal) dbzObj).intValue(); + } else { + return Integer.parseInt(dbzObj.toString()); + } + } + }; + } + + private static DebeziumDeserializationConverter convertToLong() { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) { + if (dbzObj instanceof Integer) { + return dbzObj; + } else if (dbzObj instanceof Long) { + return dbzObj; + } else if (dbzObj instanceof BigDecimal) { + return ((BigDecimal) dbzObj).longValue(); + } else { + return Long.parseLong(dbzObj.toString()); + } + } + }; + } + + private static DebeziumDeserializationConverter convertToDouble() { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) { + if (dbzObj instanceof Float) { + return dbzObj; + } else if (dbzObj instanceof Double) { + return dbzObj; + } else if (dbzObj instanceof BigDecimal) { + return ((BigDecimal) dbzObj).doubleValue(); + } else { + return Double.parseDouble(dbzObj.toString()); + } + } + }; + } + + private static DebeziumDeserializationConverter convertToFloat() { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) { + if (dbzObj instanceof Float) { + return dbzObj; + } else if (dbzObj instanceof Double) { + return ((Double) dbzObj).floatValue(); + } else if (dbzObj instanceof BigDecimal) { + return ((BigDecimal) dbzObj).floatValue(); + } else { + return Float.parseFloat(dbzObj.toString()); + } + } + }; + } + + private static DebeziumDeserializationConverter convertToDate() { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) { + return TemporalConversions.toLocalDate(dbzObj); + } + }; + } + + private static DebeziumDeserializationConverter convertToTime() { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @SuppressWarnings("MagicNumber") + @Override + public Object convert(Object dbzObj, Schema schema) { + if (dbzObj instanceof Long) { + switch (schema.name()) { + case MicroTime.SCHEMA_NAME: + return LocalTime.ofNanoOfDay((long) dbzObj * 1000L); + case NanoTime.SCHEMA_NAME: + return LocalTime.ofNanoOfDay((long) dbzObj); + default: + } + } else if (dbzObj instanceof Integer) { + return LocalTime.ofNanoOfDay((long) dbzObj * 1000_000L); + } + // get number of milliseconds of the day + return TemporalConversions.toLocalTime(dbzObj); + } + }; + } + + private static DebeziumDeserializationConverter convertToTimestamp(ZoneId serverTimeZone) { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @SuppressWarnings("MagicNumber") + @Override + public Object convert(Object dbzObj, Schema schema) { + if (dbzObj instanceof Long) { + switch (schema.name()) { + case Timestamp.SCHEMA_NAME: + return toLocalDateTime((Long) dbzObj, 0); + case MicroTimestamp.SCHEMA_NAME: + long micro = (long) dbzObj; + return toLocalDateTime(micro / 1000, (int) (micro % 1000 * 1000)); + case NanoTimestamp.SCHEMA_NAME: + long nano = (long) dbzObj; + return toLocalDateTime(nano / 1000_000, (int) (nano % 1000_000)); + default: + } + } + return TemporalConversions.toLocalDateTime(dbzObj, serverTimeZone); + } + }; + } + + @SuppressWarnings("MagicNumber") + public static LocalDateTime toLocalDateTime(long millisecond, int nanoOfMillisecond) { + // 86400000 = 24 * 60 * 60 * 1000 + int date = (int) (millisecond / 86400000); + int time = (int) (millisecond % 86400000); + if (time < 0) { + --date; + time += 86400000; + } + long nanoOfDay = time * 1_000_000L + nanoOfMillisecond; + LocalDate localDate = LocalDate.ofEpochDay(date); + LocalTime localTime = LocalTime.ofNanoOfDay(nanoOfDay); + return LocalDateTime.of(localDate, localTime); + } + + private static DebeziumDeserializationConverter convertToLocalTimeZoneTimestamp(ZoneId serverTimeZone) { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) { + if (dbzObj instanceof String) { + String str = (String) dbzObj; + // TIMESTAMP type is encoded in string type + Instant instant = Instant.parse(str); + return LocalDateTime.ofInstant(instant, serverTimeZone); + } + throw new IllegalArgumentException( + "Unable to convert to LocalDateTime from unexpected value '" + + dbzObj + + "' of type " + + dbzObj.getClass().getName()); + } + }; + } + + private static DebeziumDeserializationConverter convertToString() { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) { + return dbzObj.toString(); + } + }; + } + + private static DebeziumDeserializationConverter convertToBinary() { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) throws Exception { + if (dbzObj instanceof byte[]) { + return dbzObj; + } else if (dbzObj instanceof ByteBuffer) { + ByteBuffer byteBuffer = (ByteBuffer) dbzObj; + byte[] bytes = new byte[byteBuffer.remaining()]; + byteBuffer.get(bytes); + return bytes; + } else { + throw new UnsupportedOperationException( + "Unsupported BYTES value type: " + dbzObj.getClass().getSimpleName()); + } + } + }; + } + + private static DebeziumDeserializationConverter createDecimalConverter() { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) throws Exception { + BigDecimal bigDecimal; + if (dbzObj instanceof byte[]) { + // decimal.handling.mode=precise + bigDecimal = Decimal.toLogical(schema, (byte[]) dbzObj); + } else if (dbzObj instanceof String) { + // decimal.handling.mode=string + bigDecimal = new BigDecimal((String) dbzObj); + } else if (dbzObj instanceof Double) { + // decimal.handling.mode=double + bigDecimal = BigDecimal.valueOf((Double) dbzObj); + } else if (dbzObj instanceof BigDecimal) { + bigDecimal = (BigDecimal) dbzObj; + } else { + // fallback to string + bigDecimal = new BigDecimal(dbzObj.toString()); + } + + return bigDecimal; + } + }; + } + + private static DebeziumDeserializationConverter createRowConverter(SeaTunnelRowType rowType, + ZoneId serverTimeZone, + DebeziumDeserializationConverterFactory userDefinedConverterFactory) { + final DebeziumDeserializationConverter[] fieldConverters = + Arrays.stream(rowType.getFieldTypes()) + .map(type -> createConverter(type, serverTimeZone, userDefinedConverterFactory)) + .toArray(DebeziumDeserializationConverter[]::new); + final String[] fieldNames = rowType.getFieldNames(); + + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) throws Exception { + Struct struct = (Struct) dbzObj; + int arity = fieldNames.length; + SeaTunnelRow row = new SeaTunnelRow(arity); + for (int i = 0; i < arity; i++) { + String fieldName = fieldNames[i]; + Object fieldValue = struct.get(fieldName); + Field field = schema.field(fieldName); + if (field == null) { + row.setField(i, null); + } else { + Schema fieldSchema = field.schema(); + Object convertedField = SeaTunnelRowDebeziumDeserializationConverters.convertField(fieldConverters[i], fieldValue, fieldSchema); + row.setField(i, convertedField); + } + } + return row; + } + }; + } + + private static Object convertField( + DebeziumDeserializationConverter fieldConverter, Object fieldValue, Schema fieldSchema) + throws Exception { + if (fieldValue == null) { + return null; + } else { + return fieldConverter.convert(fieldValue, fieldSchema); + } + } + + private static DebeziumDeserializationConverter wrapIntoNullableConverter(DebeziumDeserializationConverter converter) { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) throws Exception { + if (dbzObj == null) { + return null; + } + return converter.convert(dbzObj, schema); + } + }; + } + + private static DebeziumDeserializationConverter wrapNumericConverter(DebeziumDeserializationConverter converter) { + return new DebeziumDeserializationConverter() { + private static final long serialVersionUID = 1L; + + @Override + public Object convert(Object dbzObj, Schema schema) throws Exception { + if (VariableScaleDecimal.LOGICAL_NAME.equals(schema.name())) { + SpecialValueDecimal decimal = VariableScaleDecimal.toLogical((Struct) dbzObj); + return converter.convert(decimal.getDecimalValue().orElse(BigDecimal.ZERO), schema); + } + return converter.convert(dbzObj, schema); + } + }; + } +} diff --git a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/row/SeaTunnelRowDebeziumDeserializeSchema.java b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/row/SeaTunnelRowDebeziumDeserializeSchema.java new file mode 100644 index 00000000000..8bcee31a558 --- /dev/null +++ b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/row/SeaTunnelRowDebeziumDeserializeSchema.java @@ -0,0 +1,212 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.cdc.debezium.row; + +import static com.google.common.base.Preconditions.checkNotNull; + +import org.apache.seatunnel.api.source.Collector; +import org.apache.seatunnel.api.table.type.RowKind; +import org.apache.seatunnel.api.table.type.SeaTunnelDataType; +import org.apache.seatunnel.api.table.type.SeaTunnelRow; +import org.apache.seatunnel.api.table.type.SeaTunnelRowType; +import org.apache.seatunnel.connectors.cdc.debezium.DebeziumDeserializationConverterFactory; +import org.apache.seatunnel.connectors.cdc.debezium.DebeziumDeserializationSchema; +import org.apache.seatunnel.connectors.cdc.debezium.MetadataConverter; + +import io.debezium.connector.AbstractSourceInfo; +import io.debezium.data.Envelope; +import org.apache.kafka.connect.data.Schema; +import org.apache.kafka.connect.data.Struct; +import org.apache.kafka.connect.source.SourceRecord; + +import java.io.Serializable; +import java.time.ZoneId; + +/** + * Deserialization schema from Debezium object to {@link SeaTunnelRow}. + */ +public final class SeaTunnelRowDebeziumDeserializeSchema + implements DebeziumDeserializationSchema { + private static final long serialVersionUID = 1L; + + /** + * TypeInformation of the produced {@link SeaTunnelRow}. * + */ + private final SeaTunnelDataType resultTypeInfo; + + /** + * Runtime converter that converts Kafka {@link SourceRecord}s into {@link SeaTunnelRow} consisted of + */ + private final SeaTunnelRowDebeziumDeserializationConverters converters; + + /** + * Validator to validate the row value. + */ + private final ValueValidator validator; + + /** + * Returns a builder to build {@link SeaTunnelRowDebeziumDeserializeSchema}. + */ + public static Builder builder() { + return new Builder(); + } + + SeaTunnelRowDebeziumDeserializeSchema( + SeaTunnelRowType physicalDataType, + MetadataConverter[] metadataConverters, + SeaTunnelRowType resultType, + ValueValidator validator, + ZoneId serverTimeZone, + DebeziumDeserializationConverterFactory userDefinedConverterFactory) { + this.converters = new SeaTunnelRowDebeziumDeserializationConverters( + physicalDataType, + metadataConverters, + serverTimeZone, + userDefinedConverterFactory + ); + this.resultTypeInfo = checkNotNull(resultType); + this.validator = checkNotNull(validator); + } + + @Override + public void deserialize(SourceRecord record, Collector collector) throws Exception { + Envelope.Operation operation = Envelope.operationFor(record); + Struct messageStruct = (Struct) record.value(); + Schema valueSchema = record.valueSchema(); + + Struct sourceStruct = messageStruct.getStruct(Envelope.FieldName.SOURCE); + // TODO: multi-table + String tableName = sourceStruct.getString(AbstractSourceInfo.TABLE_NAME_KEY); + + if (operation == Envelope.Operation.CREATE || operation == Envelope.Operation.READ) { + SeaTunnelRow insert = extractAfterRow(converters, record, messageStruct, valueSchema); + insert.setRowKind(RowKind.INSERT); + validator.validate(insert, RowKind.INSERT); + collector.collect(insert); + } else if (operation == Envelope.Operation.DELETE) { + SeaTunnelRow delete = extractBeforeRow(converters, record, messageStruct, valueSchema); + validator.validate(delete, RowKind.DELETE); + delete.setRowKind(RowKind.DELETE); + collector.collect(delete); + } else { + SeaTunnelRow before = extractBeforeRow(converters, record, messageStruct, valueSchema); + validator.validate(before, RowKind.UPDATE_BEFORE); + before.setRowKind(RowKind.UPDATE_BEFORE); + collector.collect(before); + + SeaTunnelRow after = extractAfterRow(converters, record, messageStruct, valueSchema); + validator.validate(after, RowKind.UPDATE_AFTER); + after.setRowKind(RowKind.UPDATE_AFTER); + collector.collect(after); + } + } + + private SeaTunnelRow extractAfterRow( + SeaTunnelRowDebeziumDeserializationConverters runtimeConverter, + SourceRecord record, + Struct value, + Schema valueSchema) throws Exception { + + Schema afterSchema = valueSchema.field(Envelope.FieldName.AFTER).schema(); + Struct after = value.getStruct(Envelope.FieldName.AFTER); + return runtimeConverter.convert(record, after, afterSchema); + } + + private SeaTunnelRow extractBeforeRow( + SeaTunnelRowDebeziumDeserializationConverters runtimeConverter, + SourceRecord record, + Struct value, + Schema valueSchema) + throws Exception { + + Schema beforeSchema = valueSchema.field(Envelope.FieldName.BEFORE).schema(); + Struct before = value.getStruct(Envelope.FieldName.BEFORE); + return runtimeConverter.convert(record, before, beforeSchema); + } + + @Override + public SeaTunnelDataType getProducedType() { + return resultTypeInfo; + } + + // ------------------------------------------------------------------------------------- + // Builder + // ------------------------------------------------------------------------------------- + + /** + * Custom validator to validate the row value. + */ + public interface ValueValidator extends Serializable { + void validate(SeaTunnelRow rowData, RowKind rowKind) throws Exception; + } + + /** + * Builder of {@link SeaTunnelRowDebeziumDeserializeSchema}. + */ + public static class Builder { + private SeaTunnelRowType physicalRowType; + private SeaTunnelRowType resultTypeInfo; + private MetadataConverter[] metadataConverters = new MetadataConverter[0]; + private ValueValidator validator = (rowData, rowKind) -> { + }; + private ZoneId serverTimeZone = ZoneId.of("UTC"); + private DebeziumDeserializationConverterFactory userDefinedConverterFactory = + DebeziumDeserializationConverterFactory.DEFAULT; + + public Builder setPhysicalRowType(SeaTunnelRowType physicalRowType) { + this.physicalRowType = physicalRowType; + return this; + } + + public Builder setMetadataConverters(MetadataConverter[] metadataConverters) { + this.metadataConverters = metadataConverters; + return this; + } + + public Builder setResultTypeInfo(SeaTunnelRowType resultTypeInfo) { + this.resultTypeInfo = resultTypeInfo; + return this; + } + + public Builder setValueValidator(ValueValidator validator) { + this.validator = validator; + return this; + } + + public Builder setServerTimeZone(ZoneId serverTimeZone) { + this.serverTimeZone = serverTimeZone; + return this; + } + + public Builder setUserDefinedConverterFactory( + DebeziumDeserializationConverterFactory userDefinedConverterFactory) { + this.userDefinedConverterFactory = userDefinedConverterFactory; + return this; + } + + public SeaTunnelRowDebeziumDeserializeSchema build() { + return new SeaTunnelRowDebeziumDeserializeSchema( + physicalRowType, + metadataConverters, + resultTypeInfo, + validator, + serverTimeZone, + userDefinedConverterFactory); + } + } +} diff --git a/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/utils/TemporalConversions.java b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/utils/TemporalConversions.java new file mode 100644 index 00000000000..a05856f271f --- /dev/null +++ b/seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/org/apache/seatunnel/connectors/cdc/debezium/utils/TemporalConversions.java @@ -0,0 +1,205 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.cdc.debezium.utils; + +import java.time.Duration; +import java.time.Instant; +import java.time.LocalDate; +import java.time.LocalDateTime; +import java.time.LocalTime; +import java.time.OffsetDateTime; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.concurrent.TimeUnit; + +/** Temporal conversion constants. */ +public final class TemporalConversions { + + static final long MILLISECONDS_PER_SECOND = TimeUnit.SECONDS.toMillis(1); + static final long MICROSECONDS_PER_SECOND = TimeUnit.SECONDS.toMicros(1); + static final long MICROSECONDS_PER_MILLISECOND = TimeUnit.MILLISECONDS.toMicros(1); + static final long NANOSECONDS_PER_MILLISECOND = TimeUnit.MILLISECONDS.toNanos(1); + static final long NANOSECONDS_PER_MICROSECOND = TimeUnit.MICROSECONDS.toNanos(1); + static final long NANOSECONDS_PER_SECOND = TimeUnit.SECONDS.toNanos(1); + static final long NANOSECONDS_PER_DAY = TimeUnit.DAYS.toNanos(1); + static final long SECONDS_PER_DAY = TimeUnit.DAYS.toSeconds(1); + static final long MICROSECONDS_PER_DAY = TimeUnit.DAYS.toMicros(1); + static final LocalDate EPOCH = LocalDate.ofEpochDay(0); + + private TemporalConversions() {} + + @SuppressWarnings("MagicNumber") + public static LocalDate toLocalDate(Object obj) { + if (obj == null) { + return null; + } + if (obj instanceof LocalDate) { + return (LocalDate) obj; + } + if (obj instanceof LocalDateTime) { + return ((LocalDateTime) obj).toLocalDate(); + } + if (obj instanceof java.sql.Date) { + return ((java.sql.Date) obj).toLocalDate(); + } + if (obj instanceof java.sql.Time) { + throw new IllegalArgumentException( + "Unable to convert to LocalDate from a java.sql.Time value '" + obj + "'"); + } + if (obj instanceof java.util.Date) { + java.util.Date date = (java.util.Date) obj; + return LocalDate.of(date.getYear() + 1900, date.getMonth() + 1, date.getDate()); + } + if (obj instanceof Long) { + // Assume the value is the epoch day number + return LocalDate.ofEpochDay((Long) obj); + } + if (obj instanceof Integer) { + // Assume the value is the epoch day number + return LocalDate.ofEpochDay((Integer) obj); + } + throw new IllegalArgumentException( + "Unable to convert to LocalDate from unexpected value '" + + obj + + "' of type " + + obj.getClass().getName()); + } + + public static LocalTime toLocalTime(Object obj) { + if (obj == null) { + return null; + } + if (obj instanceof LocalTime) { + return (LocalTime) obj; + } + if (obj instanceof LocalDateTime) { + return ((LocalDateTime) obj).toLocalTime(); + } + if (obj instanceof java.sql.Date) { + throw new IllegalArgumentException( + "Unable to convert to LocalDate from a java.sql.Date value '" + obj + "'"); + } + if (obj instanceof java.sql.Time) { + java.sql.Time time = (java.sql.Time) obj; + long millis = (int) (time.getTime() % MILLISECONDS_PER_SECOND); + int nanosOfSecond = (int) (millis * NANOSECONDS_PER_MILLISECOND); + return LocalTime.of( + time.getHours(), time.getMinutes(), time.getSeconds(), nanosOfSecond); + } + if (obj instanceof java.sql.Timestamp) { + java.sql.Timestamp timestamp = (java.sql.Timestamp) obj; + return LocalTime.of( + timestamp.getHours(), + timestamp.getMinutes(), + timestamp.getSeconds(), + timestamp.getNanos()); + } + if (obj instanceof java.util.Date) { + java.util.Date date = (java.util.Date) obj; + long millis = (int) (date.getTime() % MILLISECONDS_PER_SECOND); + int nanosOfSecond = (int) (millis * NANOSECONDS_PER_MILLISECOND); + return LocalTime.of( + date.getHours(), date.getMinutes(), date.getSeconds(), nanosOfSecond); + } + if (obj instanceof Duration) { + Long value = ((Duration) obj).toNanos(); + if (value >= 0 && value <= NANOSECONDS_PER_DAY) { + return LocalTime.ofNanoOfDay(value); + } else { + throw new IllegalArgumentException( + "Time values must use number of milliseconds greater than 0 and less than 86400000000000"); + } + } + throw new IllegalArgumentException( + "Unable to convert to LocalTime from unexpected value '" + + obj + + "' of type " + + obj.getClass().getName()); + } + + @SuppressWarnings("MagicNumber") + public static LocalDateTime toLocalDateTime(Object obj, ZoneId serverTimeZone) { + if (obj == null) { + return null; + } + if (obj instanceof OffsetDateTime) { + return ((OffsetDateTime) obj).toLocalDateTime(); + } + if (obj instanceof Instant) { + return ((Instant) obj).atOffset(ZoneOffset.UTC).toLocalDateTime(); + } + if (obj instanceof LocalDateTime) { + return (LocalDateTime) obj; + } + if (obj instanceof LocalDate) { + LocalDate date = (LocalDate) obj; + return LocalDateTime.of(date, LocalTime.MIDNIGHT); + } + if (obj instanceof LocalTime) { + LocalTime time = (LocalTime) obj; + return LocalDateTime.of(EPOCH, time); + } + if (obj instanceof java.sql.Date) { + java.sql.Date sqlDate = (java.sql.Date) obj; + LocalDate date = sqlDate.toLocalDate(); + return LocalDateTime.of(date, LocalTime.MIDNIGHT); + } + if (obj instanceof java.sql.Time) { + LocalTime localTime = toLocalTime(obj); + return LocalDateTime.of(EPOCH, localTime); + } + if (obj instanceof java.sql.Timestamp) { + java.sql.Timestamp timestamp = (java.sql.Timestamp) obj; + return LocalDateTime.of( + timestamp.getYear() + 1900, + timestamp.getMonth() + 1, + timestamp.getDate(), + timestamp.getHours(), + timestamp.getMinutes(), + timestamp.getSeconds(), + timestamp.getNanos()); + } + if (obj instanceof java.util.Date) { + java.util.Date date = (java.util.Date) obj; + long millis = (int) (date.getTime() % MILLISECONDS_PER_SECOND); + if (millis < 0) { + millis = MILLISECONDS_PER_SECOND + millis; + } + int nanosOfSecond = (int) (millis * NANOSECONDS_PER_MILLISECOND); + return LocalDateTime.of( + date.getYear() + 1900, + date.getMonth() + 1, + date.getDate(), + date.getHours(), + date.getMinutes(), + date.getSeconds(), + nanosOfSecond); + } + if (obj instanceof String) { + String str = (String) obj; + // TIMESTAMP type is encoded in string type + Instant instant = Instant.parse(str); + return LocalDateTime.ofInstant(instant, serverTimeZone); + } + throw new IllegalArgumentException( + "Unable to convert to LocalDateTime from unexpected value '" + + obj + + "' of type " + + obj.getClass().getName()); + } +} diff --git a/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/pom.xml b/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/pom.xml index e5ec3a4b4a6..6f83f810779 100644 --- a/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/pom.xml +++ b/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/pom.xml @@ -39,6 +39,11 @@ io.debezium debezium-connector-mysql + + org.apache.seatunnel + connector-jdbc + ${project.version} + diff --git a/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/MySqlSourceOptions.java b/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/MySqlSourceOptions.java deleted file mode 100644 index 7801cbe79b7..00000000000 --- a/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/config/MySqlSourceOptions.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.connectors.seatunnel.cdc.mysql.config; - -import org.apache.seatunnel.api.configuration.Option; -import org.apache.seatunnel.api.configuration.Options; - -public class MySqlSourceOptions { - public static final Option SERVER_ID = - Options.key("server-id") - .stringType() - .noDefaultValue() - .withDescription("A numeric ID or a numeric ID range of this database client, " - + "The numeric ID syntax is like '5400', the numeric ID range syntax " - + "is like '5400-5408', The numeric ID range syntax is recommended when " - + "'scan.incremental.snapshot.enabled' enabled. Every ID must be unique across all " - + "currently-running database processes in the MySQL cluster. This connector" - + " joins the MySQL cluster as another server (with this unique ID) " - + "so it can read the binlog. By default, a random number is generated between" - + " 5400 and 6400, though we recommend setting an explicit value."); -} diff --git a/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/MySqlIncrementalSource.java b/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/MySqlIncrementalSource.java index 7f01380fc3d..be3cfbc4f6b 100644 --- a/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/MySqlIncrementalSource.java +++ b/seatunnel-connectors-v2/connector-cdc/connector-cdc-mysql/src/main/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/MySqlIncrementalSource.java @@ -19,19 +19,27 @@ import org.apache.seatunnel.api.configuration.ReadonlyConfig; import org.apache.seatunnel.api.source.SeaTunnelSource; +import org.apache.seatunnel.api.table.catalog.CatalogTable; +import org.apache.seatunnel.api.table.catalog.TablePath; +import org.apache.seatunnel.api.table.type.SeaTunnelRowType; import org.apache.seatunnel.connectors.cdc.base.config.JdbcSourceConfig; import org.apache.seatunnel.connectors.cdc.base.config.SourceConfig; import org.apache.seatunnel.connectors.cdc.base.dialect.DataSourceDialect; import org.apache.seatunnel.connectors.cdc.base.dialect.JdbcDataSourceDialect; +import org.apache.seatunnel.connectors.cdc.base.option.JdbcSourceOptions; import org.apache.seatunnel.connectors.cdc.base.source.IncrementalSource; import org.apache.seatunnel.connectors.cdc.base.source.offset.OffsetFactory; import org.apache.seatunnel.connectors.cdc.debezium.DebeziumDeserializationSchema; +import org.apache.seatunnel.connectors.cdc.debezium.row.SeaTunnelRowDebeziumDeserializeSchema; import org.apache.seatunnel.connectors.seatunnel.cdc.mysql.config.MySqlSourceConfigFactory; -import org.apache.seatunnel.connectors.seatunnel.cdc.mysql.config.MySqlSourceOptions; import org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.offset.BinlogOffsetFactory; +import org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.JdbcCatalogOptions; +import org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.MySqlCatalog; import com.google.auto.service.AutoService; +import java.time.ZoneId; + @AutoService(SeaTunnelSource.class) public class MySqlIncrementalSource extends IncrementalSource { @Override @@ -42,14 +50,27 @@ public String getPluginName() { @Override public SourceConfig.Factory createSourceConfigFactory(ReadonlyConfig config) { MySqlSourceConfigFactory configFactory = new MySqlSourceConfigFactory(); - configFactory.serverId(config.get(MySqlSourceOptions.SERVER_ID)); + configFactory.serverId(config.get(JdbcSourceOptions.SERVER_ID)); + configFactory.fromReadonlyConfig(readonlyConfig); return configFactory; } + @SuppressWarnings("unchecked") @Override public DebeziumDeserializationSchema createDebeziumDeserializationSchema(ReadonlyConfig config) { - // TODO: seatunnel row - return null; + JdbcSourceConfig jdbcSourceConfig = configFactory.create(0); + String baseUrl = config.get(JdbcCatalogOptions.BASE_URL); + // TODO: support multi-table + // TODO: support metadata keys + MySqlCatalog mySqlCatalog = new MySqlCatalog("mysql", jdbcSourceConfig.getDatabaseList().get(0), jdbcSourceConfig.getUsername(), jdbcSourceConfig.getPassword(), baseUrl); + CatalogTable table = mySqlCatalog.getTable(TablePath.of(jdbcSourceConfig.getDatabaseList().get(0), jdbcSourceConfig.getTableList().get(0))); + SeaTunnelRowType physicalRowType = table.getTableSchema().toPhysicalRowDataType(); + String zoneId = config.get(JdbcSourceOptions.SERVER_TIME_ZONE); + return (DebeziumDeserializationSchema) SeaTunnelRowDebeziumDeserializeSchema.builder() + .setPhysicalRowType(physicalRowType) + .setResultTypeInfo(physicalRowType) + .setServerTimeZone(ZoneId.of(zoneId)) + .build(); } @Override diff --git a/seatunnel-connectors-v2/connector-file/connector-file-base/pom.xml b/seatunnel-connectors-v2/connector-file/connector-file-base/pom.xml index f9209305665..1b7afcb4f96 100644 --- a/seatunnel-connectors-v2/connector-file/connector-file-base/pom.xml +++ b/seatunnel-connectors-v2/connector-file/connector-file-base/pom.xml @@ -69,23 +69,6 @@ ${project.version} - - org.apache.seatunnel - seatunnel-core-base - ${project.version} - - - org.apache.orc - orc-core - - - avro - org.apache.avro - - - test - - org.apache.seatunnel connector-common diff --git a/seatunnel-core/seatunnel-core-base/pom.xml b/seatunnel-connectors-v2/connector-http/connector-http-jira/pom.xml similarity index 60% rename from seatunnel-core/seatunnel-core-base/pom.xml rename to seatunnel-connectors-v2/connector-http/connector-http-jira/pom.xml index 7852932d0a1..34b7830ead6 100644 --- a/seatunnel-core/seatunnel-core-base/pom.xml +++ b/seatunnel-connectors-v2/connector-http/connector-http-jira/pom.xml @@ -21,44 +21,20 @@ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> + connector-http org.apache.seatunnel - seatunnel-core ${revision} - ../pom.xml 4.0.0 - seatunnel-core-base + connector-http-jira - - - org.apache.seatunnel - seatunnel-api-base - ${project.version} - - - - org.apache.seatunnel - seatunnel-api-flink - ${project.version} - - - - org.apache.seatunnel - seatunnel-api-spark - ${project.version} - - org.apache.seatunnel - seatunnel-plugin-discovery + connector-http-base ${project.version} - - - com.beust - jcommander - + diff --git a/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSource.java b/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSource.java new file mode 100644 index 00000000000..6d112de4c75 --- /dev/null +++ b/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSource.java @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.jira.source; + +import static org.apache.seatunnel.connectors.seatunnel.http.util.AuthorizationUtil.getTokenByBasicAuth; + +import org.apache.seatunnel.api.common.PrepareFailException; +import org.apache.seatunnel.api.source.Boundedness; +import org.apache.seatunnel.api.source.SeaTunnelSource; +import org.apache.seatunnel.api.table.type.SeaTunnelRow; +import org.apache.seatunnel.common.config.CheckConfigUtil; +import org.apache.seatunnel.common.config.CheckResult; +import org.apache.seatunnel.common.constants.JobMode; +import org.apache.seatunnel.common.constants.PluginType; +import org.apache.seatunnel.connectors.seatunnel.common.source.AbstractSingleSplitReader; +import org.apache.seatunnel.connectors.seatunnel.common.source.SingleSplitReaderContext; +import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSource; +import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSourceReader; +import org.apache.seatunnel.connectors.seatunnel.jira.source.config.JiraSourceConfig; +import org.apache.seatunnel.connectors.seatunnel.jira.source.config.JiraSourceParameter; + +import org.apache.seatunnel.shade.com.typesafe.config.Config; + +import com.google.auto.service.AutoService; +import lombok.extern.slf4j.Slf4j; + +@Slf4j +@AutoService(SeaTunnelSource.class) +public class JiraSource extends HttpSource { + private final JiraSourceParameter jiraSourceParameter = new JiraSourceParameter(); + + @Override + public String getPluginName() { + return "Jira"; + } + + @Override + public Boundedness getBoundedness() { + if (JobMode.BATCH.equals(jobContext.getJobMode())) { + return Boundedness.BOUNDED; + } + throw new UnsupportedOperationException("Jira source connector not support unbounded operation"); + } + + @Override + public void prepare(Config pluginConfig) throws PrepareFailException { + CheckResult result = CheckConfigUtil.checkAllExists(pluginConfig, JiraSourceConfig.URL.key(), JiraSourceConfig.EMAIL.key(), JiraSourceConfig.API_TOKEN.key()); + if (!result.isSuccess()) { + throw new PrepareFailException(getPluginName(), PluginType.SOURCE, result.getMsg()); + } + //get accessToken by basic auth + String accessToken = getTokenByBasicAuth(pluginConfig.getString(JiraSourceConfig.EMAIL.key()), pluginConfig.getString(JiraSourceConfig.API_TOKEN.key())); + jiraSourceParameter.buildWithConfig(pluginConfig, accessToken); + buildSchemaWithConfig(pluginConfig); + } + + @Override + public AbstractSingleSplitReader createReader(SingleSplitReaderContext readerContext) throws Exception { + return new HttpSourceReader(this.jiraSourceParameter, readerContext, this.deserializationSchema); + } +} diff --git a/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSourceFactory.java b/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSourceFactory.java new file mode 100644 index 00000000000..c37f949b5d3 --- /dev/null +++ b/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSourceFactory.java @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.jira.source; + +import org.apache.seatunnel.api.configuration.util.Condition; +import org.apache.seatunnel.api.configuration.util.OptionRule; +import org.apache.seatunnel.api.table.factory.Factory; +import org.apache.seatunnel.api.table.factory.TableSourceFactory; +import org.apache.seatunnel.connectors.seatunnel.common.schema.SeaTunnelSchema; +import org.apache.seatunnel.connectors.seatunnel.http.config.HttpConfig; +import org.apache.seatunnel.connectors.seatunnel.http.config.HttpRequestMethod; +import org.apache.seatunnel.connectors.seatunnel.jira.source.config.JiraSourceConfig; + +import com.google.auto.service.AutoService; + +@AutoService(Factory.class) +public class JiraSourceFactory implements TableSourceFactory { + @Override + public String factoryIdentifier() { + return "Jira"; + } + + @Override + public OptionRule optionRule() { + return OptionRule.builder() + .required(JiraSourceConfig.URL) + .required(JiraSourceConfig.EMAIL) + .required(JiraSourceConfig.API_TOKEN) + .optional(JiraSourceConfig.METHOD) + .optional(JiraSourceConfig.HEADERS) + .optional(JiraSourceConfig.PARAMS) + .conditional(Condition.of(HttpConfig.METHOD, HttpRequestMethod.POST), JiraSourceConfig.BODY) + .conditional(Condition.of(HttpConfig.FORMAT, "json"), SeaTunnelSchema.SCHEMA) + .optional(JiraSourceConfig.FORMAT) + .optional(JiraSourceConfig.POLL_INTERVAL_MILLS) + .optional(JiraSourceConfig.RETRY) + .optional(JiraSourceConfig.RETRY_BACKOFF_MAX_MS) + .optional(JiraSourceConfig.RETRY_BACKOFF_MULTIPLIER_MS) + .build(); + } +} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/DeployModeConverter.java b/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/config/JiraSourceConfig.java similarity index 53% rename from seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/DeployModeConverter.java rename to seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/config/JiraSourceConfig.java index 8c9f629e301..9eda2e6e458 100644 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/DeployModeConverter.java +++ b/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/config/JiraSourceConfig.java @@ -15,20 +15,21 @@ * limitations under the License. */ -package org.apache.seatunnel.core.base.command; +package org.apache.seatunnel.connectors.seatunnel.jira.source.config; -import org.apache.seatunnel.common.config.DeployMode; +import org.apache.seatunnel.api.configuration.Option; +import org.apache.seatunnel.api.configuration.Options; +import org.apache.seatunnel.connectors.seatunnel.http.config.HttpConfig; -import com.beust.jcommander.IStringConverter; -import com.beust.jcommander.ParameterException; +public class JiraSourceConfig extends HttpConfig { + public static final String AUTHORIZATION = "Authorization"; + public static final Option EMAIL = Options.key("email") + .stringType() + .noDefaultValue() + .withDescription("Jira email"); -import java.util.Optional; - -public class DeployModeConverter implements IStringConverter { - - @Override - public DeployMode convert(String value) { - Optional deployMode = DeployMode.from(value); - return deployMode.orElseThrow(() -> new ParameterException("deploy-mode: " + value + " is not allowed.")); - } + public static final Option API_TOKEN = Options.key("api_token") + .stringType() + .noDefaultValue() + .withDescription("Jira API Token"); } diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/ConfigChecker.java b/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/config/JiraSourceParameter.java similarity index 59% rename from seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/ConfigChecker.java rename to seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/config/JiraSourceParameter.java index 30e4243e7ca..3e1ae6dbd57 100644 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/ConfigChecker.java +++ b/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/config/JiraSourceParameter.java @@ -15,25 +15,20 @@ * limitations under the License. */ -package org.apache.seatunnel.core.base.config; +package org.apache.seatunnel.connectors.seatunnel.jira.source.config; -import org.apache.seatunnel.apis.base.env.RuntimeEnv; -import org.apache.seatunnel.core.base.exception.ConfigCheckException; +import org.apache.seatunnel.connectors.seatunnel.http.config.HttpParameter; import org.apache.seatunnel.shade.com.typesafe.config.Config; -/** - * Check the config is valid. - * - * @param the environment type. - */ -public interface ConfigChecker { - - /** - * Check if the config is validated, if check fails, throw exception. - * - * @param config given config. - */ - void checkConfig(Config config) throws ConfigCheckException; +import java.util.HashMap; +public class JiraSourceParameter extends HttpParameter { + public void buildWithConfig(Config pluginConfig, String accessToken) { + super.buildWithConfig(pluginConfig); + // put authorization in headers + this.headers = this.getHeaders() == null ? new HashMap<>() : this.getHeaders(); + this.headers.put(JiraSourceConfig.AUTHORIZATION, accessToken); + this.setHeaders(this.headers); + } } diff --git a/seatunnel-connectors-v2/connector-http/connector-http-lemlist/src/main/java/org/apache/seatunnel/connectors/seatunnel/lemlist/source/LemlistSource.java b/seatunnel-connectors-v2/connector-http/connector-http-lemlist/src/main/java/org/apache/seatunnel/connectors/seatunnel/lemlist/source/LemlistSource.java index fad010e1e58..53b27f87062 100644 --- a/seatunnel-connectors-v2/connector-http/connector-http-lemlist/src/main/java/org/apache/seatunnel/connectors/seatunnel/lemlist/source/LemlistSource.java +++ b/seatunnel-connectors-v2/connector-http/connector-http-lemlist/src/main/java/org/apache/seatunnel/connectors/seatunnel/lemlist/source/LemlistSource.java @@ -20,6 +20,7 @@ import static org.apache.seatunnel.connectors.seatunnel.http.util.AuthorizationUtil.getTokenByBasicAuth; import org.apache.seatunnel.api.common.PrepareFailException; +import org.apache.seatunnel.api.common.SeaTunnelAPIErrorCode; import org.apache.seatunnel.api.source.SeaTunnelSource; import org.apache.seatunnel.api.table.type.SeaTunnelRow; import org.apache.seatunnel.common.config.CheckConfigUtil; @@ -31,6 +32,7 @@ import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSourceReader; import org.apache.seatunnel.connectors.seatunnel.lemlist.source.config.LemlistSourceConfig; import org.apache.seatunnel.connectors.seatunnel.lemlist.source.config.LemlistSourceParameter; +import org.apache.seatunnel.connectors.seatunnel.lemlist.source.exception.LemlistConnectorException; import org.apache.seatunnel.shade.com.typesafe.config.Config; @@ -50,7 +52,9 @@ public String getPluginName() { public void prepare(Config pluginConfig) throws PrepareFailException { CheckResult result = CheckConfigUtil.checkAllExists(pluginConfig, LemlistSourceConfig.URL.key(), LemlistSourceConfig.PASSWORD.key()); if (!result.isSuccess()) { - throw new PrepareFailException(getPluginName(), PluginType.SOURCE, result.getMsg()); + throw new LemlistConnectorException(SeaTunnelAPIErrorCode.CONFIG_VALIDATION_FAILED, + String.format("PluginName: %s, PluginType: %s, Message: %s", + getPluginName(), PluginType.SOURCE, result.getMsg())); } //get accessToken by basic auth String accessToken = getTokenByBasicAuth("", pluginConfig.getString(LemlistSourceConfig.PASSWORD.key())); diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/exception/ConfigCheckException.java b/seatunnel-connectors-v2/connector-http/connector-http-lemlist/src/main/java/org/apache/seatunnel/connectors/seatunnel/lemlist/source/exception/LemlistConnectorException.java similarity index 52% rename from seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/exception/ConfigCheckException.java rename to seatunnel-connectors-v2/connector-http/connector-http-lemlist/src/main/java/org/apache/seatunnel/connectors/seatunnel/lemlist/source/exception/LemlistConnectorException.java index 22d193bf5cc..6dfaebb4641 100644 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/exception/ConfigCheckException.java +++ b/seatunnel-connectors-v2/connector-http/connector-http-lemlist/src/main/java/org/apache/seatunnel/connectors/seatunnel/lemlist/source/exception/LemlistConnectorException.java @@ -15,16 +15,21 @@ * limitations under the License. */ -package org.apache.seatunnel.core.base.exception; +package org.apache.seatunnel.connectors.seatunnel.lemlist.source.exception; -public class ConfigCheckException extends CommandException { +import org.apache.seatunnel.common.exception.SeaTunnelErrorCode; +import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException; - public ConfigCheckException(String message) { - super(message); +public class LemlistConnectorException extends SeaTunnelRuntimeException { + public LemlistConnectorException(SeaTunnelErrorCode seaTunnelErrorCode, String errorMessage) { + super(seaTunnelErrorCode, errorMessage); } - public ConfigCheckException(String message, Throwable cause) { - super(message, cause); + public LemlistConnectorException(SeaTunnelErrorCode seaTunnelErrorCode, String errorMessage, Throwable cause) { + super(seaTunnelErrorCode, errorMessage, cause); } + public LemlistConnectorException(SeaTunnelErrorCode seaTunnelErrorCode, Throwable cause) { + super(seaTunnelErrorCode, cause); + } } diff --git a/seatunnel-connectors-v2/connector-http/pom.xml b/seatunnel-connectors-v2/connector-http/pom.xml index e0de4766c08..ce24c0fc920 100644 --- a/seatunnel-connectors-v2/connector-http/pom.xml +++ b/seatunnel-connectors-v2/connector-http/pom.xml @@ -37,6 +37,7 @@ connector-http-lemlist connector-http-klaviyo connector-http-onesignal + connector-http-jira connector-http-gitlab diff --git a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/state/SocketState.java b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/JdbcCatalogOptions.java similarity index 60% rename from seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/state/SocketState.java rename to seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/JdbcCatalogOptions.java index f5d58219234..93ea1e60981 100644 --- a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/state/SocketState.java +++ b/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/JdbcCatalogOptions.java @@ -15,9 +15,15 @@ * limitations under the License. */ -package org.apache.seatunnel.connectors.seatunnel.socket.state; +package org.apache.seatunnel.connectors.seatunnel.jdbc.catalog; -import java.io.Serializable; +import org.apache.seatunnel.api.configuration.Option; +import org.apache.seatunnel.api.configuration.Options; -public class SocketState implements Serializable { +public class JdbcCatalogOptions { + public static final Option BASE_URL = Options.key("base-url") + .stringType() + .noDefaultValue() + .withDescription("URL has to be without database, like \"jdbc:mysql://localhost:5432/\" or" + + "\"jdbc:mysql://localhost:5432\" rather than \"jdbc:mysql://localhost:5432/db\""); } diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/data/DataTypeValidator.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/data/DataTypeValidator.java index 414c9a53451..9e3b9ad68f1 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/data/DataTypeValidator.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/data/DataTypeValidator.java @@ -21,17 +21,21 @@ import org.apache.seatunnel.api.table.type.SeaTunnelDataType; import org.apache.seatunnel.api.table.type.SeaTunnelRowType; import org.apache.seatunnel.api.table.type.SqlType; +import org.apache.seatunnel.common.exception.CommonErrorCode; +import org.apache.seatunnel.connectors.seatunnel.mongodb.exception.MongodbConnectorException; public class DataTypeValidator { public static void validateDataType(SeaTunnelDataType dataType) throws IllegalArgumentException { switch (dataType.getSqlType()) { case TIME: - throw new IllegalArgumentException("Unsupported data type: " + dataType); + throw new MongodbConnectorException(CommonErrorCode.UNSUPPORTED_DATA_TYPE, + "Unsupported data type: " + dataType); case MAP: MapType mapType = (MapType) dataType; if (!SqlType.STRING.equals(mapType.getKeyType().getSqlType())) { - throw new IllegalArgumentException("Unsupported map key type: " + mapType.getKeyType()); + throw new MongodbConnectorException(CommonErrorCode.UNSUPPORTED_DATA_TYPE, + "Unsupported map key type: " + mapType.getKeyType()); } break; case ROW: diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/data/DefaultSerializer.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/data/DefaultSerializer.java index 5c4ae9565d7..1c193a5c24c 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/data/DefaultSerializer.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/data/DefaultSerializer.java @@ -47,6 +47,7 @@ public DefaultSerializer(@NonNull SeaTunnelRowType rowType) { this.rowType = rowType; } + @Override public Document serialize(@NonNull SeaTunnelRow row) { return convert(rowType, row); } diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/data/SeaTunnelRowBsonWriter.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/data/SeaTunnelRowBsonWriter.java index b70a9a5b4e3..8a9c6866a91 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/data/SeaTunnelRowBsonWriter.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/data/SeaTunnelRowBsonWriter.java @@ -22,6 +22,8 @@ import org.apache.seatunnel.api.table.type.SeaTunnelDataType; import org.apache.seatunnel.api.table.type.SeaTunnelRow; import org.apache.seatunnel.api.table.type.SeaTunnelRowType; +import org.apache.seatunnel.common.exception.CommonErrorCode; +import org.apache.seatunnel.connectors.seatunnel.mongodb.exception.MongodbConnectorException; import lombok.Getter; import lombok.SneakyThrows; @@ -225,27 +227,32 @@ protected void doWriteTimestamp(BsonTimestamp value) { @Override protected void doWriteJavaScriptWithScope(String value) { - throw new UnsupportedOperationException("Unsupported JavaScriptWithScope"); + throw new MongodbConnectorException(CommonErrorCode.UNSUPPORTED_OPERATION, + "Unsupported JavaScriptWithScope"); } @Override protected void doWriteMaxKey() { - throw new UnsupportedOperationException("Unsupported MaxKey"); + throw new MongodbConnectorException(CommonErrorCode.UNSUPPORTED_OPERATION, + "Unsupported MaxKey"); } @Override protected void doWriteMinKey() { - throw new UnsupportedOperationException("Unsupported MinKey"); + throw new MongodbConnectorException(CommonErrorCode.UNSUPPORTED_OPERATION, + "Unsupported MinKey"); } @Override protected void doWriteRegularExpression(BsonRegularExpression value) { - throw new UnsupportedOperationException("Unsupported BsonRegularExpression"); + throw new MongodbConnectorException(CommonErrorCode.UNSUPPORTED_OPERATION, + "Unsupported BsonRegularExpression"); } @Override protected void doWriteDBPointer(BsonDbPointer value) { - throw new UnsupportedOperationException("Unsupported BsonDbPointer"); + throw new MongodbConnectorException(CommonErrorCode.UNSUPPORTED_OPERATION, + "Unsupported BsonDbPointer"); } @Override diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/exception/MongodbConnectorException.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/exception/MongodbConnectorException.java new file mode 100644 index 00000000000..b0a20e5af99 --- /dev/null +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/exception/MongodbConnectorException.java @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.mongodb.exception; + +import org.apache.seatunnel.common.exception.SeaTunnelErrorCode; +import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException; + +public class MongodbConnectorException extends SeaTunnelRuntimeException { + + public MongodbConnectorException(SeaTunnelErrorCode seaTunnelErrorCode, String errorMessage) { + super(seaTunnelErrorCode, errorMessage); + } + + public MongodbConnectorException(SeaTunnelErrorCode seaTunnelErrorCode, String errorMessage, Throwable cause) { + super(seaTunnelErrorCode, errorMessage, cause); + } + + public MongodbConnectorException(SeaTunnelErrorCode seaTunnelErrorCode, Throwable cause) { + super(seaTunnelErrorCode, cause); + } +} diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSink.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSink.java index 13f136aaaba..f8e69f6dc52 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSink.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/sink/MongodbSink.java @@ -22,6 +22,7 @@ import static org.apache.seatunnel.connectors.seatunnel.mongodb.config.MongodbConfig.URI; import org.apache.seatunnel.api.common.PrepareFailException; +import org.apache.seatunnel.api.common.SeaTunnelAPIErrorCode; import org.apache.seatunnel.api.sink.SeaTunnelSink; import org.apache.seatunnel.api.sink.SinkWriter; import org.apache.seatunnel.api.table.type.SeaTunnelDataType; @@ -34,6 +35,7 @@ import org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSimpleSink; import org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSinkWriter; import org.apache.seatunnel.connectors.seatunnel.mongodb.config.MongodbParameters; +import org.apache.seatunnel.connectors.seatunnel.mongodb.exception.MongodbConnectorException; import org.apache.seatunnel.shade.com.typesafe.config.Config; import org.apache.seatunnel.shade.com.typesafe.config.ConfigBeanFactory; @@ -58,7 +60,9 @@ public String getPluginName() { public void prepare(Config config) throws PrepareFailException { CheckResult result = CheckConfigUtil.checkAllExists(config, URI.key(), DATABASE.key(), COLLECTION.key()); if (!result.isSuccess()) { - throw new PrepareFailException(getPluginName(), PluginType.SOURCE, result.getMsg()); + throw new MongodbConnectorException(SeaTunnelAPIErrorCode.CONFIG_VALIDATION_FAILED, + String.format("PluginName: %s, PluginType: %s, Message: %s", + getPluginName(), PluginType.SINK, result.getMsg())); } this.params = ConfigBeanFactory.create(config, MongodbParameters.class); diff --git a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/source/MongodbSource.java b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/source/MongodbSource.java index 6b24228301d..545069b87e5 100644 --- a/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/source/MongodbSource.java +++ b/seatunnel-connectors-v2/connector-mongodb/src/main/java/org/apache/seatunnel/connectors/seatunnel/mongodb/source/MongodbSource.java @@ -22,6 +22,7 @@ import static org.apache.seatunnel.connectors.seatunnel.mongodb.config.MongodbConfig.URI; import org.apache.seatunnel.api.common.PrepareFailException; +import org.apache.seatunnel.api.common.SeaTunnelAPIErrorCode; import org.apache.seatunnel.api.source.Boundedness; import org.apache.seatunnel.api.source.SeaTunnelSource; import org.apache.seatunnel.api.table.type.SeaTunnelDataType; @@ -35,6 +36,7 @@ import org.apache.seatunnel.connectors.seatunnel.common.source.AbstractSingleSplitSource; import org.apache.seatunnel.connectors.seatunnel.common.source.SingleSplitReaderContext; import org.apache.seatunnel.connectors.seatunnel.mongodb.config.MongodbParameters; +import org.apache.seatunnel.connectors.seatunnel.mongodb.exception.MongodbConnectorException; import org.apache.seatunnel.shade.com.typesafe.config.Config; import org.apache.seatunnel.shade.com.typesafe.config.ConfigBeanFactory; @@ -57,7 +59,9 @@ public String getPluginName() { public void prepare(Config config) throws PrepareFailException { CheckResult result = CheckConfigUtil.checkAllExists(config, URI.key(), DATABASE.key(), COLLECTION.key()); if (!result.isSuccess()) { - throw new PrepareFailException(getPluginName(), PluginType.SOURCE, result.getMsg()); + throw new MongodbConnectorException(SeaTunnelAPIErrorCode.CONFIG_VALIDATION_FAILED, + String.format("PluginName: %s, PluginType: %s, Message: %s", + getPluginName(), PluginType.SOURCE, result.getMsg())); } this.params = ConfigBeanFactory.create(config, MongodbParameters.class); diff --git a/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentryConfig.java b/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/config/SentryConfig.java similarity index 97% rename from seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentryConfig.java rename to seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/config/SentryConfig.java index ab79b8d9436..2297342b562 100644 --- a/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentryConfig.java +++ b/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/config/SentryConfig.java @@ -15,7 +15,7 @@ * limitations under the License. */ -package org.apache.seatunnel.connectors.seatunnel.sentry.sink; +package org.apache.seatunnel.connectors.seatunnel.sentry.config; import org.apache.seatunnel.api.configuration.Option; import org.apache.seatunnel.api.configuration.Options; diff --git a/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/exception/SentryConnectorException.java b/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/exception/SentryConnectorException.java new file mode 100644 index 00000000000..27078c2408d --- /dev/null +++ b/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/exception/SentryConnectorException.java @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.sentry.exception; + +import org.apache.seatunnel.common.exception.SeaTunnelErrorCode; +import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException; + +public class SentryConnectorException extends SeaTunnelRuntimeException { + + public SentryConnectorException(SeaTunnelErrorCode seaTunnelErrorCode, String errorMessage) { + super(seaTunnelErrorCode, errorMessage); + } + + public SentryConnectorException(SeaTunnelErrorCode seaTunnelErrorCode, String errorMessage, Throwable cause) { + super(seaTunnelErrorCode, errorMessage, cause); + } + + public SentryConnectorException(SeaTunnelErrorCode seaTunnelErrorCode, Throwable cause) { + super(seaTunnelErrorCode, cause); + } +} diff --git a/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySink.java b/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySink.java index 8c8eb542237..230aae43acd 100644 --- a/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySink.java +++ b/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySink.java @@ -18,14 +18,17 @@ package org.apache.seatunnel.connectors.seatunnel.sentry.sink; import org.apache.seatunnel.api.common.PrepareFailException; +import org.apache.seatunnel.api.common.SeaTunnelAPIErrorCode; import org.apache.seatunnel.api.sink.SeaTunnelSink; -import org.apache.seatunnel.api.sink.SinkWriter.Context; +import org.apache.seatunnel.api.sink.SinkWriter; import org.apache.seatunnel.api.table.type.SeaTunnelDataType; import org.apache.seatunnel.api.table.type.SeaTunnelRow; import org.apache.seatunnel.api.table.type.SeaTunnelRowType; import org.apache.seatunnel.common.constants.PluginType; import org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSimpleSink; import org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSinkWriter; +import org.apache.seatunnel.connectors.seatunnel.sentry.config.SentryConfig; +import org.apache.seatunnel.connectors.seatunnel.sentry.exception.SentryConnectorException; import org.apache.seatunnel.shade.com.typesafe.config.Config; @@ -37,10 +40,11 @@ * @description: SentrySink class */ @AutoService(SeaTunnelSink.class) -public class SentrySink extends AbstractSimpleSink { +public class SentrySink extends AbstractSimpleSink { private SeaTunnelRowType seaTunnelRowType; private Config pluginConfig; + @Override public String getPluginName() { return SentryConfig.SENTRY; @@ -49,8 +53,13 @@ public String getPluginName() { @Override public void prepare(Config pluginConfig) throws PrepareFailException { if (!pluginConfig.hasPath(SentryConfig.DSN.key())) { - throw new PrepareFailException(getPluginName(), PluginType.SINK, - String.format("Config must include column : %s", SentryConfig.DSN)); + throw new SentryConnectorException(SeaTunnelAPIErrorCode.CONFIG_VALIDATION_FAILED, + String.format("PluginName: %s, PluginType: %s, Message: %s", + getPluginName(), PluginType.SINK, + String.format("Config must include column : %s", + SentryConfig.DSN) + ) + ); } this.pluginConfig = pluginConfig; @@ -67,7 +76,7 @@ public SeaTunnelDataType getConsumedType() { } @Override - public AbstractSinkWriter createWriter(Context context) throws IOException { - return new SentrySinkWriter(seaTunnelRowType, context, pluginConfig); + public AbstractSinkWriter createWriter(SinkWriter.Context context) throws IOException { + return new SentrySinkWriter(seaTunnelRowType, pluginConfig); } } diff --git a/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySinkFactory.java b/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySinkFactory.java index 7fe6275f9f1..a3d6a9772f5 100644 --- a/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySinkFactory.java +++ b/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySinkFactory.java @@ -20,6 +20,7 @@ import org.apache.seatunnel.api.configuration.util.OptionRule; import org.apache.seatunnel.api.table.factory.Factory; import org.apache.seatunnel.api.table.factory.TableSinkFactory; +import org.apache.seatunnel.connectors.seatunnel.sentry.config.SentryConfig; import com.google.auto.service.AutoService; diff --git a/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySinkWriter.java b/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySinkWriter.java index 9580dee2b83..d29c05c7bc6 100644 --- a/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySinkWriter.java +++ b/seatunnel-connectors-v2/connector-sentry/src/main/java/org/apache/seatunnel/connectors/seatunnel/sentry/sink/SentrySinkWriter.java @@ -17,10 +17,10 @@ package org.apache.seatunnel.connectors.seatunnel.sentry.sink; -import org.apache.seatunnel.api.sink.SinkWriter; import org.apache.seatunnel.api.table.type.SeaTunnelRow; import org.apache.seatunnel.api.table.type.SeaTunnelRowType; import org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSinkWriter; +import org.apache.seatunnel.connectors.seatunnel.sentry.config.SentryConfig; import org.apache.seatunnel.shade.com.typesafe.config.Config; @@ -33,32 +33,32 @@ * @description: SentrySinkWriter class */ -public class SentrySinkWriter extends AbstractSinkWriter { +public class SentrySinkWriter extends AbstractSinkWriter { private SeaTunnelRowType seaTunnelRowType; + public SentrySinkWriter(SeaTunnelRowType seaTunnelRowType, - SinkWriter.Context context, Config pluginConfig) { SentryOptions options = new SentryOptions(); options.setDsn(pluginConfig.getString(SentryConfig.DSN.key())); - if (pluginConfig.hasPath(SentryConfig.ENV.key())){ + if (pluginConfig.hasPath(SentryConfig.ENV.key())) { options.setEnvironment(pluginConfig.getString(SentryConfig.ENV.key())); } - if (pluginConfig.hasPath(SentryConfig.RELEASE.key())){ + if (pluginConfig.hasPath(SentryConfig.RELEASE.key())) { options.setRelease(pluginConfig.getString(SentryConfig.RELEASE.key())); } - if (pluginConfig.hasPath(SentryConfig.CACHE_DIRPATH.key())){ + if (pluginConfig.hasPath(SentryConfig.CACHE_DIRPATH.key())) { options.setCacheDirPath(pluginConfig.getString(SentryConfig.CACHE_DIRPATH.key())); } - if (pluginConfig.hasPath(SentryConfig.MAX_CACHEITEMS.key())){ + if (pluginConfig.hasPath(SentryConfig.MAX_CACHEITEMS.key())) { options.setMaxCacheItems(pluginConfig.getInt(SentryConfig.MAX_CACHEITEMS.key())); } - if (pluginConfig.hasPath(SentryConfig.MAX_QUEUESIZE.key())){ + if (pluginConfig.hasPath(SentryConfig.MAX_QUEUESIZE.key())) { options.setMaxQueueSize(pluginConfig.getInt(SentryConfig.MAX_QUEUESIZE.key())); } - if (pluginConfig.hasPath(SentryConfig.FLUSH_TIMEOUTMILLIS.key())){ + if (pluginConfig.hasPath(SentryConfig.FLUSH_TIMEOUTMILLIS.key())) { options.setFlushTimeoutMillis(pluginConfig.getLong(SentryConfig.FLUSH_TIMEOUTMILLIS.key())); } - if (pluginConfig.hasPath(SentryConfig.ENABLE_EXTERNAL_CONFIGURATION.key())){ + if (pluginConfig.hasPath(SentryConfig.ENABLE_EXTERNAL_CONFIGURATION.key())) { options.setEnableExternalConfiguration(pluginConfig.getBoolean(SentryConfig.ENABLE_EXTERNAL_CONFIGURATION.key())); } Sentry.init(options); diff --git a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SinkConfig.java b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/config/SinkConfig.java similarity index 96% rename from seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SinkConfig.java rename to seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/config/SinkConfig.java index 4ebd152103c..e06d704b936 100644 --- a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SinkConfig.java +++ b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/config/SinkConfig.java @@ -15,7 +15,7 @@ * limitations under the License. */ -package org.apache.seatunnel.connectors.seatunnel.socket.sink; +package org.apache.seatunnel.connectors.seatunnel.socket.config; import static org.apache.seatunnel.connectors.seatunnel.socket.config.SocketSinkConfigOptions.HOST; import static org.apache.seatunnel.connectors.seatunnel.socket.config.SocketSinkConfigOptions.MAX_RETRIES; diff --git a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/exception/SocketConnectorErrorCode.java b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/exception/SocketConnectorErrorCode.java new file mode 100644 index 00000000000..74d6d4c28eb --- /dev/null +++ b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/exception/SocketConnectorErrorCode.java @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.socket.exception; + +import org.apache.seatunnel.common.exception.SeaTunnelErrorCode; + +public enum SocketConnectorErrorCode implements SeaTunnelErrorCode { + + SOCKET_SERVER_CONNECT_FAILED("SOCKET-01", "Cannot connect to socket server"), + SEND_MESSAGE_TO_SOCKET_SERVER_FAILED("SOCKET-02", "Failed to send message to socket server"), + SOCKET_WRITE_FAILED("SOCKET-03", "Unable to write; interrupted while doing another attempt"); + + private final String code; + + private final String description; + + SocketConnectorErrorCode(String code, String description) { + this.code = code; + this.description = description; + } + + @Override + public String getCode() { + return this.code; + } + + @Override + public String getDescription() { + return this.description; + } + + @Override + public String getErrorMessage() { + return SeaTunnelErrorCode.super.getErrorMessage(); + } +} diff --git a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/exception/SocketConnectorException.java b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/exception/SocketConnectorException.java new file mode 100644 index 00000000000..ff9a3d30857 --- /dev/null +++ b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/exception/SocketConnectorException.java @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.connectors.seatunnel.socket.exception; + +import org.apache.seatunnel.common.exception.SeaTunnelErrorCode; +import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException; + +public class SocketConnectorException extends SeaTunnelRuntimeException { + + public SocketConnectorException(SeaTunnelErrorCode seaTunnelErrorCode, String errorMessage) { + super(seaTunnelErrorCode, errorMessage); + } + + public SocketConnectorException(SeaTunnelErrorCode seaTunnelErrorCode, String errorMessage, Throwable cause) { + super(seaTunnelErrorCode, errorMessage, cause); + } + + public SocketConnectorException(SeaTunnelErrorCode seaTunnelErrorCode, Throwable cause) { + super(seaTunnelErrorCode, cause); + } +} diff --git a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SocketClient.java b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SocketClient.java index 3e83044b38f..fb485134d52 100644 --- a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SocketClient.java +++ b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SocketClient.java @@ -19,6 +19,9 @@ import org.apache.seatunnel.api.serialization.SerializationSchema; import org.apache.seatunnel.api.table.type.SeaTunnelRow; +import org.apache.seatunnel.connectors.seatunnel.socket.config.SinkConfig; +import org.apache.seatunnel.connectors.seatunnel.socket.exception.SocketConnectorErrorCode; +import org.apache.seatunnel.connectors.seatunnel.socket.exception.SocketConnectorException; import lombok.extern.slf4j.Slf4j; @@ -61,7 +64,9 @@ public void open() throws IOException { createConnection(); } } catch (IOException e) { - throw new IOException("Cannot connect to socket server at " + hostName + ":" + port, e); + throw new SocketConnectorException(SocketConnectorErrorCode.SOCKET_SERVER_CONNECT_FAILED, + String.format("Cannot connect to socket server at %s:%d", + hostName, port), e); } } @@ -70,37 +75,22 @@ public void write(SeaTunnelRow row) throws IOException { try { outputStream.write(msg); outputStream.flush(); - } catch (IOException e) { // if no re-tries are enable, fail immediately if (maxNumRetries == 0) { - throw new IOException( - "Failed to send message '" - + row - + "' to socket server at " - + hostName - + ":" - + port - + ". Connection re-tries are not enabled.", - e); + throw new SocketConnectorException(SocketConnectorErrorCode.SEND_MESSAGE_TO_SOCKET_SERVER_FAILED, + String.format("Failed to send message '%s' to socket server at %s:%d. Connection re-tries are not enabled.", + row, hostName, port), e); } log.error( - "Failed to send message '" - + row - + "' to socket server at " - + hostName - + ":" - + port - + ". Trying to reconnect...", - e); + "Failed to send message '{}' to socket server at {}:{}. Trying to reconnect...", + row, hostName, port, e); synchronized (SocketClient.class) { IOException lastException = null; retries = 0; - while (isRunning && (maxNumRetries < 0 || retries < maxNumRetries)) { - // first, clean up the old resources try { if (outputStream != null) { @@ -127,33 +117,22 @@ public void write(SeaTunnelRow row) throws IOException { return; } catch (IOException ee) { lastException = ee; - log.error( - "Re-connect to socket server and send message failed. Retry time(s): " - + retries, - ee); + log.error("Re-connect to socket server and send message failed. Retry time(s): {}", + retries, ee); } try { this.wait(CONNECTION_RETRY_DELAY); - } - catch (InterruptedException ex) { + } catch (InterruptedException ex) { Thread.currentThread().interrupt(); - throw new IOException( - "unable to write; interrupted while doing another attempt", e); + throw new SocketConnectorException(SocketConnectorErrorCode.SOCKET_WRITE_FAILED, + "unable to write; interrupted while doing another attempt", e); } } if (isRunning) { - throw new IOException( - "Failed to send message '" - + row - + "' to socket server at " - + hostName - + ":" - + port - + ". Failed after " - + retries - + " retries.", - lastException); + throw new SocketConnectorException(SocketConnectorErrorCode.SEND_MESSAGE_TO_SOCKET_SERVER_FAILED, + String.format("Failed to send message '%s' to socket server at %s:%d. Failed after %d retries.", + row, hostName, port, retries), lastException); } } } diff --git a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SocketSink.java b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SocketSink.java index 1bc88fa5aa5..51ad054202d 100644 --- a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SocketSink.java +++ b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SocketSink.java @@ -21,6 +21,7 @@ import static org.apache.seatunnel.connectors.seatunnel.socket.config.SocketSinkConfigOptions.PORT; import org.apache.seatunnel.api.common.PrepareFailException; +import org.apache.seatunnel.api.common.SeaTunnelAPIErrorCode; import org.apache.seatunnel.api.sink.SeaTunnelSink; import org.apache.seatunnel.api.sink.SinkWriter; import org.apache.seatunnel.api.table.type.SeaTunnelDataType; @@ -31,6 +32,8 @@ import org.apache.seatunnel.common.constants.PluginType; import org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSimpleSink; import org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSinkWriter; +import org.apache.seatunnel.connectors.seatunnel.socket.config.SinkConfig; +import org.apache.seatunnel.connectors.seatunnel.socket.exception.SocketConnectorException; import org.apache.seatunnel.shade.com.typesafe.config.Config; @@ -54,7 +57,10 @@ public void prepare(Config pluginConfig) throws PrepareFailException { this.pluginConfig = pluginConfig; CheckResult result = CheckConfigUtil.checkAllExists(pluginConfig, PORT.key(), HOST.key()); if (!result.isSuccess()) { - throw new PrepareFailException(getPluginName(), PluginType.SINK, result.getMsg()); + throw new SocketConnectorException(SeaTunnelAPIErrorCode.CONFIG_VALIDATION_FAILED, + String.format("PluginName: %s, PluginType: %s, Message: %s", + getPluginName(), PluginType.SINK, result.getMsg()) + ); } sinkConfig = new SinkConfig(pluginConfig); } @@ -65,12 +71,12 @@ public void setTypeInfo(SeaTunnelRowType seaTunnelRowType) { } @Override - public SeaTunnelDataType getConsumedType() { + public SeaTunnelDataType getConsumedType() { return seaTunnelRowType; } @Override - public AbstractSinkWriter createWriter(SinkWriter.Context context) throws IOException { + public AbstractSinkWriter createWriter(SinkWriter.Context context) throws IOException { return new SocketSinkWriter(sinkConfig, seaTunnelRowType); } } diff --git a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SocketSinkWriter.java b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SocketSinkWriter.java index 577835927ea..c901abfc152 100644 --- a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SocketSinkWriter.java +++ b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/sink/SocketSinkWriter.java @@ -21,6 +21,7 @@ import org.apache.seatunnel.api.table.type.SeaTunnelRow; import org.apache.seatunnel.api.table.type.SeaTunnelRowType; import org.apache.seatunnel.connectors.seatunnel.common.sink.AbstractSinkWriter; +import org.apache.seatunnel.connectors.seatunnel.socket.config.SinkConfig; import org.apache.seatunnel.format.json.JsonSerializationSchema; import java.io.IOException; diff --git a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/source/SocketSource.java b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/source/SocketSource.java index 82fab08bf86..3939eb0646b 100644 --- a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/source/SocketSource.java +++ b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/source/SocketSource.java @@ -22,6 +22,7 @@ import org.apache.seatunnel.api.common.JobContext; import org.apache.seatunnel.api.common.PrepareFailException; +import org.apache.seatunnel.api.common.SeaTunnelAPIErrorCode; import org.apache.seatunnel.api.source.Boundedness; import org.apache.seatunnel.api.source.SeaTunnelSource; import org.apache.seatunnel.api.table.type.BasicType; @@ -35,6 +36,7 @@ import org.apache.seatunnel.connectors.seatunnel.common.source.AbstractSingleSplitReader; import org.apache.seatunnel.connectors.seatunnel.common.source.AbstractSingleSplitSource; import org.apache.seatunnel.connectors.seatunnel.common.source.SingleSplitReaderContext; +import org.apache.seatunnel.connectors.seatunnel.socket.exception.SocketConnectorException; import org.apache.seatunnel.shade.com.typesafe.config.Config; @@ -59,7 +61,10 @@ public String getPluginName() { public void prepare(Config pluginConfig) throws PrepareFailException { CheckResult result = CheckConfigUtil.checkAllExists(pluginConfig, PORT.key(), HOST.key()); if (!result.isSuccess()) { - throw new PrepareFailException(getPluginName(), PluginType.SOURCE, result.getMsg()); + throw new SocketConnectorException(SeaTunnelAPIErrorCode.CONFIG_VALIDATION_FAILED, + String.format("PluginName: %s, PluginType: %s, Message: %s", + getPluginName(), PluginType.SOURCE, result.getMsg()) + ); } this.parameter = new SocketSourceParameter(pluginConfig); } @@ -71,12 +76,11 @@ public void setJobContext(JobContext jobContext) { @Override public SeaTunnelDataType getProducedType() { - return new SeaTunnelRowType(new String[]{"value"}, new SeaTunnelDataType[]{BasicType.STRING_TYPE}); + return new SeaTunnelRowType(new String[] {"value"}, new SeaTunnelDataType[] {BasicType.STRING_TYPE}); } @Override - public AbstractSingleSplitReader createReader(SingleSplitReaderContext readerContext) - throws Exception { + public AbstractSingleSplitReader createReader(SingleSplitReaderContext readerContext) throws Exception { return new SocketSourceReader(this.parameter, readerContext); } } diff --git a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/source/SocketSourceParameter.java b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/source/SocketSourceParameter.java index 09d0017872b..cb927986192 100644 --- a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/source/SocketSourceParameter.java +++ b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/source/SocketSourceParameter.java @@ -28,8 +28,8 @@ import java.util.Objects; public class SocketSourceParameter implements Serializable { - private String host; - private Integer port; + private final String host; + private final Integer port; public String getHost() { return StringUtils.isBlank(host) ? HOST.defaultValue() : host; diff --git a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/source/SocketSourceReader.java b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/source/SocketSourceReader.java index 8a86fa47bb3..34bd2b8f13b 100644 --- a/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/source/SocketSourceReader.java +++ b/seatunnel-connectors-v2/connector-socket/src/main/java/org/apache/seatunnel/connectors/seatunnel/socket/source/SocketSourceReader.java @@ -37,7 +37,7 @@ public class SocketSourceReader extends AbstractSingleSplitReader private final SocketSourceParameter parameter; private final SingleSplitReaderContext context; private Socket socket; - private String delimiter = "\n"; + private final String delimiter = "\n"; SocketSourceReader(SocketSourceParameter parameter, SingleSplitReaderContext context) { this.parameter = parameter; @@ -70,10 +70,10 @@ public void pollNext(Collector output) throws Exception { int delimPos; while (buffer.length() >= this.delimiter.length() && (delimPos = buffer.indexOf(this.delimiter)) != -1) { String record = buffer.substring(0, delimPos); - if (this.delimiter.equals("\n") && record.endsWith("\r")) { + if (record.endsWith("\r")) { record = record.substring(0, record.length() - 1); } - output.collect(new SeaTunnelRow(new Object[]{record})); + output.collect(new SeaTunnelRow(new Object[] {record})); buffer.delete(0, delimPos + this.delimiter.length()); } if (Boundedness.BOUNDED.equals(context.getBoundedness())) { @@ -84,7 +84,7 @@ record = record.substring(0, record.length() - 1); } } if (buffer.length() > 0) { - output.collect(new SeaTunnelRow(new Object[]{buffer.toString()})); + output.collect(new SeaTunnelRow(new Object[] {buffer.toString()})); } } } diff --git a/seatunnel-core/pom.xml b/seatunnel-core/pom.xml index ce058c65956..05428290063 100644 --- a/seatunnel-core/pom.xml +++ b/seatunnel-core/pom.xml @@ -31,7 +31,6 @@ pom - seatunnel-core-base seatunnel-core-starter seatunnel-flink-starter seatunnel-spark-starter diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/Seatunnel.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/Seatunnel.java deleted file mode 100644 index 4c70e776d62..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/Seatunnel.java +++ /dev/null @@ -1,72 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base; - -import org.apache.seatunnel.apis.base.command.CommandArgs; -import org.apache.seatunnel.common.config.ConfigRuntimeException; -import org.apache.seatunnel.core.base.command.Command; -import org.apache.seatunnel.core.base.exception.CommandException; - -import lombok.extern.slf4j.Slf4j; -import org.apache.commons.lang3.exception.ExceptionUtils; - -@Slf4j -public class Seatunnel { - - /** - * This method is the entrypoint of SeaTunnel. - * - * @param command commandArgs - * @param commandType - */ - public static void run(Command command) throws CommandException { - try { - command.execute(); - } catch (ConfigRuntimeException e) { - showConfigError(e); - throw e; - } catch (Exception e) { - showFatalError(e); - throw e; - } - } - - private static void showConfigError(Throwable throwable) { - log.error( - "\n\n===============================================================================\n\n"); - String errorMsg = throwable.getMessage(); - log.error("Config Error:\n"); - log.error("Reason: {} \n", errorMsg); - log.error( - "\n===============================================================================\n\n\n"); - } - - private static void showFatalError(Throwable throwable) { - log.error( - "\n\n===============================================================================\n\n"); - String errorMsg = throwable.getMessage(); - log.error("Fatal Error, \n"); - // FIX - log.error( - "Please submit bug report in https://github.com/apache/incubator-seatunnel/issues\n"); - log.error("Reason:{} \n", errorMsg); - log.error("Exception StackTrace:{} ", ExceptionUtils.getStackTrace(throwable)); - log.error( - "\n===============================================================================\n\n\n"); - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/Starter.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/Starter.java deleted file mode 100644 index e5d2c2fe27b..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/Starter.java +++ /dev/null @@ -1,33 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base; - -import java.util.List; - -/** - * a Starter is for building a commandline start command - * based on different engine for SeaTunnel job. - */ -public interface Starter { - - /** - * return the SeaTunnel job commandline start commands - */ - List buildCommands() throws Exception; - -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/AbstractCommandArgs.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/AbstractCommandArgs.java deleted file mode 100644 index 022f78457c7..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/AbstractCommandArgs.java +++ /dev/null @@ -1,104 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.command; - -import org.apache.seatunnel.apis.base.command.CommandArgs; -import org.apache.seatunnel.common.config.DeployMode; -import org.apache.seatunnel.core.base.config.EngineType; - -import com.beust.jcommander.Parameter; - -import java.util.Collections; -import java.util.List; - -public abstract class AbstractCommandArgs implements CommandArgs { - - @Parameter(names = {"-c", "--config"}, - description = "Config file", - required = true) - private String configFile; - - @Parameter(names = {"-i", "--variable"}, - splitter = NoopParameterSplitter.class, - description = "variable substitution, such as -i city=beijing, or -i date=20190318") - private List variables = Collections.emptyList(); - - // todo: use command type enum - @Parameter(names = {"-ck", "--check"}, - description = "check config") - private boolean checkConfig = false; - - @Parameter(names = {"-h", "--help"}, - help = true, - description = "Show the usage message") - private boolean help = false; - - /** - * Undefined parameters parsed will be stored here as engine original command parameters. - */ - private List originalParameters; - - public String getConfigFile() { - return configFile; - } - - public void setConfigFile(String configFile) { - this.configFile = configFile; - } - - public List getVariables() { - return variables; - } - - public void setVariables(List variables) { - this.variables = variables; - } - - public boolean isCheckConfig() { - return checkConfig; - } - - public void setCheckConfig(boolean checkConfig) { - this.checkConfig = checkConfig; - } - - public boolean isHelp() { - return help; - } - - public void setHelp(boolean help) { - this.help = help; - } - - public List getOriginalParameters() { - return originalParameters; - } - - public void setOriginalParameters(List originalParameters) { - this.originalParameters = originalParameters; - } - - public EngineType getEngineType() { - throw new UnsupportedOperationException("abstract class CommandArgs not support this method"); - } - - public DeployMode getDeployMode() { - throw new UnsupportedOperationException("abstract class CommandArgs not support this method"); - } - -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/BaseTaskExecuteCommand.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/BaseTaskExecuteCommand.java deleted file mode 100644 index 908931da5a4..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/BaseTaskExecuteCommand.java +++ /dev/null @@ -1,148 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.command; - -import org.apache.seatunnel.apis.base.env.RuntimeEnv; -import org.apache.seatunnel.apis.base.plugin.Plugin; -import org.apache.seatunnel.apis.base.plugin.PluginClosedException; -import org.apache.seatunnel.common.Constants; -import org.apache.seatunnel.common.config.CheckResult; -import org.apache.seatunnel.common.config.Common; -import org.apache.seatunnel.common.config.DeployMode; -import org.apache.seatunnel.core.base.utils.CompressionUtils; - -import lombok.extern.slf4j.Slf4j; - -import java.io.File; -import java.util.List; -import java.util.Objects; - -/** - * Base task execute command. - * - * @param command args. - */ -@Slf4j -public abstract class BaseTaskExecuteCommand implements Command { - - /** - * Check the plugin config. - * - * @param plugins plugin list. - */ - @SafeVarargs - protected final void baseCheckConfig(List>... plugins) { - pluginCheck(plugins); - deployModeCheck(); - } - - /** - * Execute prepare method defined in {@link Plugin}. - * - * @param env runtimeEnv - * @param plugins plugin list - */ - @SafeVarargs - protected final void prepare(E env, List>... plugins) { - for (List> pluginList : plugins) { - pluginList.forEach(plugin -> plugin.prepare(env)); - } - } - - /** - * Execute close method defined in {@link Plugin} - * - * @param plugins plugin list - */ - @SafeVarargs - protected final void close(List>... plugins) { - PluginClosedException exceptionHolder = null; - for (List> pluginList : plugins) { - for (Plugin plugin : pluginList) { - try (Plugin closed = plugin) { - // ignore - } catch (Exception e) { - exceptionHolder = exceptionHolder == null ? - new PluginClosedException("below plugins closed error:") : exceptionHolder; - exceptionHolder.addSuppressed(new PluginClosedException( - String.format("plugin %s closed error", plugin.getClass()), e)); - } - } - } - if (exceptionHolder != null) { - throw exceptionHolder; - } - } - - /** - * Print the logo. - */ - protected void showAsciiLogo() { - String printAsciiLogo = System.getenv("SEATUNNEL_PRINT_ASCII_LOGO"); - if ("true".equalsIgnoreCase(printAsciiLogo)) { - log.info('\n' + Constants.ST_LOGO); - log.info(Constants.COPYRIGHT_LINE); - } - } - - /** - * Execute the checkConfig method defined in {@link Plugin}. - * - * @param plugins plugin list - */ - private void pluginCheck(List>... plugins) { - for (List> pluginList : plugins) { - for (Plugin plugin : pluginList) { - CheckResult checkResult; - try { - checkResult = plugin.checkConfig(); - } catch (Exception e) { - checkResult = CheckResult.error(e.getMessage()); - } - if (!checkResult.isSuccess()) { - log.error("Plugin[{}] contains invalid config, error: {} \n", plugin.getClass().getName(), checkResult.getMsg()); - System.exit(-1); // invalid configuration - } - } - } - } - - private void deployModeCheck() { - final DeployMode mode = Common.getDeployMode(); - if (DeployMode.CLUSTER == mode) { - - log.info("preparing cluster mode work dir files..."); - File workDir = new File("."); - - for (File file : Objects.requireNonNull(workDir.listFiles())) { - log.warn("\t list file: {} ", file.getAbsolutePath()); - } - // decompress plugin dir - File compressedFile = new File("plugins.tar.gz"); - - try { - File tempFile = CompressionUtils.unGzip(compressedFile, workDir); - CompressionUtils.unTar(tempFile, workDir); - } catch (Exception e) { - log.error("failed to decompress plugins.tar.gz", e); - System.exit(-1); - } - log.info("succeeded to decompress plugins.tar.gz"); - } - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/CommandBuilder.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/CommandBuilder.java deleted file mode 100644 index 09e4e83f745..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/CommandBuilder.java +++ /dev/null @@ -1,25 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.command; - -import org.apache.seatunnel.apis.base.command.CommandArgs; - -@FunctionalInterface -public interface CommandBuilder { - Command buildCommand(T commandArgs); -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/NoopParameterSplitter.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/NoopParameterSplitter.java deleted file mode 100644 index 8a07d4c1da3..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/command/NoopParameterSplitter.java +++ /dev/null @@ -1,34 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.command; - -import com.beust.jcommander.converters.IParameterSplitter; - -import java.util.Collections; -import java.util.List; - -/** - * An implementation class that does nothing on the value of variable. - */ -public class NoopParameterSplitter implements IParameterSplitter { - - @Override - public List split(String value) { - return Collections.singletonList(value); - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/AbstractExecutionContext.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/AbstractExecutionContext.java deleted file mode 100644 index 92d4232c350..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/AbstractExecutionContext.java +++ /dev/null @@ -1,92 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.config; - -import org.apache.seatunnel.apis.base.api.BaseSink; -import org.apache.seatunnel.apis.base.api.BaseSource; -import org.apache.seatunnel.apis.base.api.BaseTransform; -import org.apache.seatunnel.apis.base.env.RuntimeEnv; -import org.apache.seatunnel.common.constants.JobMode; -import org.apache.seatunnel.common.constants.PluginType; -import org.apache.seatunnel.plugin.discovery.PluginIdentifier; - -import org.apache.seatunnel.shade.com.typesafe.config.Config; - -import java.net.URL; -import java.util.Arrays; -import java.util.List; -import java.util.function.Function; -import java.util.stream.Collectors; -import java.util.stream.Stream; - -/** - * The ExecutionContext contains all configuration needed to run the job. - * - * @param environment type. - */ -public abstract class AbstractExecutionContext { - - private final Config config; - private final EngineType engine; - - private final ENVIRONMENT environment; - private final JobMode jobMode; - - public AbstractExecutionContext(Config config, EngineType engine) { - this.config = config; - this.engine = engine; - this.environment = new EnvironmentFactory(config, engine).getEnvironment(); - this.jobMode = environment.getJobMode(); - } - - public Config getRootConfig() { - return config; - } - - public EngineType getEngine() { - return engine; - } - - public ENVIRONMENT getEnvironment() { - return environment; - } - - public JobMode getJobMode() { - return jobMode; - } - - public abstract List> getSources(); - - public abstract List> getTransforms(); - - public abstract List> getSinks(); - - public abstract List getPluginJars(); - - @SuppressWarnings("checkstyle:Indentation") - protected List getPluginIdentifiers(PluginType... pluginTypes) { - return Arrays.stream(pluginTypes).flatMap((Function>) pluginType -> { - List configList = config.getConfigList(pluginType.getType()); - return configList.stream() - .map(pluginConfig -> PluginIdentifier - .of(engine.getEngine(), - pluginType.getType(), - pluginConfig.getString("plugin_name"))); - }).collect(Collectors.toList()); - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/ConfigBuilder.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/ConfigBuilder.java deleted file mode 100644 index 4a1d4485b3e..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/ConfigBuilder.java +++ /dev/null @@ -1,70 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.config; - -import org.apache.seatunnel.common.config.ConfigRuntimeException; - -import org.apache.seatunnel.shade.com.typesafe.config.Config; -import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory; -import org.apache.seatunnel.shade.com.typesafe.config.ConfigRenderOptions; -import org.apache.seatunnel.shade.com.typesafe.config.ConfigResolveOptions; - -import lombok.extern.slf4j.Slf4j; - -import java.nio.file.Path; - -/** - * Used to build the {@link Config} from file. - */ -@Slf4j -public class ConfigBuilder { - - private final Path configFile; - private final Config config; - - public ConfigBuilder(Path configFile) { - this.configFile = configFile; - this.config = load(); - } - - private Config load() { - - if (configFile == null) { - throw new ConfigRuntimeException("Please specify config file"); - } - - log.info("Loading config file: {}", configFile); - - // variables substitution / variables resolution order: - // config file --> system environment --> java properties - Config config = ConfigFactory - .parseFile(configFile.toFile()) - .resolve(ConfigResolveOptions.defaults().setAllowUnresolved(true)) - .resolveWith(ConfigFactory.systemProperties(), - ConfigResolveOptions.defaults().setAllowUnresolved(true)); - - ConfigRenderOptions options = ConfigRenderOptions.concise().setFormatted(true); - log.info("parsed config file: {}", config.root().render(options)); - return config; - } - - public Config getConfig() { - return config; - } - -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/ConfigParser.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/ConfigParser.java deleted file mode 100644 index 8886b62894a..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/ConfigParser.java +++ /dev/null @@ -1,45 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.config; - -import org.apache.seatunnel.shade.com.typesafe.config.Config; -import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory; -import org.apache.seatunnel.shade.com.typesafe.config.ConfigResolveOptions; - -import java.io.File; -import java.io.FileNotFoundException; -import java.util.Map; -import java.util.stream.Collectors; - -public class ConfigParser { - - public static Map getConfigEnvValues(String configFile) throws FileNotFoundException { - File file = new File(configFile); - if (!file.exists()) { - throw new FileNotFoundException("config file '" + file + "' does not exists!"); - } - Config appConfig = ConfigFactory.parseFile(file) - .resolve(ConfigResolveOptions.defaults().setAllowUnresolved(true)) - .resolveWith(ConfigFactory.systemProperties(), ConfigResolveOptions.defaults().setAllowUnresolved(true)); - - return appConfig.getConfig("env") - .entrySet() - .stream() - .collect(Collectors.toMap(Map.Entry::getKey, e -> e.getValue().unwrapped().toString())); - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/EngineType.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/EngineType.java deleted file mode 100644 index bb8f4ddd060..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/EngineType.java +++ /dev/null @@ -1,34 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.config; - -public enum EngineType { - SPARK("spark"), - FLINK("flink"), - ; - - private final String engine; - - EngineType(String engine) { - this.engine = engine; - } - - public String getEngine() { - return engine; - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/EnvironmentFactory.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/EnvironmentFactory.java deleted file mode 100644 index 0386159ccd4..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/EnvironmentFactory.java +++ /dev/null @@ -1,95 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.config; - -import org.apache.seatunnel.apis.base.env.RuntimeEnv; -import org.apache.seatunnel.common.constants.JobMode; -import org.apache.seatunnel.common.constants.PluginType; -import org.apache.seatunnel.flink.FlinkEnvironment; -import org.apache.seatunnel.spark.SparkEnvironment; - -import org.apache.seatunnel.shade.com.typesafe.config.Config; - -import java.util.List; - -/** - * Used to create the {@link RuntimeEnv}. - * - * @param environment type - */ -public class EnvironmentFactory { - - private static final String PLUGIN_NAME_KEY = "plugin_name"; - - private final Config config; - private final EngineType engine; - - public EnvironmentFactory(Config config, EngineType engine) { - this.config = config; - this.engine = engine; - } - - // todo:put this method into submodule to avoid dependency on the engine - public synchronized ENVIRONMENT getEnvironment() { - Config envConfig = config.getConfig("env"); - boolean enableHive = checkIsContainHive(); - ENVIRONMENT env; - switch (engine) { - case SPARK: - env = (ENVIRONMENT) new SparkEnvironment().setEnableHive(enableHive); - break; - case FLINK: - env = (ENVIRONMENT) new FlinkEnvironment(); - break; - default: - throw new IllegalArgumentException("Engine: " + engine + " is not supported"); - } - env.setConfig(envConfig) - .setJobMode(getJobMode(envConfig)).prepare(); - return env; - } - - private boolean checkIsContainHive() { - List sourceConfigList = config.getConfigList(PluginType.SOURCE.getType()); - for (Config c : sourceConfigList) { - if (c.getString(PLUGIN_NAME_KEY).toLowerCase().contains("hive")) { - return true; - } - } - List sinkConfigList = config.getConfigList(PluginType.SINK.getType()); - for (Config c : sinkConfigList) { - if (c.getString(PLUGIN_NAME_KEY).toLowerCase().contains("hive")) { - return true; - } - } - return false; - } - - private JobMode getJobMode(Config envConfig) { - JobMode jobMode; - if (envConfig.hasPath("job.mode")) { - jobMode = envConfig.getEnum(JobMode.class, "job.mode"); - } else { - //Compatible with previous logic - List sourceConfigList = config.getConfigList(PluginType.SOURCE.getType()); - jobMode = sourceConfigList.get(0).getString(PLUGIN_NAME_KEY).toLowerCase().endsWith("stream") ? JobMode.STREAMING : JobMode.BATCH; - } - return jobMode; - } - -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/ExecutionFactory.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/ExecutionFactory.java deleted file mode 100644 index 25c4f4d7dc5..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/config/ExecutionFactory.java +++ /dev/null @@ -1,82 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.config; - -import org.apache.seatunnel.apis.base.api.BaseSink; -import org.apache.seatunnel.apis.base.api.BaseSource; -import org.apache.seatunnel.apis.base.api.BaseTransform; -import org.apache.seatunnel.apis.base.env.Execution; -import org.apache.seatunnel.apis.base.env.RuntimeEnv; -import org.apache.seatunnel.flink.FlinkEnvironment; -import org.apache.seatunnel.flink.batch.FlinkBatchExecution; -import org.apache.seatunnel.flink.stream.FlinkStreamExecution; -import org.apache.seatunnel.spark.SparkEnvironment; -import org.apache.seatunnel.spark.batch.SparkBatchExecution; -import org.apache.seatunnel.spark.stream.SparkStreamingExecution; -import org.apache.seatunnel.spark.structuredstream.StructuredStreamingExecution; - -import lombok.extern.slf4j.Slf4j; - -/** - * Used to create {@link Execution}. - * - * @param environment type - */ -@Slf4j -public class ExecutionFactory { - - public AbstractExecutionContext executionContext; - - public ExecutionFactory(AbstractExecutionContext executionContext) { - this.executionContext = executionContext; - } - - public Execution, BaseTransform, BaseSink, ENVIRONMENT> createExecution() { - Execution execution = null; - switch (executionContext.getEngine()) { - case SPARK: - SparkEnvironment sparkEnvironment = (SparkEnvironment) executionContext.getEnvironment(); - switch (executionContext.getJobMode()) { - case STREAMING: - execution = new SparkStreamingExecution(sparkEnvironment); - break; - case STRUCTURED_STREAMING: - execution = new StructuredStreamingExecution(sparkEnvironment); - break; - default: - execution = new SparkBatchExecution(sparkEnvironment); - } - break; - case FLINK: - FlinkEnvironment flinkEnvironment = (FlinkEnvironment) executionContext.getEnvironment(); - switch (executionContext.getJobMode()) { - case STREAMING: - execution = new FlinkStreamExecution(flinkEnvironment); - break; - default: - execution = new FlinkBatchExecution(flinkEnvironment); - } - break; - default: - throw new IllegalArgumentException("No suitable engine"); - } - log.info("current execution is [{}]", execution.getClass().getName()); - return (Execution, BaseTransform, BaseSink, ENVIRONMENT>) execution; - } - -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/constants/Constants.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/constants/Constants.java deleted file mode 100644 index 4ebb0792fd8..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/constants/Constants.java +++ /dev/null @@ -1,22 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.constants; - -public class Constants { - public static final int USAGE_EXIT_CODE = 234; -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/exception/CommandException.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/exception/CommandException.java deleted file mode 100644 index 33f2013b22a..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/exception/CommandException.java +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.exception; - -public class CommandException extends Exception { - public CommandException(String message) { - super(message); - } - - public CommandException(String message, Throwable cause) { - super(message, cause); - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/exception/CommandExecuteException.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/exception/CommandExecuteException.java deleted file mode 100644 index a3a8ee421ca..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/exception/CommandExecuteException.java +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.exception; - -public class CommandExecuteException extends CommandException { - public CommandExecuteException(String message) { - super(message); - } - - public CommandExecuteException(String message, Throwable cause) { - super(message, cause); - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/utils/CommandLineUtils.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/utils/CommandLineUtils.java deleted file mode 100644 index 748933b2b31..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/utils/CommandLineUtils.java +++ /dev/null @@ -1,64 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.utils; - -import static org.apache.seatunnel.core.base.constants.Constants.USAGE_EXIT_CODE; - -import org.apache.seatunnel.core.base.command.AbstractCommandArgs; - -import com.beust.jcommander.JCommander; -import com.beust.jcommander.ParameterException; -import com.beust.jcommander.UnixStyleUsageFormatter; - -public class CommandLineUtils { - - private CommandLineUtils() { - throw new UnsupportedOperationException("CommandLineUtils is a utility class and cannot be instantiated"); - } - - public static T parse(String[] args, T obj) { - return parse(args, obj, null, false); - } - - public static T parse(String[] args, T obj, String programName, boolean acceptUnknownOptions) { - JCommander jCommander = JCommander.newBuilder() - .programName(programName) - .addObject(obj) - .acceptUnknownOptions(acceptUnknownOptions) - .build(); - try { - jCommander.parse(args); - // The args is not belongs to SeaTunnel, add into engine original parameters - obj.setOriginalParameters(jCommander.getUnknownOptions()); - } catch (ParameterException e) { - System.err.println(e.getLocalizedMessage()); - exit(jCommander); - } - - if (obj.isHelp()) { - exit(jCommander); - } - return obj; - } - - private static void exit(JCommander jCommander) { - jCommander.setUsageFormatter(new UnixStyleUsageFormatter(jCommander)); - jCommander.usage(); - System.exit(USAGE_EXIT_CODE); - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/utils/CompressionUtils.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/utils/CompressionUtils.java deleted file mode 100644 index 636c2c9cbc9..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/utils/CompressionUtils.java +++ /dev/null @@ -1,160 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.utils; - -import lombok.extern.slf4j.Slf4j; -import org.apache.commons.compress.archivers.ArchiveException; -import org.apache.commons.compress.archivers.ArchiveStreamFactory; -import org.apache.commons.compress.archivers.tar.TarArchiveEntry; -import org.apache.commons.compress.archivers.tar.TarArchiveInputStream; -import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream; -import org.apache.commons.compress.compressors.gzip.GzipCompressorOutputStream; -import org.apache.commons.compress.utils.IOUtils; - -import java.io.BufferedOutputStream; -import java.io.File; -import java.io.FileInputStream; -import java.io.FileNotFoundException; -import java.io.FileOutputStream; -import java.io.IOException; -import java.io.InputStream; -import java.io.OutputStream; -import java.nio.file.FileVisitResult; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.SimpleFileVisitor; -import java.nio.file.attribute.BasicFileAttributes; -import java.util.LinkedList; -import java.util.List; -import java.util.zip.GZIPInputStream; - -@Slf4j -public final class CompressionUtils { - - private CompressionUtils() { - } - - /** - * Compress directory to a 'tar.gz' format file. - * - * @param inputDir all files in the directory will be included, except for symbolic links. - * @param outputFile the output tarball file. - */ - public static void tarGzip(final Path inputDir, final Path outputFile) throws IOException { - log.info("Tar directory '{}' to file '{}'.", inputDir, outputFile); - try (OutputStream out = Files.newOutputStream(outputFile); - BufferedOutputStream bufferedOut = new BufferedOutputStream(out); - GzipCompressorOutputStream gzOut = new GzipCompressorOutputStream(bufferedOut); - TarArchiveOutputStream tarOut = new TarArchiveOutputStream(gzOut)) { - Files.walkFileTree(inputDir, new SimpleFileVisitor() { - @Override - public FileVisitResult visitFile(Path path, BasicFileAttributes attrs) throws IOException { - if (attrs.isSymbolicLink()) { - return FileVisitResult.CONTINUE; - } - String fileName = inputDir.relativize(path).toString(); - TarArchiveEntry archiveEntry = new TarArchiveEntry(path.toFile(), fileName); - tarOut.putArchiveEntry(archiveEntry); - Files.copy(path, tarOut); - tarOut.closeArchiveEntry(); - return FileVisitResult.CONTINUE; - } - }); - tarOut.finish(); - log.info("Creating tar file '{}'.", outputFile); - } catch (IOException e) { - log.error("Error when tar directory '{}' to file '{}'.", inputDir, outputFile); - throw e; - } - } - - /** - * Untar an input file into an output file. - *

- * The output file is created in the output folder, having the same name - * as the input file, minus the '.tar' extension. - * - * @param inputFile the input .tar file - * @param outputDir the output directory file. - * @throws IOException io exception - * @throws FileNotFoundException file not found exception - * @throws ArchiveException archive exception - */ - public static void unTar(final File inputFile, final File outputDir) throws IOException, ArchiveException { - - log.info("Untaring {} to dir {}.", inputFile.getAbsolutePath(), outputDir.getAbsolutePath()); - - final List untaredFiles = new LinkedList<>(); - try (final InputStream is = new FileInputStream(inputFile); - final TarArchiveInputStream debInputStream = (TarArchiveInputStream) new ArchiveStreamFactory().createArchiveInputStream("tar", is)) { - TarArchiveEntry entry = null; - while ((entry = (TarArchiveEntry) debInputStream.getNextEntry()) != null) { - final File outputFile = new File(outputDir, entry.getName()).toPath().normalize().toFile(); - if (!outputFile.toPath().normalize().startsWith(outputDir.toPath())) { - throw new IllegalStateException("Bad zip entry"); - } - if (entry.isDirectory()) { - log.info("Attempting to write output directory {}.", outputFile.getAbsolutePath()); - if (!outputFile.exists()) { - log.info("Attempting to create output directory {}.", outputFile.getAbsolutePath()); - if (!outputFile.mkdirs()) { - throw new IllegalStateException(String.format("Couldn't create directory %s.", outputFile.getAbsolutePath())); - } - } - } else { - log.info("Creating output file {}.", outputFile.getAbsolutePath()); - File outputParentFile = outputFile.getParentFile(); - if (outputParentFile != null && !outputParentFile.exists()) { - outputParentFile.mkdirs(); - } - final OutputStream outputFileStream = new FileOutputStream(outputFile); - IOUtils.copy(debInputStream, outputFileStream); - outputFileStream.close(); - } - untaredFiles.add(outputFile); - } - } - } - - /** - * Ungzip an input file into an output file. - *

- * The output file is created in the output folder, having the same name - * as the input file, minus the '.gz' extension. - * - * @param inputFile the input .gz file - * @param outputDir the output directory file. - * @return The {@link File} with the ungzipped content. - * @throws IOException io exception - * @throws FileNotFoundException file not found exception - */ - public static File unGzip(final File inputFile, final File outputDir) throws IOException { - - log.info("Unzipping {} to dir {}.", inputFile.getAbsolutePath(), outputDir.getAbsolutePath()); - - final File outputFile = new File(outputDir, inputFile.getName().substring(0, inputFile.getName().length() - 3)); - - try (final FileInputStream fis = new FileInputStream(inputFile); - final GZIPInputStream in = new GZIPInputStream(fis); - final FileOutputStream out = new FileOutputStream(outputFile)) { - IOUtils.copy(in, out); - } - return outputFile; - } - -} diff --git a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/utils/FileUtils.java b/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/utils/FileUtils.java deleted file mode 100644 index edc15b17b49..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/main/java/org/apache/seatunnel/core/base/utils/FileUtils.java +++ /dev/null @@ -1,70 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.utils; - -import org.apache.seatunnel.core.base.command.AbstractCommandArgs; - -import java.io.File; -import java.nio.file.Path; -import java.nio.file.Paths; - -public class FileUtils { - - private FileUtils() { - throw new UnsupportedOperationException("This class cannot be instantiated"); - } - - /** - * Get the seatunnel config path. - * In client mode, the path to the config file is directly given by user. - * In cluster mode, the path to the config file is the `executor path/config file name`. - * - * @param args args - * @return path of the seatunnel config file. - */ - public static Path getConfigPath(AbstractCommandArgs args) { - checkNotNull(args, "args"); - checkNotNull(args.getDeployMode(), "deploy mode"); - switch (args.getDeployMode()) { - case CLIENT: - return Paths.get(args.getConfigFile()); - case CLUSTER: - return Paths.get(getFileName(args.getConfigFile())); - default: - throw new IllegalArgumentException("Unsupported deploy mode: " + args.getDeployMode()); - } - } - - /** - * Get the file name from the given path. - * e.g. seatunnel/conf/config.conf -> config.conf - * - * @param filePath the path to the file - * @return file name - */ - private static String getFileName(String filePath) { - checkNotNull(filePath, "file path"); - return filePath.substring(filePath.lastIndexOf(File.separatorChar) + 1); - } - - private static void checkNotNull(T arg, String argName) { - if (arg == null) { - throw new IllegalArgumentException(argName + " cannot be null"); - } - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/test/java/org/apache/seatunnel/core/base/command/BaseTaskExecuteCommandTest.java b/seatunnel-core/seatunnel-core-base/src/test/java/org/apache/seatunnel/core/base/command/BaseTaskExecuteCommandTest.java deleted file mode 100644 index 2d41887b55f..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/test/java/org/apache/seatunnel/core/base/command/BaseTaskExecuteCommandTest.java +++ /dev/null @@ -1,120 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.command; - -import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertThrows; - -import org.apache.seatunnel.apis.base.plugin.Plugin; -import org.apache.seatunnel.apis.base.plugin.PluginClosedException; -import org.apache.seatunnel.flink.FlinkEnvironment; - -import org.apache.seatunnel.shade.com.typesafe.config.Config; - -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; - -import java.util.ArrayList; -import java.util.List; - -public class BaseTaskExecuteCommandTest { - - private static int CLOSE_TIMES = 0; - - @BeforeEach - public void setUp() { - CLOSE_TIMES = 0; - } - - @Test - public void testClose() { - List pluginListA = new ArrayList<>(); - pluginListA.add(new MockPlugin()); - pluginListA.add(new MockPlugin()); - List pluginListB = new ArrayList<>(); - pluginListB.add(new MockPlugin()); - pluginListB.add(new MockPlugin()); - MockTaskExecutorCommand mockTaskExecutorCommand = new MockTaskExecutorCommand(); - mockTaskExecutorCommand.close(pluginListA, pluginListB); - assertEquals(Integer.parseInt("0"), CLOSE_TIMES); - } - - @Test - public void testExceptionClose() { - List pluginListA = new ArrayList<>(); - pluginListA.add(new MockExceptionPlugin()); - pluginListA.add(new MockExceptionPlugin()); - List pluginListB = new ArrayList<>(); - pluginListB.add(new MockExceptionPlugin()); - pluginListB.add(new MockExceptionPlugin()); - MockTaskExecutorCommand mockTaskExecutorCommand = new MockTaskExecutorCommand(); - try { - mockTaskExecutorCommand.close(pluginListA, pluginListB); - } catch (Exception ex) { - // just print into console - ex.printStackTrace(); - } - assertEquals(Integer.parseInt("4"), CLOSE_TIMES); - assertThrows(PluginClosedException.class, () -> mockTaskExecutorCommand.close(pluginListA)); - } - - private static class MockPlugin implements Plugin { - - @Override - public void setConfig(Config config) { - } - - @Override - public Config getConfig() { - return null; - } - - @Override - public void close() { - - } - - } - - private static class MockExceptionPlugin implements Plugin { - - @Override - public void setConfig(Config config) { - } - - @Override - public Config getConfig() { - return null; - } - - @Override - public void close() { - CLOSE_TIMES++; - throw new PluginClosedException("Test close with exception, closeTimes:" + CLOSE_TIMES); - } - - } - - private static class MockTaskExecutorCommand extends BaseTaskExecuteCommand { - - @Override - public void execute() { - - } - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/test/java/org/apache/seatunnel/core/base/utils/CompressionUtilsTest.java b/seatunnel-core/seatunnel-core-base/src/test/java/org/apache/seatunnel/core/base/utils/CompressionUtilsTest.java deleted file mode 100644 index 4e6e225dcad..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/test/java/org/apache/seatunnel/core/base/utils/CompressionUtilsTest.java +++ /dev/null @@ -1,50 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.utils; - -import static org.junit.jupiter.api.Assertions.assertTrue; - -import org.junit.jupiter.api.Test; - -import java.io.File; -import java.io.IOException; -import java.nio.file.Files; -import java.nio.file.Path; -import java.util.Comparator; - -public class CompressionUtilsTest { - - @Test - public void tar() throws IOException { - Path pluginRootDir = Files.createTempDirectory("plugins_"); - Path outputFile = Files.createTempFile("plugins_", ".tar.gz"); - Path pluginDir = Files.createDirectory(pluginRootDir.resolve("plugin1")); - Path pluginLibDir = Files.createDirectory(pluginDir.resolve("lib")); - Files.createFile(pluginLibDir.resolve("a.jar")); - Files.createFile(pluginLibDir.resolve("b.jar")); - CompressionUtils.tarGzip(pluginRootDir, outputFile); - assertTrue(Files.exists(outputFile)); - - Files.walk(pluginRootDir) - .sorted(Comparator.reverseOrder()) - .map(Path::toFile) - .forEach(File::delete); - - Files.delete(outputFile); - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/test/java/org/apache/seatunnel/core/base/utils/FileUtilsTest.java b/seatunnel-core/seatunnel-core-base/src/test/java/org/apache/seatunnel/core/base/utils/FileUtilsTest.java deleted file mode 100644 index efd1f8924fb..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/test/java/org/apache/seatunnel/core/base/utils/FileUtilsTest.java +++ /dev/null @@ -1,58 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.seatunnel.core.base.utils; - -import org.apache.seatunnel.common.config.DeployMode; -import org.apache.seatunnel.core.base.command.AbstractCommandArgs; - -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; - -import java.net.URISyntaxException; -import java.nio.file.Path; -import java.nio.file.Paths; - -public class FileUtilsTest { - - @Test - public void getConfigPath() throws URISyntaxException { - // test client mode. - SparkCommandArgs sparkCommandArgs = new SparkCommandArgs(); - sparkCommandArgs.setDeployMode(DeployMode.CLIENT); - Path expectConfPath = Paths.get(FileUtilsTest.class.getResource("/flink.batch.conf").toURI()); - sparkCommandArgs.setConfigFile(expectConfPath.toString()); - Assertions.assertEquals(expectConfPath, FileUtils.getConfigPath(sparkCommandArgs)); - - // test cluster mode - sparkCommandArgs.setDeployMode(DeployMode.CLUSTER); - Assertions.assertEquals("flink.batch.conf", FileUtils.getConfigPath(sparkCommandArgs).toString()); - } - - private static class SparkCommandArgs extends AbstractCommandArgs { - private DeployMode deployMode; - - public void setDeployMode(DeployMode deployMode) { - this.deployMode = deployMode; - } - - @Override - public DeployMode getDeployMode() { - return deployMode; - } - } -} diff --git a/seatunnel-core/seatunnel-core-base/src/test/resources/flink.batch.conf b/seatunnel-core/seatunnel-core-base/src/test/resources/flink.batch.conf deleted file mode 100644 index 45b3e853272..00000000000 --- a/seatunnel-core/seatunnel-core-base/src/test/resources/flink.batch.conf +++ /dev/null @@ -1,56 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -###### -###### This config file is a demonstration of batch processing in SeaTunnel config -###### - -env { - # You can set flink configuration here - execution.parallelism = 1 -} - -source { - # This is a example input plugin **only for test and demonstrate the feature input plugin** - FileSource { - path = "hdfs://localhost:9000/output/text" - format.type = "text" - schema = "string" - result_table_name = "test" - } - - # If you would like to get more information about how to configure seatunnel and see full list of input plugins, - # please go to https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Fake -} - -transform { - Sql { - sql = "select * from test" - } - - # If you would like to get more information about how to configure seatunnel and see full list of filter plugins, - # please go to https://seatunnel.apache.org/docs/flink/configuration/transform-plugins/Sql -} - -sink { - # choose stdout output plugin to output data to console - ConsoleSink { - } - - # If you would like to get more information about how to configure seatunnel and see full list of output plugins, - # please go to https://seatunnel.apache.org/docs/flink/configuration/sink-plugins/Console -} diff --git a/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/FileUtils.java b/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/FileUtils.java index 98ddc2ec949..39e5d30939e 100644 --- a/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/FileUtils.java +++ b/seatunnel-core/seatunnel-core-starter/src/main/java/org/apache/seatunnel/core/starter/utils/FileUtils.java @@ -17,12 +17,17 @@ package org.apache.seatunnel.core.starter.utils; +import org.apache.seatunnel.common.exception.CommonErrorCode; +import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException; import org.apache.seatunnel.core.starter.command.AbstractCommandArgs; +import lombok.extern.slf4j.Slf4j; + import java.io.File; import java.nio.file.Path; import java.nio.file.Paths; +@Slf4j public class FileUtils { private FileUtils() { @@ -50,6 +55,18 @@ public static Path getConfigPath(AbstractCommandArgs args) { } } + /** + * Check whether the conf file exists. + * + * @param configFile the path of the config file + */ + public static void checkConfigExist(Path configFile) { + if (!configFile.toFile().exists()) { + String message = "Can't find config file: " + configFile; + throw new SeaTunnelRuntimeException(CommonErrorCode.FILE_OPERATION_FAILED, message); + } + } + /** * Get the file name from the given path. * e.g. seatunnel/conf/config.conf -> config.conf diff --git a/seatunnel-core/seatunnel-flink-starter/src/main/java/org/apache/seatunnel/core/starter/flink/command/FlinkApiTaskExecuteCommand.java b/seatunnel-core/seatunnel-flink-starter/src/main/java/org/apache/seatunnel/core/starter/flink/command/FlinkApiTaskExecuteCommand.java index 19395db1a67..8de18115548 100644 --- a/seatunnel-core/seatunnel-flink-starter/src/main/java/org/apache/seatunnel/core/starter/flink/command/FlinkApiTaskExecuteCommand.java +++ b/seatunnel-core/seatunnel-flink-starter/src/main/java/org/apache/seatunnel/core/starter/flink/command/FlinkApiTaskExecuteCommand.java @@ -17,6 +17,8 @@ package org.apache.seatunnel.core.starter.flink.command; +import static org.apache.seatunnel.core.starter.utils.FileUtils.checkConfigExist; + import org.apache.seatunnel.core.starter.command.Command; import org.apache.seatunnel.core.starter.config.ConfigBuilder; import org.apache.seatunnel.core.starter.exception.CommandExecuteException; @@ -46,7 +48,7 @@ public FlinkApiTaskExecuteCommand(FlinkCommandArgs flinkCommandArgs) { @Override public void execute() throws CommandExecuteException { Path configFile = FileUtils.getConfigPath(flinkCommandArgs); - + checkConfigExist(configFile); Config config = new ConfigBuilder(configFile).getConfig(); FlinkExecution seaTunnelTaskExecution = new FlinkExecution(config); try { diff --git a/seatunnel-core/seatunnel-spark-starter/src/main/java/org/apache/seatunnel/core/starter/spark/command/SparkApiTaskExecuteCommand.java b/seatunnel-core/seatunnel-spark-starter/src/main/java/org/apache/seatunnel/core/starter/spark/command/SparkApiTaskExecuteCommand.java index 5fdc4e28dfd..e3d2aa44560 100644 --- a/seatunnel-core/seatunnel-spark-starter/src/main/java/org/apache/seatunnel/core/starter/spark/command/SparkApiTaskExecuteCommand.java +++ b/seatunnel-core/seatunnel-spark-starter/src/main/java/org/apache/seatunnel/core/starter/spark/command/SparkApiTaskExecuteCommand.java @@ -17,6 +17,8 @@ package org.apache.seatunnel.core.starter.spark.command; +import static org.apache.seatunnel.core.starter.utils.FileUtils.checkConfigExist; + import org.apache.seatunnel.core.starter.command.Command; import org.apache.seatunnel.core.starter.config.ConfigBuilder; import org.apache.seatunnel.core.starter.exception.CommandExecuteException; @@ -46,6 +48,7 @@ public SparkApiTaskExecuteCommand(SparkCommandArgs sparkCommandArgs) { @Override public void execute() throws CommandExecuteException { Path configFile = FileUtils.getConfigPath(sparkCommandArgs); + checkConfigExist(configFile); Config config = new ConfigBuilder(configFile).getConfig(); try { SparkExecution seaTunnelTaskExecution = new SparkExecution(config); diff --git a/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/command/ClientExecuteCommand.java b/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/command/ClientExecuteCommand.java index 9c21d6ee4f9..237bce78aa4 100644 --- a/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/command/ClientExecuteCommand.java +++ b/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/command/ClientExecuteCommand.java @@ -17,6 +17,8 @@ package org.apache.seatunnel.core.starter.seatunnel.command; +import static org.apache.seatunnel.core.starter.utils.FileUtils.checkConfigExist; + import org.apache.seatunnel.core.starter.command.Command; import org.apache.seatunnel.core.starter.exception.CommandExecuteException; import org.apache.seatunnel.core.starter.seatunnel.args.ClientCommandArgs; @@ -73,6 +75,7 @@ public void execute() throws CommandExecuteException { System.out.println(jobState); } else { Path configFile = FileUtils.getConfigPath(clientCommandArgs); + checkConfigExist(configFile); JobConfig jobConfig = new JobConfig(); jobConfig.setName(clientCommandArgs.getJobName()); JobExecutionEnvironment jobExecutionEnv = diff --git a/seatunnel-dist/pom.xml b/seatunnel-dist/pom.xml index a1a3fbee64f..4a2c7c02887 100644 --- a/seatunnel-dist/pom.xml +++ b/seatunnel-dist/pom.xml @@ -366,6 +366,7 @@ org.apache.seatunnel connector-rabbitmq + connector-http-jira ${project.version} provided diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/pom.xml b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/pom.xml index d3fd0e8469d..ed1cf6bd81f 100644 --- a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/pom.xml +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/pom.xml @@ -68,6 +68,12 @@ ${project.version} test + + org.apache.seatunnel + connector-http-jira + ${project.version} + test + diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpJiraIT.java b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpJiraIT.java new file mode 100644 index 00000000000..94a513e171c --- /dev/null +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpJiraIT.java @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.e2e.connector.http; + +import org.apache.seatunnel.e2e.common.TestResource; +import org.apache.seatunnel.e2e.common.TestSuiteBase; +import org.apache.seatunnel.e2e.common.container.TestContainer; + +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.TestTemplate; +import org.testcontainers.containers.Container; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.output.Slf4jLogConsumer; +import org.testcontainers.containers.wait.strategy.HttpWaitStrategy; +import org.testcontainers.lifecycle.Startables; +import org.testcontainers.utility.DockerImageName; +import org.testcontainers.utility.DockerLoggerFactory; +import org.testcontainers.utility.MountableFile; + +import java.io.File; +import java.io.IOException; +import java.util.stream.Stream; + +public class HttpJiraIT extends TestSuiteBase implements TestResource { + + private static final String IMAGE = "mockserver/mockserver:5.14.0"; + + private GenericContainer mockserverContainer; + + @BeforeAll + @Override + public void startUp() { + this.mockserverContainer = new GenericContainer<>(DockerImageName.parse(IMAGE)) + .withNetwork(NETWORK) + .withNetworkAliases("mockserver") + .withExposedPorts(1080) + .withCopyFileToContainer(MountableFile.forHostPath(new File(HttpJiraIT.class.getResource( + "/mockserver-jira-config.json").getPath()).getAbsolutePath()), + "/tmp/mockserver-jira-config.json") + .withEnv("MOCKSERVER_INITIALIZATION_JSON_PATH", "/tmp/mockserver-jira-config.json") + .withLogConsumer(new Slf4jLogConsumer(DockerLoggerFactory.getLogger(IMAGE))) + .waitingFor(new HttpWaitStrategy().forPath("/").forStatusCode(404)); + Startables.deepStart(Stream.of(mockserverContainer)).join(); + } + + @AfterAll + @Override + public void tearDown() { + if (mockserverContainer != null) { + mockserverContainer.stop(); + } + } + + @TestTemplate + public void testHttpJiraSourceToAssertSink(TestContainer container) throws IOException, InterruptedException { + Container.ExecResult execResult = container.executeJob("/jira_json_to_assert.conf"); + Assertions.assertEquals(0, execResult.getExitCode()); + } +} diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/jira_json_to_assert.conf b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/jira_json_to_assert.conf new file mode 100644 index 00000000000..3be4e859870 --- /dev/null +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/jira_json_to_assert.conf @@ -0,0 +1,76 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +env { + execution.parallelism = 1 + job.mode = "BATCH" +} + +source { + Jira { + url = "http://mockserver:1080/rest/api/3/search" + email = "admin@test.com" + api_token = "token" + method = "GET" + format = "json" + schema = { + fields { + expand = string + startAt = int + maxResults = int + total = string + } + } + } +} + +sink { + Console {} + Assert { + rules { + field_rules = [ + { + field_name = expand + field_type = string + field_value = [ + { + rule_type = NOT_NULL + } + ] + }, + { + field_name = startAt + field_type = int + field_value = [ + { + rule_type = NOT_NULL + } + ] + }, + { + field_name = maxResults + field_type = int + field_value = [ + { + rule_type = NOT_NULL + } + ] + } + ] + } + } +} diff --git a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-jira-config.json b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-jira-config.json new file mode 100644 index 00000000000..0072f7f9320 --- /dev/null +++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-jira-config.json @@ -0,0 +1,20 @@ +// https://www.mock-server.com/mock_server/getting_started.html#request_matchers + +[ + { + "httpRequest": { + "method" : "GET", + "path": "/rest/api/3/search" + }, + "httpResponse": { + "body": [ + { + "expand": "schema,names", + "startAt": 0, + "maxResults": 50, + "total": 3 + } + ] + } + } +] diff --git a/seatunnel-engine/seatunnel-engine-client/src/main/java/org/apache/seatunnel/engine/client/job/JobExecutionEnvironment.java b/seatunnel-engine/seatunnel-engine-client/src/main/java/org/apache/seatunnel/engine/client/job/JobExecutionEnvironment.java index 4a04245cae3..5f9a6ffcdc8 100644 --- a/seatunnel-engine/seatunnel-engine-client/src/main/java/org/apache/seatunnel/engine/client/job/JobExecutionEnvironment.java +++ b/seatunnel-engine/seatunnel-engine-client/src/main/java/org/apache/seatunnel/engine/client/job/JobExecutionEnvironment.java @@ -19,6 +19,7 @@ import org.apache.seatunnel.api.common.JobContext; import org.apache.seatunnel.common.config.Common; +import org.apache.seatunnel.common.utils.FileUtils; import org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient; import org.apache.seatunnel.engine.common.config.JobConfig; import org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException; @@ -34,19 +35,14 @@ import org.apache.commons.lang3.tuple.ImmutablePair; import java.io.IOException; -import java.net.MalformedURLException; import java.net.URL; -import java.nio.file.FileVisitOption; import java.nio.file.Files; -import java.nio.file.Path; import java.util.ArrayList; import java.util.Collections; import java.util.HashSet; import java.util.List; import java.util.Set; import java.util.concurrent.ExecutionException; -import java.util.stream.Collectors; -import java.util.stream.Stream; public class JobExecutionEnvironment { @@ -88,16 +84,7 @@ public JobExecutionEnvironment(JobConfig jobConfig, String jobFilePath, private Set searchPluginJars() { try { if (Files.exists(Common.pluginRootDir())) { - try (Stream paths = Files.walk(Common.pluginRootDir(), FileVisitOption.FOLLOW_LINKS)) { - return paths.filter(path -> path.toString().endsWith(".jar")) - .map(path -> { - try { - return path.toUri().toURL(); - } catch (MalformedURLException e) { - throw new SeaTunnelEngineException(e); - } - }).collect(Collectors.toSet()); - } + return new HashSet<>(FileUtils.searchJarFiles(Common.pluginRootDir())); } } catch (IOException | SeaTunnelEngineException e) { LOGGER.warning(String.format("Can't search plugin jars in %s.", Common.pluginRootDir()), e); diff --git a/seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java b/seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java index b5d49e2fcaa..9a861a4e7b2 100644 --- a/seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java +++ b/seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscovery.java @@ -18,8 +18,16 @@ package org.apache.seatunnel.plugin.discovery; import org.apache.seatunnel.api.common.PluginIdentifierInterface; +import org.apache.seatunnel.api.table.factory.Factory; +import org.apache.seatunnel.api.table.factory.FactoryUtil; +import org.apache.seatunnel.api.table.factory.TableSinkFactory; +import org.apache.seatunnel.api.table.factory.TableSourceFactory; +import org.apache.seatunnel.api.table.factory.TableTransformFactory; import org.apache.seatunnel.apis.base.plugin.Plugin; import org.apache.seatunnel.common.config.Common; +import org.apache.seatunnel.common.constants.CollectionConstants; +import org.apache.seatunnel.common.constants.PluginType; +import org.apache.seatunnel.common.utils.FileUtils; import org.apache.seatunnel.common.utils.ReflectionUtils; import org.apache.seatunnel.shade.com.typesafe.config.Config; @@ -31,16 +39,20 @@ import org.apache.commons.lang3.ArrayUtils; import org.apache.commons.lang3.StringUtils; +import javax.annotation.Nonnull; import javax.annotation.Nullable; import java.io.File; import java.io.FileFilter; +import java.io.IOException; import java.net.MalformedURLException; import java.net.URL; import java.net.URLClassLoader; import java.nio.file.Path; +import java.util.ArrayList; import java.util.Collection; import java.util.Collections; +import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Optional; @@ -80,6 +92,10 @@ public AbstractPluginDiscovery(String pluginSubDir) { this(Common.connectorJarDir(pluginSubDir), loadConnectorPluginConfig()); } + public AbstractPluginDiscovery(Path pluginDir) { + this(pluginDir, loadConnectorPluginConfig()); + } + public AbstractPluginDiscovery(Path pluginDir, Config pluginConfig) { this(pluginDir, pluginConfig, DEFAULT_URL_TO_CLASSLOADER); @@ -117,6 +133,28 @@ public List getAllPlugins(List pluginIdentifiers) { .collect(Collectors.toList()); } + /** + * Get all support plugin by plugin type + * + * @param pluginType plugin type, not support transform + * @return the all plugin identifier of the engine with artifactId + */ + public static @Nonnull Map getAllSupportedPlugins(PluginType pluginType) { + Config config = loadConnectorPluginConfig(); + Map pluginIdentifiers = new HashMap<>(); + if (config.isEmpty() || !config.hasPath(CollectionConstants.SEATUNNEL_PLUGIN)) { + return pluginIdentifiers; + } + Config engineConfig = config.getConfig(CollectionConstants.SEATUNNEL_PLUGIN); + if (engineConfig.hasPath(pluginType.getType())) { + engineConfig.getConfig(pluginType.getType()).entrySet().forEach(entry -> { + pluginIdentifiers.put(PluginIdentifier.of(CollectionConstants.SEATUNNEL_PLUGIN, pluginType.getType(), entry.getKey()), + entry.getValue().unwrapped().toString()); + }); + } + return pluginIdentifiers; + } + @Override public T createPluginInstance(PluginIdentifier pluginIdentifier) { return (T) createPluginInstance(pluginIdentifier, Collections.EMPTY_LIST); @@ -160,6 +198,32 @@ public T createPluginInstance(PluginIdentifier pluginIdentifier, Collection throw new RuntimeException("Plugin " + pluginIdentifier + " not found."); } + /** + * Get all support plugin already in SEATUNNEL_HOME, only support connector-v2 + * + * @param pluginType choose which type plugin should be returned + * @return the all plugin identifier of the engine + */ + @SuppressWarnings("unchecked") + public @Nonnull List getAllPlugin(PluginType pluginType) throws IOException { + List files = FileUtils.searchJarFiles(pluginDir); + List plugins = new ArrayList<>(); + List factories; + if (pluginType.equals(PluginType.SOURCE)) { + factories = FactoryUtil.discoverFactories(new URLClassLoader(files.toArray(new URL[0])), TableSourceFactory.class); + } else if (pluginType.equals(PluginType.SINK)) { + factories = FactoryUtil.discoverFactories(new URLClassLoader(files.toArray(new URL[0])), TableSinkFactory.class); + } else if (pluginType.equals(PluginType.TRANSFORM)) { + factories = FactoryUtil.discoverFactories(new URLClassLoader(files.toArray(new URL[0])), TableTransformFactory.class); + } else { + throw new IllegalArgumentException("Unsupported plugin type: " + pluginType); + } + factories.forEach(plugin -> { + plugins.add(PluginIdentifier.of("seatunnel", pluginType.getType(), ((Factory) plugin).factoryIdentifier())); + }); + return plugins; + } + @Nullable private T loadPluginInstance(PluginIdentifier pluginIdentifier, ClassLoader classLoader) { ServiceLoader serviceLoader = ServiceLoader.load(getPluginBaseClass(), classLoader); diff --git a/seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/seatunnel/SeaTunnelTransformPluginDiscovery.java b/seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/seatunnel/SeaTunnelTransformPluginDiscovery.java index 395e857ed46..94ede43e107 100644 --- a/seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/seatunnel/SeaTunnelTransformPluginDiscovery.java +++ b/seatunnel-plugin-discovery/src/main/java/org/apache/seatunnel/plugin/discovery/seatunnel/SeaTunnelTransformPluginDiscovery.java @@ -18,12 +18,13 @@ package org.apache.seatunnel.plugin.discovery.seatunnel; import org.apache.seatunnel.api.transform.SeaTunnelTransform; +import org.apache.seatunnel.common.config.Common; import org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery; public class SeaTunnelTransformPluginDiscovery extends AbstractPluginDiscovery { public SeaTunnelTransformPluginDiscovery() { - super("seatunnel"); + super(Common.libDir()); } @Override diff --git a/seatunnel-plugin-discovery/src/test/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscoveryTest.java b/seatunnel-plugin-discovery/src/test/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscoveryTest.java new file mode 100644 index 00000000000..0b64df270e7 --- /dev/null +++ b/seatunnel-plugin-discovery/src/test/java/org/apache/seatunnel/plugin/discovery/AbstractPluginDiscoveryTest.java @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.seatunnel.plugin.discovery; + +import org.apache.seatunnel.common.config.Common; +import org.apache.seatunnel.common.config.DeployMode; +import org.apache.seatunnel.common.constants.PluginType; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.DisabledOnOs; +import org.junit.jupiter.api.condition.OS; + +import java.util.Map; +import java.util.Objects; + +@DisabledOnOs(OS.WINDOWS) +public class AbstractPluginDiscoveryTest { + + @Test + public void testGetAllPlugins() { + Common.setDeployMode(DeployMode.CLIENT); + System.setProperty("SEATUNNEL_HOME", Objects.requireNonNull(AbstractPluginDiscoveryTest.class.getResource("/home")).getPath()); + Map sourcePlugins = AbstractPluginDiscovery.getAllSupportedPlugins(PluginType.SOURCE); + Assertions.assertEquals(27, sourcePlugins.size()); + + Map sinkPlugins = AbstractPluginDiscovery.getAllSupportedPlugins(PluginType.SINK); + Assertions.assertEquals(30, sinkPlugins.size()); + + } + +} diff --git a/seatunnel-core/seatunnel-core-base/src/test/resources/plugin-mapping.properties b/seatunnel-plugin-discovery/src/test/resources/home/connectors/plugin-mapping.properties similarity index 60% rename from seatunnel-core/seatunnel-core-base/src/test/resources/plugin-mapping.properties rename to seatunnel-plugin-discovery/src/test/resources/home/connectors/plugin-mapping.properties index 619911fe942..b3f9707aaf5 100644 --- a/seatunnel-core/seatunnel-core-base/src/test/resources/plugin-mapping.properties +++ b/seatunnel-plugin-discovery/src/test/resources/home/connectors/plugin-mapping.properties @@ -41,11 +41,13 @@ flink.sink.FileSink = seatunnel-connector-flink-file flink.sink.InfluxDbSink = seatunnel-connector-flink-influxdb flink.sink.JdbcSink = seatunnel-connector-flink-jdbc flink.sink.Kafka = seatunnel-connector-flink-kafka +flink.sink.AssertSink = seatunnel-connector-flink-assert # Spark Source spark.source.ElasticSearch = seatunnel-connector-spark-elasticsearch spark.source.Fake = seatunnel-connector-spark-fake +spark.source.FakeStream = seatunnel-connector-spark-fake spark.source.FeishuSheet = seatunnel-connector-spark-feishu spark.source.File = seatunnel-connector-spark-file spark.source.Hbase = seatunnel-connector-spark-hbase @@ -83,3 +85,63 @@ spark.sink.MongoDB = seatunnel-connector-spark-mongodb spark.sink.Phoenix = seatunnel-connector-spark-phoenix spark.sink.Redis = seatunnel-connector-spark-redis spark.sink.TiDB = seatunnel-connector-spark-tidb + +# SeaTunnel new connector API + +seatunnel.source.FakeSource = connector-fake +seatunnel.sink.Console = connector-console +seatunnel.sink.Assert = connector-assert +seatunnel.source.Kafka = connector-kafka +seatunnel.sink.Kafka = connector-kafka +seatunnel.source.Http = connector-http-base +seatunnel.sink.Http = connector-http-base +seatunnel.sink.Feishu = connector-http-feishu +seatunnel.source.Socket = connector-socket +seatunnel.sink.Hive = connector-hive +seatunnel.source.Hive = connector-hive +seatunnel.source.Clickhouse = connector-clickhouse +seatunnel.sink.Clickhouse = connector-clickhouse +seatunnel.sink.ClickhouseFile = connector-clickhouse +seatunnel.source.Jdbc = connector-jdbc +seatunnel.sink.Jdbc = connector-jdbc +seatunnel.source.Kudu = connector-kudu +seatunnel.sink.Kudu = connector-kudu +seatunnel.sink.Email = connector-email +seatunnel.source.HdfsFile = connector-file-hadoop +seatunnel.sink.HdfsFile = connector-file-hadoop +seatunnel.source.LocalFile = connector-file-local +seatunnel.sink.LocalFile = connector-file-local +seatunnel.source.OssFile = connector-file-oss +seatunnel.sink.OssFile = connector-file-oss +seatunnel.source.Pulsar = connector-pulsar +seatunnel.source.Hudi = connector-hudi +seatunnel.sink.DingTalk = connector-dingtalk +seatunnel.source.Elasticsearch = connector-elasticsearch +seatunnel.sink.Elasticsearch = connector-elasticsearch +seatunnel.source.IoTDB = connector-iotdb +seatunnel.sink.IoTDB = connector-iotdb +seatunnel.source.Neo4j = connector-neo4j +seatunnel.sink.Neo4j = connector-neo4j +seatunnel.source.FtpFile = connector-file-ftp +seatunnel.sink.FtpFile = connector-file-ftp +seatunnel.source.SftpFile = connector-file-sftp +seatunnel.sink.SftpFile = connector-file-sftp +seatunnel.sink.Socket = connector-socket +seatunnel.source.Redis = connector-redis +seatunnel.sink.Redis = connector-redis +seatunnel.sink.DataHub = connector-datahub +seatunnel.sink.Sentry = connector-sentry +seatunnel.source.MongoDB = connector-mongodb +seatunnel.sink.MongoDB = connector-mongodb +seatunnel.source.Iceberg = connector-iceberg +seatunnel.source.InfluxDB = connector-influxdb +seatunnel.source.S3File = connector-file-s3 +seatunnel.sink.S3File = connector-file-s3 +seatunnel.source.AmazonDynamodb = connector-amazondynamodb +seatunnel.sink.AmazonDynamodb = connector-amazondynamodb +seatunnel.source.Cassandra = connector-cassandra +seatunnel.sink.Cassandra = connector-cassandra +seatunnel.sink.StarRocks = connector-starrocks +seatunnel.source.MyHours = connector-http-myhours +seatunnel.sink.InfluxDB = connector-influxdb +seatunnel.source.GoogleSheets = connector-google-sheets