[INLONG-2161][Feature] Manager support getClusterConfig #2182

vernedeng · 2022-01-18T08:49:45Z

Title Name: [INLONG-2161][Feature] Manager support getClusterConfig

Motivation

getClusterConfig is the interface that Sort Stand-alone acquire id params and sink params from.
Id params is the params of upstream data store such as the topic of kafka and pulsar which Sort Stand-alone consume from.
While sink params is the dispatch params of Sort Stand-alone Sink, which determine which kafka , pulsar or hive cluster the produced data send to. hence it's something like kafka zklist, brokerlist, pulsar serviceUrl, token..

Here is the example that the config be like.

Modifications

Describe the modifications you've done.

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? JavaDocs
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

… params

…of kafka

…est format

gosonzhang · 2022-01-18T12:22:27Z

@ImVan, Why are there so many code submissions?

From the feature description, this modified code should be very small, is there no rebase when submitting the code?

Please rebase the code first, and then submit it, thanks!

codecov-commenter · 2022-01-18T13:11:01Z

Codecov Report

Merging #2182 (e91c026) into master (58a3f51) will increase coverage by 0.33%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #2182      +/-   ##
============================================
+ Coverage     12.22%   12.55%   +0.33%     
- Complexity     1154     1201      +47     
============================================
  Files           413      417       +4     
  Lines         35243    35735     +492     
  Branches       5544     5590      +46     
============================================
+ Hits           4308     4487     +179     
- Misses        30173    30448     +275     
- Partials        762      800      +38

Impacted Files	Coverage Δ
...inlong/tubemq/manager/service/TaskServiceImpl.java	`3.65% <0.00%> (-4.88%)`	⬇️
.../java/org/apache/flume/sink/tubemq/TubemqSink.java	`51.42% <0.00%> (-4.00%)`	⬇️
...ong/tubemq/manager/service/ClusterServiceImpl.java	`50.00% <0.00%> (-2.95%)`	⬇️
...ng/tubemq/server/master/metrics/MasterMetrics.java	`90.00% <0.00%> (-2.50%)`	⬇️
...emq/server/broker/metrics/BrokerMetricsHolder.java	`69.69% <0.00%> (-1.74%)`	⬇️
.../tubemq/server/common/utils/WebParameterUtils.java	`17.94% <0.00%> (-0.75%)`	⬇️
...ng/tubemq/server/broker/metrics/BrokerMetrics.java	`97.29% <0.00%> (-0.75%)`	⬇️
.../tubemq/client/producer/SimpleMessageProducer.java	`12.50% <0.00%> (-0.57%)`	⬇️
...emq/server/master/metrics/MasterMetricsHolder.java	`86.30% <0.00%> (-0.55%)`	⬇️
.../producer/qltystats/DefaultBrokerRcvQltyStats.java	`44.14% <0.00%> (-0.40%)`	⬇️
... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 58a3f51...e91c026. Read the comment docs.

vernedeng · 2022-01-18T13:15:39Z

@gosonzhang
It's a new interface that Sort Stand-alone acquire cluster config including id params and sink params.
Id params is the params of upstream data store such as the topic of kafka and pulsar which Sort Stand-alone consume from.
While sink params is the dispatch params of Sort Stand-alone Sink, which determine which kafka , pulsar or hive cluster the produced data send to. hence it's something like kafka zklist, brokerlist, pulsar serviceUrl, token.
Due to the diversity of upstream data store type and downstrem sink type, there must be many tables to maintain those params of different type of sources and sinks.
Most of files are auto generated by Mybatis code generator, such as those xxxMapper.java and xxxMapper.xml
What we need to care about are those xxxxServiceImpl.java , espacially SortClusterConfigServiceImpl.java which is the core logic that how to generate a valid ClusterConfig

wardlican · 2022-01-19T01:44:37Z

"result: true" and "errorCode" are not reasonable in the protocol. It is suggested to change them to: "msg" and "code". use return code to judge whether it is successful or not.

wardlican · 2022-01-19T02:07:25Z

inlong-manager/manager-dao/src/main/resources/mappers/TaskIdParamsKafkaEntityMapper.xml

+    where id = #{id,jdbcType=INTEGER}
+  </delete>
+  <insert id="insert" parameterType="org.apache.inlong.manager.dao.entity.TaskIdParamsKafkaEntity">
+    insert into task_id_params_kafka (id, parent_name, topic


need to sort out the format here

wardlican · 2022-01-19T02:08:12Z

inlong-manager/manager-dao/src/main/resources/mappers/TaskIdParamsPulsarEntityMapper.xml

+    where id = #{id,jdbcType=INTEGER}
+  </delete>
+  <insert id="insert" parameterType="org.apache.inlong.manager.dao.entity.TaskIdParamsPulsarEntity">
+    insert into task_id_params_pulsar (id, parent_name, topic


…st result

vernedeng · 2022-01-19T06:46:05Z

"result: true" and "errorCode" are not reasonable in the protocol. It is suggested to change them to: "msg" and "code". use return code to judge whether it is successful or not.

Thx @wardlican , very nice suggestion. The fields "result" and "errCode" has been replaced with "msg" and "code"

luchunliang · 2022-01-19T01:39:22Z

...mmon/src/main/java/org/apache/inlong/manager/common/pojo/sort/SortClusterConfigResponse.java

+import java.util.Map;
+
+@Data
+@ApiModel("sort cluster config")


Avoid to use lombok plugin for readability. Coder need to add getter/setter code when coder want to find usages. Coder want to check source code, then coder must to install lombok plugin.

luchunliang · 2022-01-19T01:42:04Z

...main/java/org/apache/inlong/manager/common/pojo/sort/SortStandAloneClusterConfigRequest.java

+@ApiModel("Sort-StandAlone cluster config request")
+public class SortStandAloneClusterConfigRequest {
+
+    @ApiModelProperty(value = "cluster name")


luchunliang · 2022-01-19T01:42:44Z

...anager/manager-dao/src/main/java/org/apache/inlong/manager/dao/entity/SortClusterConfig.java

+
+    private String taskName;
+
+    private static final long serialVersionUID = 1L;


incorrect serialVersionUID

static variable is before member variable.

luchunliang · 2022-01-19T01:43:43Z

...manager/manager-dao/src/main/java/org/apache/inlong/manager/dao/entity/TaskConfigEntity.java

+
+    private String inlongStreamId;
+
+    private static final long serialVersionUID = 1L;


luchunliang · 2022-01-19T01:43:53Z

.../manager-dao/src/main/java/org/apache/inlong/manager/dao/entity/TaskIdParamsKafkaEntity.java

+
+    private String topic;
+
+    private static final long serialVersionUID = 1L;


luchunliang · 2022-01-19T01:44:13Z

...anager-dao/src/main/java/org/apache/inlong/manager/dao/entity/TaskSinkParamsKafkaEntity.java

+
+    private String brokerList;
+
+    private static final long serialVersionUID = 1L;


serialVersionUID can be generated by IDE.

luchunliang · 2022-01-19T01:49:30Z

inlong-manager/manager-dao/src/main/resources/generatorConfig.xml

+               enableUpdateByPrimaryKey="true"
+               enableDeleteByPrimaryKey="true" enableInsert="true"
+               enableCountByExample="false" enableDeleteByExample="false"
+               enableSelectByExample="false" enableUpdateByExample="false"/>-->


Invalid code need to be removed.

luchunliang · 2022-01-19T01:54:55Z

.../src/main/java/org/apache/inlong/manager/service/core/impl/SortClusterConfigServiceImpl.java

+        }
+
+        // if md5 is the same as last request, return RESP_CODE_NO_UPDATE
+        String jsonClusterConfig = gson.toJson(clusterConfig);


gson can not sort parameter name, then md5 maybe not be a stable value.

vernedeng added 24 commits January 15, 2022 21:43

[INLONG-2161][Feature][Sort] Defined the tables of id params and sink…

6ea3419

… params

[INLONG-2161][Feature][Sort] Generate entity and mapper of task config

bb75f79

[INLONG-2161][Feature][Sort] Generate entity and mapper of id params …

8a8cd92

…of kafka

[INLONG-2161][Feature][Sort] Generate entity and mapper

6619204

[INLONG-2161][Feature][Sort] Add sort stand alone cluster config requ…

a07f5a7

…est format

[INLONG-2161][Feature][Sort] Add license

97adcaf

[INLONG-2161][Feature][Sort] Add annotation

2b89b24

[INLONG-2161][Feature][Sort] Remove duplicate mapper

75722fc

[INLONG-2161][Feature][Sort] Add TaskIdParamsKafkaService

36ea93e

[INLONG-2161][Feature][Sort] Add TaskIdParamsPulsarServicce

7264747

[INLONG-2161][Feature][Sort] Add TaskSinkParamsKafkaService

eeb2fb0

[INLONG-2161][Feature][Sort] Add TaskSinkParamsPulsarService

3773acd

[INLONG-2161][Feature][Sort] Update es sink params

6813566

[INLONG-2161][Feature][Sort] Add TaskSinkParamsEsService

a03bc83

[INLONG-2161][Feature][Sort] Fix could not found bean error

fb5a213

[INLONG-2161][Feature][Sort] Add sort cluster config table

803bc62

[INLONG-2161][Feature][Sort] Fix sort task mappers

9c7a79a

[INLONG-2161][Feature][Sort] Add license to mapper

ec00af4

[INLONG-2161][Feature][Sort] Add SortClusterConfigService

949c6e1

[INLONG-2161][Feature][Sort] Change dao response as map format

82034c6

[INLONG-2161][Feature][Sort] Add controller of cluster config

82bc530

[INLONG-2161][Feature] Use response wrapper

0341fbb

[INLONG-2161][Feature][Sort] Fix type problem in mapper.xml

a1862de

[INLONG-2161][Feature][Manager] Fix jackson serialization problem

ee460f7

github-actions bot added the component/manager label Jan 18, 2022

vernedeng added 2 commits January 18, 2022 17:12

[INLONG-2161][Feature][Manager] Fix PR Problem

6d9cfad

[INLONG-2161][Feature][Manager] Fix PR problem

c60e742

dockerzhang changed the title ~~Feature/inlong 2161 Manager support getClusterConfig~~ [INLONG-2161][Feature] Manager support getClusterConfig Jan 18, 2022

[INLONG-2161][Feature][Manager] Fix PR problem

d6b2a1e

vernedeng closed this Jan 18, 2022

vernedeng reopened this Jan 18, 2022

wardlican reviewed Jan 19, 2022

View reviewed changes

[INLONG-2161][Feature][Manager] Use "code" and "msg" to present reque…

c6999c6

…st result

[INLONG-2161][Feature][Manager] Fix check style problem

e91c026

luchunliang approved these changes Jan 19, 2022

View reviewed changes

wardlican approved these changes Jan 19, 2022

View reviewed changes

luchunliang approved these changes Jan 19, 2022

View reviewed changes

vernedeng closed this Jan 21, 2022

vernedeng deleted the feature/INLONG-2161 branch January 23, 2022 07:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[INLONG-2161][Feature] Manager support getClusterConfig #2182

[INLONG-2161][Feature] Manager support getClusterConfig #2182

vernedeng commented Jan 18, 2022 •

edited

Loading

gosonzhang commented Jan 18, 2022

codecov-commenter commented Jan 18, 2022 •

edited

Loading

vernedeng commented Jan 18, 2022 •

edited

Loading

wardlican commented Jan 19, 2022 •

edited

Loading

wardlican Jan 19, 2022

wardlican Jan 19, 2022

vernedeng commented Jan 19, 2022

luchunliang Jan 19, 2022

luchunliang Jan 19, 2022

luchunliang Jan 19, 2022

luchunliang Jan 19, 2022

luchunliang Jan 19, 2022

luchunliang Jan 19, 2022

luchunliang Jan 19, 2022

luchunliang Jan 19, 2022

luchunliang Jan 19, 2022


		private String taskName;

		private static final long serialVersionUID = 1L;


		private String inlongStreamId;

		private static final long serialVersionUID = 1L;


		private String topic;

		private static final long serialVersionUID = 1L;


		private String brokerList;

		private static final long serialVersionUID = 1L;

[INLONG-2161][Feature] Manager support getClusterConfig #2182

[INLONG-2161][Feature] Manager support getClusterConfig #2182

Conversation

vernedeng commented Jan 18, 2022 • edited Loading

Title Name: [INLONG-2161][Feature] Manager support getClusterConfig

Motivation

Modifications

Verifying this change

Documentation

gosonzhang commented Jan 18, 2022

codecov-commenter commented Jan 18, 2022 • edited Loading

Codecov Report

vernedeng commented Jan 18, 2022 • edited Loading

wardlican commented Jan 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vernedeng commented Jan 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vernedeng commented Jan 18, 2022 •

edited

Loading

codecov-commenter commented Jan 18, 2022 •

edited

Loading

vernedeng commented Jan 18, 2022 •

edited

Loading

wardlican commented Jan 19, 2022 •

edited

Loading