Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Manager] Add Apache Doris load node management #5043

Closed
1 of 2 tasks
Tracked by #5044
dockerzhang opened this issue Jul 14, 2022 · 4 comments
Closed
1 of 2 tasks
Tracked by #5044

[Feature][Manager] Add Apache Doris load node management #5043

dockerzhang opened this issue Jul 14, 2022 · 4 comments
Assignees
Milestone

Comments

@dockerzhang
Copy link
Contributor

dockerzhang commented Jul 14, 2022

Description

Add Apache Doris Node Management, you can refer to:
https://inlong.apache.org/docs/next/design_and_concept/how_to_extend_data_node_for_manager

Dashboard:
https://inlong.apache.org/docs/next/design_and_concept/how_to_write_plugin_dashboard

Use case

No response

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

@dockerzhang dockerzhang added the good first issue Good for newcomers label Jul 14, 2022
@dockerzhang dockerzhang changed the title [Feature] Add Apache Doris Node Management [Feature] Add Apache Doris Load Node Management Jul 29, 2022
@Loveca
Copy link

Loveca commented Jul 29, 2022

Motivation

To surport the ability to Doris data integration, we need to add Apache Doris Load Node for management

Design

The design mainly follows the document Manager Plugin

  1. Add corresponding SinkType enumeration in enumeration type org.apache.inlong.manager.common.Enums.
  2. In org.apache.inlong.manager.common.Pojo.Sink,create folder path,create the corresponding entity class.
  3. In the org.Apache.Inlong.Manager.Service.Sink path, created under the corresponding tools
  4. Support data source to LoadNode conversion function, reference code org.Apache. Inlong.Manager.Service.Sort.Util.LoadNodeUtils

Implementation

  1. Add DORIS in enumeration type org.apache.inlong.manager.common.enums.SinkType
  2. Create folder "Doris" in org.apache.inlong.manager.common.pojo.sink, and create corresponding entity class:
  • DorisSink
  • DorisSinkDTO
  • DorisSinkRequest
  • DorisColumnInfo
  • DorisTableInfo
  1. Create folder "doris" in org.apache.inlong.manager.service.sink and implement the class:
  • DorisSinkOperator
  1. Add createLoadNode function in org.apache.inlong.manager.service.sort.util.LoadNodeUtils, it is like as follows:
public static DorisLoadNode createLoadNode(DorisSink dorisSink, List<FieldInfo> fieldInfos,
            List<FieldRelation> fieldRelations, Map<String, String> properties){
                 \\TODO 
      }

@Loveca
Copy link

Loveca commented Jul 29, 2022

@dockerzhang hi, can you assign this to me?

@dockerzhang
Copy link
Contributor Author

@Loveca LGTM

@healchow healchow changed the title [Feature] Add Apache Doris Load Node Management [Feature][Manager] Add Apache Doris load node management Aug 16, 2022
@healchow healchow added this to the 1.4.0 milestone Sep 27, 2022
@healchow
Copy link
Member

Duplicate with #6232, close this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment