Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INLONG-5043][Manager] Add Apache Doris load node management #5989

Closed
wants to merge 4 commits into from

Conversation

lucaspeng12138
Copy link
Contributor

Prepare a Pull Request

Motivation

To surport the ability to Doris data integration, we need to add Apache Doris Load Node for management

Design

The design mainly follows the document Manager Plugin

  1. Add corresponding SinkType enumeration in enumeration type org.apache.inlong.manager.common.Enums.
  2. In org.apache.inlong.manager.common.Pojo.Sink,create folder path,create the corresponding entity class.
  3. In the org.Apache.Inlong.Manager.Service.Sink path, created under the corresponding tools
  4. Support data source to LoadNode conversion function, reference code org.Apache. Inlong.Manager.Service.Sort.Util.LoadNodeUtils

Implementation

  1. Add DORIS in enumeration type org.apache.inlong.manager.common.enums.SinkType
  2. Create folder "Doris" in org.apache.inlong.manager.common.pojo.sink, and create corresponding entity class:
  • DorisSink
  • DorisSinkDTO
  • DorisSinkRequest
  • DorisColumnInfo
  • DorisTableInfo
  1. Create folder "doris" in org.apache.inlong.manager.service.sink and implement the class:
  • DorisSinkOperator
  1. Add createLoadNode function in org.apache.inlong.manager.service.sort.util.LoadNodeUtils, it is like as follows:
    public static DorisLoadNode createLoadNode(DorisSink dorisSink, List<FieldInfo> fieldInfos, List<FieldRelation> fieldRelations, Map<String, String> properties){ \\TODO }

@lucaspeng12138 lucaspeng12138 marked this pull request as draft September 22, 2022 06:56
@healchow healchow marked this pull request as ready for review September 27, 2022 02:28
@EMsnap
Copy link
Contributor

EMsnap commented Sep 27, 2022

there may be a few changes on sink parameter since we are optimizing the doris connector, so I suggest hold for now

@dockerzhang
Copy link
Contributor

duplicated with #6233, close it.

@lucaspeng12138 lucaspeng12138 deleted the dev-lucas-lca branch October 31, 2022 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][Manager] Add Apache Doris load node management
5 participants