Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Agent] Support collects data from the Apache Doris #5042

Closed
1 of 2 tasks
Tracked by #5044
dockerzhang opened this issue Jul 14, 2022 · 4 comments
Closed
1 of 2 tasks
Tracked by #5044

[Feature][Agent] Support collects data from the Apache Doris #5042

dockerzhang opened this issue Jul 14, 2022 · 4 comments

Comments

@dockerzhang
Copy link
Contributor

Description

you can refer to:
https://inlong.apache.org/docs/next/design_and_concept/how_to_write_plugin_agent

Use case

No response

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

@Loveca
Copy link

Loveca commented Jul 23, 2022

Hi, I want to do this task.

@dockerzhang
Copy link
Contributor Author

@Loveca assigned to you.

@Loveca
Copy link

Loveca commented Jul 28, 2022

Motivition

Add Apache Doris Extract Node for Agent

About Doris Data Export

Data Export is a function provided by Doris to Export data. This function exports data in a table or partition specified by users to a remote storage system, such as HDFS/BOS, in text format through the Broker process. You can also export it locally.

Export To HDFS

EXPORT TABLE db1.tbl1 
PARTITION (p1,p2)
[WHERE [expr]]
TO "hdfs://host/path/to/export/" 
PROPERTIES
(
    "label" = "mylabel",
    "column_separator"=",",
    "columns" = "col1,col2",
    "exec_mem_limit"="2147483648",
    "timeout" = "3600"
)
WITH BROKER "hdfs"
(
    "username" = "user",
    "password" = "passwd"
);

Export To Local

EXPORT TABEL tablename TO "file:///local_file_path"

Design

image

1.Doris exports the data to the HDFS data file or local data file through Export command
2. The InLong Agent reads the corresponding data file

Implementation

  • Reader: Implements DorisReader to read the data files exported by Doris
  • Source: Implements DorisSource, implements Split logic, and returns Reader list
  • Sink: Use ProxySink

@dockerzhang
Copy link
Contributor Author

In most usage scenarios, Doris is more used for storage than collection. So this issue is temporarily closed

@healchow healchow changed the title [Feature] Add Apache Doris Extract Node for Agent [Feature][Agent] Support collects data from the Apache Doris Nov 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants