Skip to content

Commit

Permalink
[INLONG-566][Doc] Update the Introduction (apache#567)
Browse files Browse the repository at this point in the history
  • Loading branch information
dockerzhang authored Oct 14, 2022
1 parent b44b35b commit 05def02
Show file tree
Hide file tree
Showing 3 changed files with 10 additions and 10 deletions.
8 changes: 4 additions & 4 deletions docs/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ values={[
## Modules
Apache InLong serves the entire life cycle from data collection to landing, and provides different processing modules according to different stages of data, including the next modules:

- **inlong-agent**, data collection agent, supports reading regular logs from specified directories or files and reporting data one by one. In the future, DB collection capabilities will also be expanded.
- **inlong-agent**, data collection services, including file collection, DB collection, etc.
- **inlong-dataproxy**, a Proxy component based on Flume-ng, supports data transmission blocking, placing retransmission, and has the ability to forward received data to different MQ (message queues).
- **inlong-tubemq**, Tencent's self-developed message queuing service, focuses on high-performance storage and transmission of massive data in big data scenarios and has a relatively good core advantage in mass practice and low cost.
- **inlong-sort**, after consuming data from different MQ services, perform ETL processing, and then aggregate and write the data into Apache Hive, ClickHouse, Hbase, IceBerg, etc.
Expand All @@ -71,11 +71,11 @@ Apache InLong serves the entire life cycle from data collection to landing, and
| | File | None | Standard |
| | Kafka | 2.x | Lightweight, Standard |
| | MySQL | 5.6, 5.7, 8.0.x | Lightweight, Standard |
| | MongoDB | >= 3.6 | Lightweight |
| | MongoDB | >= 3.6 | Lightweight, Standard |
| | Oracle | 11,12,19 | Lightweight |
| | PostgreSQL | 9.6, 10, 11, 12 | Lightweight |
| | PostgreSQL | 9.6, 10, 11, 12 | Lightweight, Standard |
| | Pulsar | 2.8.x | Lightweight |
| | SQLServer | 2012, 2014, 2016, 2017, 2019 | Lightweight |
| | SQLServer | 2012, 2014, 2016, 2017, 2019 | Lightweight, Standard |
| Load Node | Auto Consumption | None | Standard |
| | Hive | 1.x, 2.x, 3.x | Lightweight, Standard |
| | Iceberg | 0.12.x | Lightweight, Standard |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ sidebar_position: 1
| Group | 数据流组,包含多个数据流,一个Group 代表一个数据接入 | Group 有ID、Name 等属性 |
| Stream | 数据流,一个数据流有具体的流向 | Stream 有ID、Name、数据字段等属性 |
| Node | 数据节点,包括`Extract Node``Load Node`,分别代表数据源类型和数据流向目标类型 | |
| InLongMsg | InLong 数据格式,如果从消息队列中直接消费,需要先进行`InLongMsg` 解析 | |
| InLongMsg | InLong 数据格式,如果从消息队列中直接消费,需要先进行 `InLongMsg` 解析 | |
| Agent | 标准架构使用 Agent 进行数据采集,Agent 代表不同类型的采集能力 | 包含文件 Agent、SQL Agent、Binlog Agent 等 |
| DataProxy | 将接收到的数据转发到不同的消息队列 | 支持数据发送阻塞和落盘重发 |
| Sort | 数据流分拣 | 主要有基于 Flink 的 sort-flink,sort-standalone 本地分拣 |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,13 +53,13 @@ values={[
## 模块
Apache InLong 服务于数据采集到落地的整个生命周期,按数据的不同阶段提供不同的处理模块,主要包括:

- **inlong-agent**数据采集 Agent,支持从指定目录或文件读取常规日志、逐条上报。后续也将扩展 DB 采集等能力
- **inlong-agent**数据采集服务,包括文件采集、DB采集等
- **inlong-dataproxy**,一个基于 Flume-ng 的 Proxy 组件,支持数据发送阻塞和落盘重发,拥有将接收到的数据转发到不同 MQ(消息队列)的能力。
- **inlong-tubemq**,腾讯自研的消息队列服务,专注于大数据场景下海量数据的高性能存储和传输,在海量实践和低成本方面有着良好的核心优势。
- **inlong-sort**,对从不同的 MQ 消费到的数据进行 ETL 处理,然后汇聚并写入 Hive、ClickHouse、Hbase、Iceberg 等存储系统。
- **inlong-manager**,提供完整的数据服务管控能力,包括元数据、任务流、权限,OpenAPI 等。
- **inlong-dashboard**,用于管理数据接入的前端页面,简化整个 InLong 管控平台的使用。
- **inlong-audit**对InLong系统的Agent、DataProxy、Sort模块的入流量、出流量进行实时审计对账。
- **inlong-audit**对 InLong 系统的 Agent、DataProxy、Sort 模块的入流量、出流量进行实时审计对账。

## 已支持数据节点(更新中)
| Type | Name | Version | Architecture |
Expand All @@ -68,11 +68,11 @@ Apache InLong 服务于数据采集到落地的整个生命周期,按数据的
| | File | None | Standard |
| | Kafka | 2.x | Lightweight, Standard |
| | MySQL | 5.6, 5.7, 8.0.x | Lightweight, Standard |
| | MongoDB | >= 3.6 | Lightweight |
| | MongoDB | >= 3.6 | Lightweight, Standard |
| | Oracle | 11,12,19 | Lightweight |
| | PostgreSQL | 9.6, 10, 11, 12 | Lightweight |
| | PostgreSQL | 9.6, 10, 11, 12 | Lightweight, Standard |
| | Pulsar | 2.8.x | Lightweight |
| | SQLServer | 2012, 2014, 2016, 2017, 2019 | Lightweight |
| | SQLServer | 2012, 2014, 2016, 2017, 2019 | Lightweight, Standard |
| Load Node | Auto Consumption | None | Standard |
| | Hive | 1.x, 2.x, 3.x | Lightweight, Standard |
| | Iceberg | 0.12.x | Lightweight, Standard |
Expand Down

0 comments on commit 05def02

Please sign in to comment.