Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[INLONG-437][Doc] Update the Agent Documents for release 1.2.0 #438

Merged
merged 4 commits into from
Jun 21, 2022
Merged

[INLONG-437][Doc] Update the Agent Documents for release 1.2.0 #438

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c7356b8
[INLONG-437][Doc] Update the Agent Documents for release 1.2.0
dockerzhang Jun 21, 2022
9aa601d
[INLONG-437][Doc] Update the Agent Documents for release 1.2.0
dockerzhang Jun 21, 2022
6c9a1fc
[INLONG-437][Doc] Update the Agent Documents for release 1.2.0
dockerzhang Jun 21, 2022
e17d602
[INLONG-437][Doc] Update the Agent Documents for release 1.2.0
dockerzhang Jun 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions docs/design_and_concept/basic_concept.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,16 @@ title: Basic Concept
sidebar_position: 1
---

| Name | Description | Other |
|--------------------------|-----------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|
| Standard Architecture | Contains all InLong components such as InLong Agent/Manager/MQ/Sort/Dashboard | Suitable for massive data and large-scale production environments |
| Lightweight Architecture | Contains only one component of InLong Sort, which can be used with Manager/Dashboard | The lightweight architecture is simple and flexible, suitable for small-scale data |
| Group | Data Streams Group, it contains multiple data streams, and one Group represents one data ingestion. | Group has attributes such as ID and Name. |
| Stream | Data Stream, a stream has a specific flow direction. | Stream has attributes such as ID, Name, and data fields. |
| Node | Data Node, including `Extract Node` and `Load Node`, stands for the data source and sink types separately. | |
| InLongMsg | InLong data format, if you consume message directly from the message queue, you need to perform `InLongMsg` parsing first. | |
| Agent | Represents various collection capabilities. | It contains File Agent, SQL Agent, Binlog Agent, etc. |
| DataProxy | Forward received data to different message queues. | Supports data transmission blocking, placing retransmission. |
| Sort | Data stream sorting | Sort-flink based on Flink, sort-standalone for local sorting. |
| TubeMQ | InLong's self-developed message queuing service | It can also be called Tube, with low-cost, high-performance features. |
| Pulsar | [Apache Pulsar](https://pulsar.apache.org/), a high-performance, high-consistency message queue service | |
| Name | Description | Other |
|--------------------------|----------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|
| Standard Architecture | Contains all InLong components such as InLong Agent/Manager/MQ/Sort/Dashboard | Suitable for massive data and large-scale production environments |
| Lightweight Architecture | Contains only one component of InLong Sort, which can be used with Manager/Dashboard | The lightweight architecture is simple and flexible, suitable for small-scale data |
| Group | Data Streams Group, it contains multiple data streams, and one Group represents one data ingestion. | Group has attributes such as ID and Name. |
| Stream | Data Stream, a stream has a specific flow direction. | Stream has attributes such as ID, Name, and data fields. |
| Node | Data Node, including `Extract Node` and `Load Node`, stands for the data source and sink types separately. | |
| InLongMsg | InLong data format, if you consume message directly from the message queue, you need to perform `InLongMsg` parsing first. | |
| Agent | The standard architecture uses Agent for data collection, and Agent represents different types of collection capabilities. | It contains File Agent, SQL Agent, Binlog Agent, etc. |
| DataProxy | Forward received data to different message queues. | Supports data transmission blocking, placing retransmission. |
| Sort | Data stream sorting. | Sort-flink based on Flink, sort-standalone for local sorting. |
| TubeMQ | InLong's self-developed message queuing service | It can also be called Tube, with low-cost, high-performance features. |
| Pulsar | [Apache Pulsar](https://pulsar.apache.org/), a high-performance, high-consistency message queue service | |
59 changes: 21 additions & 38 deletions docs/design_and_concept/how_to_write_plugin_agent.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@ title: Agent Plugin
sidebar_position: 2
---

# Overview
## Overview

This article is aimed at InLong-Agent plug-in developers, trying to explain the process of developing an Agent plug-in as comprehensively as possible, and strive to eliminate the confusion of developers and make plug-in development easier.
In Standard Architecture, we can collect various types of data through InLong Agent. InLong Agent supports extending new collection types in the form of plug-ins. This article will guide developers on customizing the new Agent collection plug-ins.

## Before Development
## Concepts and Models

InLong Agent itself, as a data collection framework, is constructed with a Job + Task architecture. And abstract data source reading and writing into Reader/Sink plug-ins, which are incorporated into the entire framework.
InLong Agent is a data collection framework, adopted `Job` + `Task` architectural model. And abstract data source reading and writing into Reader/Sink plugins.

Developers need to be clear about the concepts of Job and Task:

Expand All @@ -18,37 +18,33 @@ Developers need to be clear about the concepts of Job and Task:

A Task contains the following components:

- Reader: Reader is a data collection module, which is responsible for collecting data from the data source and sending the data to the channel.
- Sink: Sink is a data writing module, responsible for continuously fetching data from the channel and writing the data to the destination.
- Channel: Channel is used to connect reader and sink, as a data transmission channel for both, and plays a role in monitoring data writing and reading
- Reader: a data collection module, which is responsible for collecting data from the data source and sending the data to the Channel.
- Sink: a data writing module, responsible for continuously fetching data from the Channel and writing the data to the destination.
- Channel: connect Reader and Sink, as a data transmission channel for both, and plays a role in monitoring data writing and reading.

As a developer, you only need to develop specific Source, Reader and Sink. If the data needs to be persisted to the local disk, use the persistent Channel, otherwise use the memory Channel
When extending an Agent plugin, you need to develop specific Source, Reader and Sink. If the data needs to be persisted to the local disk, use the persistent Channel, otherwise use the memory Channel

## Demonstration

The Job\Task\Reader\Sink\Channel concept introduced above can be represented by the following figure:
The Job/Task/Reader/Sink/Channel concept introduced above can be represented by the following figure:
![](img/Agent_Flow.png)

1. The user submits a Job (via the manager or via curl), and the Job defines the Source, Channel, and Sink that need to be used (defined by the fully qualified name of the class)
2. The framework starts the Job and creates the Source through the reflection mechanism
3. The framework starts the Source and calls the Split interface of the Source to generate one or more Tasks
4. When a Task is generated, a Reader (a type of Source will generate a corresponding reader), a User-configured Channel and a User-configured Sink are generated at the same time
5. Task starts to execute, Reader starts to read data to Channel, Sink fetches data from Channel and sends it
6. All the information needed for Job and Task execution is encapsulated in the JobProfile

## Reference Demo

Please understand the above process by reading the Job class, Task class, TextFileSource class, TextFileReader class, and ProxySink class in the Agent source
- The user submits a Job (via the manager), and the Job defines the Source, Channel, and Sink that need to be used (defined by the fully qualified name of the class)
- The framework starts the Job and creates the Source through the reflection mechanism
- The framework starts the Source and calls the Split interface of the Source to generate one or more Tasks
- When a Task is generated, a Reader (a type of Source will generate a corresponding reader), a User-configured Channel and a User-configured Sink are generated at the same time
- Task starts to execute, Reader starts to read data to Channel, Sink fetches data from Channel and sends it
- All the information needed for Job and Task execution is encapsulated in the JobProfile

## Development Process

1. First develop Source, implement split logic, and return ReaderList
2. The developed Reader implements the logic of reading data and writing to Channel
3. The sink under development implements the logic of fetching data from the channel and writing it to the specified sink
- First develop Source, implement split logic, and return ReaderList
- The developed Reader implements the logic of reading data and writing to Channel
- The sink under development implements the logic of fetching data from the channel and writing it to the specified sink

## Programming must know interface
## Interface

Some of the plug-ins that will be developed below, the classes and interfaces that need to be known are as follows:
The following will introduce the classes and interfaces you need to know to develop an Agent plug-in.

### Reader
```java
Expand Down Expand Up @@ -189,17 +185,4 @@ public interface Message {
}
```

Developers can expand customized Message according to this interface. For example, ProxyMessage contains InLongGroupId, InLongStreamId and other attributes


## Last but not Least

All new plugins must have a document in the `InLong` official wiki. The document needs to include but not limited to the following:

1. **Quick introduction**: Introduce the usage scenarios and features of the plug-in.
2. **Implementation principle**: Introduce the underlying principle of plug-in implementation, such as `sqlReader` to read data in the database by executing Sql query
3. **Configuration Instructions**
- Give the json configuration file of synchronization tasks in typical scenarios.
- Introduce the meaning of each parameter, whether it is required, default value, value range and other constraints.
4. **Restrictions**: Are there other restrictions on use.
5. **FAQ**: Frequently asked questions by users.
Developers can expand customized Message according to this interface. For example, ProxyMessage contains InLongGroupId, InLongStreamId and other attributes
66 changes: 66 additions & 0 deletions docs/modules/agent/metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
title: Monitor Metrics
sidebar_position: 3
---

## JMX Configuration
Agent provides the ability of monitoring indicators in JMX and Prometheus mode, and JMX mode is used by default. The monitoring indicators have been registered to MBeanServer
Users can add similar JMX (port and authentication are adjusted according to the situation) to the startup parameters of the Agent to realize the collection of monitoring indicators from the remote end.

```shell
-Dcom.sun.management.jmxremote
-Djava.rmi.server.hostname=127.0.0.1
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
```

## Prometheus Configuration
You can declare whether to enable Prometheus and HTTPServer port in `agent.properties`.

```properties
# the default is false
agent.prometheus.enable=true
# the default is 8080
agent.prometheus.exporter.port=8080
```

## Appendix: Metrics Items

### AgentTaskMetric
| property | description |
| ---- | ---- |
| runningTasks | tasks currently being executed |
| retryingTasks | Tasks that are currently being retried |
| fatalTasks | The total number of currently failed tasks |


### JobMetrics
| property | description |
| ---- | ---- |
| runningJobs | the total number of currently running jobs |
| fatalJobs | the total number of currently failed jobs |

### PluginMetric
| property | description |
| ---- | ---- |
| readNum | the number of reads |
| sendNum | the number of sent items |
| sendFailedNum | the number of failed sending |
| readFailedNum | the number of failed reads |
| readSuccessNum | the number of successful reads |
| sendSuccessNum | the number of successfully sent |

### SourceMetric

| property | type | description |
|----------------------------|---------|--------------------------------------------------------------------|
| agent_source_count_success | Counter | the success message count in agent source since agent started |
| agent_source_count_fail | Counter | the sink success message count in agent source since agent started |

### SinkMetric

| property | type | description |
|--------------------------|---------|--------------------------------------------------------------------|
| agent_sink_count_success | Counter | the sink success message count in agent source since agent started |
| agent_sink_count_fail | Counter | the sink failed message count in agent source since agent started |
Loading