Skip to content

Commit dde6f9f

Browse files
CosmosNiCosmosNi
authored
[Feature][Connector-V2] Add prometheus source and sink (#7265)
1 parent 8d9c6a3 commit dde6f9f

File tree

41 files changed

+27752
-2
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+27752
-2
lines changed

‎.github/workflows/labeler/label-scope-conf.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,11 @@ http:
132132
- changed-files:
133133
- any-glob-to-any-file: seatunnel-connectors-v2/connector-http/**
134134
- all-globs-to-all-files: '!seatunnel-connectors-v2/connector-!(http)/**'
135+
prometheus:
136+
- all:
137+
- changed-files:
138+
- any-glob-to-any-file: seatunnel-connectors-v2/connector-prometheus/**
139+
- all-globs-to-all-files: '!seatunnel-connectors-v2/connector-!(prometheus)/**'
135140
hudi:
136141
- all:
137142
- changed-files:

‎config/plugin_config‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,7 @@ connector-tdengine
8888
connector-web3j
8989
connector-milvus
9090
connector-activemq
91+
connector-prometheus
9192
connector-sls
9293
connector-qdrant
9394
connector-typesense

‎docs/en/connector-v2/sink/Prometheus.md‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
# Prometheus
2+
3+
> Prometheus sink connector
4+
5+
## Support Those Engines
6+
7+
> Spark<br/>
8+
> Flink<br/>
9+
> SeaTunnel Zeta<br/>
10+
11+
## Key Features
12+
13+
- [ ] [exactly-once](../../concept/connector-v2-features.md)
14+
- [ ] [cdc](../../concept/connector-v2-features.md)
15+
- [x] [support multiple table write](../../concept/connector-v2-features.md)
16+
17+
## Description
18+
19+
Used to launch web hooks using data.
20+
21+
> For example, if the data from upstream is [`label: {"__name__": "test1"}, value: 1.2.3,time:2024-08-15T17:00:00`], the body content is the following: `{"label":{"__name__": "test1"}, "value":"1.23","time":"2024-08-15T17:00:00"}`
22+
23+
**Tips: Prometheus sink only support `post json` webhook and the data from source will be treated as body content in web hook.And does not support passing past data**
24+
25+
## Supported DataSource Info
26+
27+
In order to use the Http connector, the following dependencies are required.
28+
They can be downloaded via install-plugin.sh or from the Maven central repository.
29+
30+
| Datasource | Supported Versions | Dependency |
31+
|------------|--------------------|------------------------------------------------------------------------------------------------------------------|
32+
| Http | universal | [Download](https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-connectors-v2/connector-prometheus) |
33+
34+
## Sink Options
35+
36+
| Name | Type | Required | Default | Description |
37+
|-----------------------------|--------|----------|---------|-------------------------------------------------------------------------------------------------------------|
38+
| url | String | Yes | - | Http request url |
39+
| headers | Map | No | - | Http headers |
40+
| retry | Int | No | - | The max retry times if request http return to `IOException` |
41+
| retry_backoff_multiplier_ms | Int | No | 100 | The retry-backoff times(millis) multiplier if request http failed |
42+
| retry_backoff_max_ms | Int | No | 10000 | The maximum retry-backoff times(millis) if request http failed |
43+
| connect_timeout_ms | Int | No | 12000 | Connection timeout setting, default 12s. |
44+
| socket_timeout_ms | Int | No | 60000 | Socket timeout setting, default 60s. |
45+
| key_timestamp | Int | NO | - | prometheus timestamp key . |
46+
| key_label | String | yes | - | prometheus label key |
47+
| key_value | Double | yes | - | prometheus value |
48+
| batch_size | Int | false | 1024 | prometheus batch size write |
49+
| flush_interval | Long | false | 300000L | prometheus flush commit interval |
50+
| common-options | | No | - | Sink plugin common parameters, please refer to [Sink Common Options](../sink-common-options.md) for details |
51+
52+
## Example
53+
54+
simple:
55+
56+
```hocon
57+
env {
58+
parallelism = 1
59+
job.mode = "BATCH"
60+
}
61+
62+
source {
63+
FakeSource {
64+
schema = {
65+
fields {
66+
c_map = "map<string, string>"
67+
c_double = double
68+
c_timestamp = timestamp
69+
}
70+
}
71+
result_table_name = "fake"
72+
rows = [
73+
{
74+
kind = INSERT
75+
fields = [{"__name__": "test1"}, 1.23, "2024-08-15T17:00:00"]
76+
},
77+
{
78+
kind = INSERT
79+
fields = [{"__name__": "test2"}, 1.23, "2024-08-15T17:00:00"]
80+
}
81+
]
82+
}
83+
}
84+
85+
86+
sink {
87+
Prometheus {
88+
url = "http://prometheus:9090/api/v1/write"
89+
key_label = "c_map"
90+
key_value = "c_double"
91+
key_timestamp = "c_timestamp"
92+
batch_size = 1
93+
}
94+
}
95+
96+
```
97+
98+
## Changelog
99+
100+
### 2.3.8-beta 2024-08-22
101+
102+
- Add Http Sink Connector
103+

‎docs/en/connector-v2/source/Prometheus.md‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# Prometheus
2+
3+
> Prometheus source connector
4+
5+
## Description
6+
7+
Used to read data from Prometheus.
8+
9+
## Key features
10+
11+
- [x] [batch](../../concept/connector-v2-features.md)
12+
- [ ] [stream](../../concept/connector-v2-features.md)
13+
- [ ] [parallelism](../../concept/connector-v2-features.md)
14+
15+
## Options
16+
17+
| name | type | required | default value |
18+
|-----------------------------|---------|----------|-----------------|
19+
| url | String | Yes | - |
20+
| query | String | Yes | - |
21+
| query_type | String | Yes | Instant |
22+
| content_field | String | Yes | $.data.result.* |
23+
| schema.fields | Config | Yes | - |
24+
| format | String | No | json |
25+
| params | Map | Yes | - |
26+
| poll_interval_millis | int | No | - |
27+
| retry | int | No | - |
28+
| retry_backoff_multiplier_ms | int | No | 100 |
29+
| retry_backoff_max_ms | int | No | 10000 |
30+
| enable_multi_lines | boolean | No | false |
31+
| common-options | config | No | - |
32+
33+
### url [String]
34+
35+
http request url
36+
37+
### query [String]
38+
39+
Prometheus expression query string
40+
41+
### query_type [String]
42+
43+
Instant/Range
44+
45+
1. Instant : The following endpoint evaluates an instant query at a single point in time
46+
2. Range : The following endpoint evaluates an expression query over a range of time
47+
48+
https://prometheus.io/docs/prometheus/latest/querying/api/
49+
50+
### params [Map]
51+
52+
http request params
53+
54+
### poll_interval_millis [int]
55+
56+
request http api interval(millis) in stream mode
57+
58+
### retry [int]
59+
60+
The max retry times if request http return to `IOException`
61+
62+
### retry_backoff_multiplier_ms [int]
63+
64+
The retry-backoff times(millis) multiplier if request http failed
65+
66+
### retry_backoff_max_ms [int]
67+
68+
The maximum retry-backoff times(millis) if request http failed
69+
70+
### format [String]
71+
72+
the format of upstream data, default `json`.
73+
74+
### schema [Config]
75+
76+
Fill in a fixed value
77+
78+
```hocon
79+
schema = {
80+
fields {
81+
metric = "map<string, string>"
82+
value = double
83+
time = long
84+
}
85+
}
86+
87+
```
88+
89+
#### fields [Config]
90+
91+
the schema fields of upstream data
92+
93+
### common options
94+
95+
Source plugin common parameters, please refer to [Source Common Options](../source-common-options.md) for details
96+
97+
## Example
98+
99+
### Instant:
100+
101+
```hocon
102+
source {
103+
Prometheus {
104+
result_table_name = "http"
105+
url = "http://mockserver:1080"
106+
query = "up"
107+
query_type = "Instant"
108+
content_field = "$.data.result.*"
109+
format = "json"
110+
schema = {
111+
fields {
112+
metric = "map<string, string>"
113+
value = double
114+
time = long
115+
}
116+
}
117+
}
118+
}
119+
```
120+
121+
### Range
122+
123+
```hocon
124+
source {
125+
Prometheus {
126+
result_table_name = "http"
127+
url = "http://mockserver:1080"
128+
query = "up"
129+
query_type = "Range"
130+
content_field = "$.data.result.*"
131+
format = "json"
132+
start = "2024-07-22T20:10:30.781Z"
133+
end = "2024-07-22T20:11:00.781Z"
134+
step = "15s"
135+
schema = {
136+
fields {
137+
metric = "map<string, string>"
138+
value = double
139+
time = long
140+
}
141+
}
142+
}
143+
}
144+
```
145+
146+
## Changelog
147+
148+
### next version
149+
150+
- Add Prometheus Source Connector
151+
- Reduce configuration items
152+

‎docs/zh/connector-v2/sink/Prometheus.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# Prometheus
2+
3+
> Prometheus 数据接收器
4+
5+
## 引擎支持
6+
7+
> Spark<br/>
8+
> Flink<br/>
9+
> SeaTunnel Zeta<br/>
10+
11+
## 主要特性
12+
13+
- [ ] [exactly-once](../../concept/connector-v2-features.md)
14+
- [ ] [cdc](../../concept/connector-v2-features.md)
15+
- [x] [support multiple table write](../../concept/connector-v2-features.md)
16+
17+
## 描述
18+
19+
接收Source端传入的数据，利用数据触发 web hooks。
20+
21+
> 例如，来自上游的数据为 [`label: {"__name__": "test1"}, value: 1.2.3,time:2024-08-15T17:00:00`], 则body内容如下: `{"label":{"__name__": "test1"}, "value":"1.23","time":"2024-08-15T17:00:00"}`
22+
23+
**Tips: Prometheus 数据接收器 仅支持 `post json` 类型的 web hook，source 数据将被视为 webhook 中的 body 内容。并且不支持传递过去太久的数据**
24+
25+
## 支持的数据源信息
26+
27+
想使用 Prometheus 连接器，需要安装以下必要的依赖。可以通过运行 install-plugin.sh 脚本或者从 Maven 中央仓库下载这些依赖
28+
29+
| 数据源 | 支持版本 | 依赖 |
30+
|------|-----------|------------------------------------------------------------------------------------------------------------------|
31+
| Http | universal | [Download](https://mvnrepository.com/artifact/org.apache.seatunnel/seatunnel-connectors-v2/connector-prometheus) |
32+
33+
## 接收器选项
34+
35+
| Name | Type | Required | Default | Description |
36+
|-----------------------------|--------|----------|---------|-------------------------------------------------------------------|
37+
| url | String | Yes | - | Http 请求链接 |
38+
| headers | Map | No | - | Http 标头 |
39+
| retry | Int | No | - | 如果请求http返回`IOException`的最大重试次数 |
40+
| retry_backoff_multiplier_ms | Int | No | 100 | http请求失败，重试回退次数（毫秒）乘数 |
41+
| retry_backoff_max_ms | Int | No | 10000 | http请求失败，最大重试回退时间(毫秒) |
42+
| connect_timeout_ms | Int | No | 12000 | 连接超时设置，默认12s |
43+
| socket_timeout_ms | Int | No | 60000 | 套接字超时设置，默认为60s |
44+
| key_timestamp | Int | NO | - | prometheus时间戳的key. |
45+
| key_label | String | yes | - | prometheus标签的key |
46+
| key_value | Double | yes | - | prometheus值的key |
47+
| batch_size | Int | false | 1024 | prometheus批量写入大小 |
48+
| flush_interval | Long | false | 300000L | prometheus定时写入 |
49+
| common-options | | No | - | Sink插件常用参数，请参考 [Sink常用选项 ](../sink-common-options.md) 了解详情 |
50+
51+
## 示例
52+
53+
简单示例:
54+
55+
```hocon
56+
env {
57+
parallelism = 1
58+
job.mode = "BATCH"
59+
}
60+
61+
source {
62+
FakeSource {
63+
schema = {
64+
fields {
65+
c_map = "map<string, string>"
66+
c_double = double
67+
c_timestamp = timestamp
68+
}
69+
}
70+
result_table_name = "fake"
71+
rows = [
72+
{
73+
kind = INSERT
74+
fields = [{"__name__": "test1"}, 1.23, "2024-08-15T17:00:00"]
75+
},
76+
{
77+
kind = INSERT
78+
fields = [{"__name__": "test2"}, 1.23, "2024-08-15T17:00:00"]
79+
}
80+
]
81+
}
82+
}
83+
84+
85+
sink {
86+
Prometheus {
87+
url = "http://prometheus:9090/api/v1/write"
88+
key_label = "c_map"
89+
key_value = "c_double"
90+
key_timestamp = "c_timestamp"
91+
batch_size = 1
92+
}
93+
}
94+
```
95+
96+
## Changelog
97+
98+
### 2.3.8-beta 2024-08-22
99+
100+
- 添加prometheus接收连接器
101+

0 commit comments

Comments
 (0)