Skip to content

Commit 46abc6d

Browse files
TaoZexEricJoy2048
TaoZex
and
EricJoy2048
authored
[Feature][Connector-V2][Notion] Add Notion source connector (#3470)
* [Feature][Connector-V2][Notion]Add Notion source connector * [Feature][Connector-V2][Notion]update responce field * [Feature][Connector-V2][Notion]Unified exception for Notion connector * [Feature][Connector-V2][Notion] add version option rule * [Feature][Connector-V2][Notion] Improve json parse option rule * [Feature][Connector-V2][Notion] Update e2e config * [Feature][Connector-V2][Notion] Update Notion doc * [Feature][Connector-V2][Notion] Update source factory Co-authored-by: Eric <gaojun2048@gmail.com>
1 parent 54f594d commit 46abc6d

File tree

16 files changed

+851
-1
lines changed

16 files changed

+851
-1
lines changed

‎docs/en/connector-v2/source/Lemlist.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ http request url
4040

4141
### password [String]
4242

43-
API key for login, you can get it at this link:
43+
API key for login, you can get more detail at this link:
4444

4545
https://app.lemlist.com/settings/integrations
4646

‎docs/en/connector-v2/source/Notion.md‎

Lines changed: 305 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,305 @@
1+
# Notion
2+
3+
> Notion source connector
4+
5+
## Description
6+
7+
Used to read data from Notion.
8+
9+
## Key features
10+
11+
- [x] [batch](../../concept/connector-v2-features.md)
12+
- [ ] [stream](../../concept/connector-v2-features.md)
13+
- [ ] [exactly-once](../../concept/connector-v2-features.md)
14+
- [x] [schema projection](../../concept/connector-v2-features.md)
15+
- [ ] [parallelism](../../concept/connector-v2-features.md)
16+
- [ ] [support user-defined split](../../concept/connector-v2-features.md)
17+
18+
## Options
19+
20+
| name | type | required | default value |
21+
| --------------------------- | ------ | -------- | ------------- |
22+
| url | String | Yes | - |
23+
| password | String | Yes | - |
24+
| version | String | Yes | - |
25+
| method | String | No | get |
26+
| schema.fields | Config | No | - |
27+
| format | String | No | json |
28+
| params | Map | No | - |
29+
| body | String | No | - |
30+
| json_field | Config | No | - |
31+
| content_json | String | No | - |
32+
| poll_interval_ms | int | No | - |
33+
| retry | int | No | - |
34+
| retry_backoff_multiplier_ms | int | No | 100 |
35+
| retry_backoff_max_ms | int | No | 10000 |
36+
| common-options | config | No | - |
37+
38+
### url [String]
39+
40+
http request url
41+
42+
### password [String]
43+
44+
API key for login, you can get more detail at this link:
45+
46+
https://developers.notion.com/docs/authorization
47+
48+
### version [String]
49+
50+
The Notion API is versioned. API versions are named for the date the version is released
51+
52+
### method [String]
53+
54+
http request method, only supports GET, POST method
55+
56+
### params [Map]
57+
58+
http params
59+
60+
### body [String]
61+
62+
http body
63+
64+
### poll_interval_ms [int]
65+
66+
request http api interval(millis) in stream mode
67+
68+
### retry [int]
69+
70+
The max retry times if request http return to `IOException`
71+
72+
### retry_backoff_multiplier_ms [int]
73+
74+
The retry-backoff times(millis) multiplier if request http failed
75+
76+
### retry_backoff_max_ms [int]
77+
78+
The maximum retry-backoff times(millis) if request http failed
79+
80+
### format [String]
81+
82+
the format of upstream data, now only support `json` `text`, default `json`.
83+
84+
when you assign format is `json`, you should also assign schema option, for example:
85+
86+
upstream data is the following:
87+
88+
```json
89+
{
90+
"code": 200,
91+
"data": "get success",
92+
"success": true
93+
}
94+
```
95+
96+
you should assign schema as the following:
97+
98+
```hocon
99+
100+
schema {
101+
fields {
102+
code = int
103+
data = string
104+
success = boolean
105+
}
106+
}
107+
108+
```
109+
110+
connector will generate data as the following:
111+
112+
| code | data | success |
113+
|------|-------------|---------|
114+
| 200 | get success | true |
115+
116+
when you assign format is `text`, connector will do nothing for upstream data, for example:
117+
118+
upstream data is the following:
119+
120+
```json
121+
{
122+
"code": 200,
123+
"data": "get success",
124+
"success": true
125+
}
126+
```
127+
128+
connector will generate data as the following:
129+
130+
| content |
131+
|---------|
132+
| {"code": 200, "data": "get success", "success": true} |
133+
134+
### schema [Config]
135+
136+
#### fields [Config]
137+
138+
the schema fields of upstream data
139+
140+
### content_json [String]
141+
142+
This parameter can get some json data.If you only need the data in the 'book' section, configure `content_field = "$.store.book.*"`.
143+
144+
If your return data looks something like this.
145+
146+
```json
147+
{
148+
"store": {
149+
"book": [
150+
{
151+
"category": "reference",
152+
"author": "Nigel Rees",
153+
"title": "Sayings of the Century",
154+
"price": 8.95
155+
},
156+
{
157+
"category": "fiction",
158+
"author": "Evelyn Waugh",
159+
"title": "Sword of Honour",
160+
"price": 12.99
161+
}
162+
],
163+
"bicycle": {
164+
"color": "red",
165+
"price": 19.95
166+
}
167+
},
168+
"expensive": 10
169+
}
170+
```
171+
You can configure `content_field = "$.store.book.*"` and the result returned looks like this:
172+
173+
```json
174+
[
175+
{
176+
"category": "reference",
177+
"author": "Nigel Rees",
178+
"title": "Sayings of the Century",
179+
"price": 8.95
180+
},
181+
{
182+
"category": "fiction",
183+
"author": "Evelyn Waugh",
184+
"title": "Sword of Honour",
185+
"price": 12.99
186+
}
187+
]
188+
```
189+
Then you can get the desired result with a simpler schema,like
190+
191+
```hocon
192+
Http {
193+
url = "http://mockserver:1080/contentjson/mock"
194+
method = "GET"
195+
format = "json"
196+
content_field = "$.store.book.*"
197+
schema = {
198+
fields {
199+
category = string
200+
author = string
201+
title = string
202+
price = string
203+
}
204+
}
205+
}
206+
```
207+
208+
Here is an example:
209+
210+
- Test data can be found at this link [mockserver-contentjson-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-contentjson-config.json)
211+
- See this link for task configuration [http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
212+
213+
214+
### json_field [Config]
215+
216+
This parameter helps you configure the schema,so this parameter must be used with schema.
217+
218+
If your data looks something like this:
219+
220+
```json
221+
{
222+
"store": {
223+
"book": [
224+
{
225+
"category": "reference",
226+
"author": "Nigel Rees",
227+
"title": "Sayings of the Century",
228+
"price": 8.95
229+
},
230+
{
231+
"category": "fiction",
232+
"author": "Evelyn Waugh",
233+
"title": "Sword of Honour",
234+
"price": 12.99
235+
}
236+
],
237+
"bicycle": {
238+
"color": "red",
239+
"price": 19.95
240+
}
241+
},
242+
"expensive": 10
243+
}
244+
```
245+
246+
You can get the contents of 'book' by configuring the task as follows:
247+
248+
```hocon
249+
source {
250+
Http {
251+
url = "http://mockserver:1080/jsonpath/mock"
252+
method = "GET"
253+
format = "json"
254+
json_field = {
255+
category = "$.store.book[*].category"
256+
author = "$.store.book[*].author"
257+
title = "$.store.book[*].title"
258+
price = "$.store.book[*].price"
259+
}
260+
schema = {
261+
fields {
262+
category = string
263+
author = string
264+
title = string
265+
price = string
266+
}
267+
}
268+
}
269+
}
270+
```
271+
272+
- Test data can be found at this link [mockserver-jsonpath-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-jsonpath-config.json)
273+
- See this link for task configuration [http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
274+
275+
### common options
276+
277+
Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details
278+
279+
## Example
280+
281+
```hocon
282+
Notion {
283+
url = "https://api.notion.com/v1/users"
284+
password = "Seatunnel-test"
285+
version = "2022-06-28"
286+
content_field = "$.results.*"
287+
schema = {
288+
fields {
289+
object = string
290+
id = string
291+
type = string
292+
person = {
293+
email = string
294+
}
295+
avatar_url = string
296+
}
297+
}
298+
}
299+
```
300+
301+
## Changelog
302+
303+
### next version
304+
305+
- Add Notion Source Connector

‎plugin-mapping.properties‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,7 @@ seatunnel.sink.Slack = connector-slack
154154
seatunnel.source.OneSignal = connector-http-onesignal
155155
seatunnel.source.Jira = connector-http-jira
156156
seatunnel.source.Gitlab = connector-http-gitlab
157+
seatunnel.source.Notion = connector-http-notion
157158
seatunnel.sink.RabbitMQ = connector-rabbitmq
158159
seatunnel.source.RabbitMQ = connector-rabbitmq
159160
seatunnel.source.OpenMldb = connector-openmldb

‎seatunnel-connectors-v2/connector-http/connector-http-notion/pom.xml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
4+
Licensed to the Apache Software Foundation (ASF) under one or more
5+
contributor license agreements. See the NOTICE file distributed with
6+
this work for additional information regarding copyright ownership.
7+
The ASF licenses this file to You under the Apache License, Version 2.0
8+
(the "License"); you may not use this file except in compliance with
9+
the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing, software
14+
distributed under the License is distributed on an "AS IS" BASIS,
15+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
See the License for the specific language governing permissions and
17+
limitations under the License.
18+
19+
-->
20+
<project xmlns="http://maven.apache.org/POM/4.0.0"
21+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
22+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
23+
<parent>
24+
<artifactId>connector-http</artifactId>
25+
<groupId>org.apache.seatunnel</groupId>
26+
<version>${revision}</version>
27+
</parent>
28+
<modelVersion>4.0.0</modelVersion>
29+
30+
<artifactId>connector-http-notion</artifactId>
31+
32+
<dependencies>
33+
<dependency>
34+
<groupId>org.apache.seatunnel</groupId>
35+
<artifactId>connector-http-base</artifactId>
36+
<version>${project.version}</version>
37+
</dependency>
38+
</dependencies>
39+
40+
</project>

0 commit comments

Comments
 (0)