[Feature][Transform] add JsonPath transform #5632

FuYouJ · 2023-10-15T12:53:19Z

related issue

#5633

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

[Transform][Jsonpath]add JsonPathTransform

[Transform][Jsonpath][Doc] add doc

[Transform][Jsonpath][Doc]update release

[Transform][Jsonpath]private variables may not use concurrent containers

ruanwenjun · 2023-10-16T01:59:33Z

pom.xml

@@ -128,6 +128,7 @@
        <testcontainer.version>1.17.6</testcontainer.version>
        <spotless.version>2.29.0</spotless.version>
        <jsqlparser.version>4.5</jsqlparser.version>
+        <jsonpath.version>2.8.0</jsonpath.version>


There already exist 2.7.0, it's not a good idea to used multiple version json-path.

not found a dependency for 2.7.0 yet.Could you point out where it is thank you

seatunnel/seatunnel-connectors-v2/connector-http/connector-http-base/pom.xml

Lines 32 to 37 in a15b79f

<properties>

<httpclient.version>4.5.13</httpclient.version>

<httpcore.version>4.4.4</httpcore.version>

<guava-retrying.version>2.0.0</guava-retrying.version>

<json-path.version>2.7.0</json-path.version>

</properties>

…ttpbase [Transform][Jsonpath] jsonpath dependency use the same version with httpbase

FuYouJ · 2023-10-17T13:40:16Z

@EricJoy2048 could you review this pr,thx

FuYouJ · 2023-10-17T13:52:12Z

@EricJoy2048 could you review this pr,thx

jsonpath depend on

net.minidev:json-smart:2.4.7
net.minidev:accessors-smart:2.4.7
org.ow2.asm:asm:9.1

seatunnel-dist/release-docs/LICENSE

seatunnel-dist/release-docs/licenses/LICENSE-jayway-jsonpath.txt

… anthor apache2.0 file

TyrantLucifer

Please add parse array and parse nest object test cases.

TyrantLucifer · 2023-10-20T06:16:29Z

...l-transforms-v2/src/main/java/org/apache/seatunnel/transform/jsonpath/JsonPathTransform.java

+import static org.apache.seatunnel.transform.exception.JsonPathTransformErrorCode.JSON_PATH_COMPILE_ERROR;
+
+@Slf4j
+@AutoService(JsonPathTransform.class)


Why not Transform.class?

TyrantLucifer · 2023-10-20T06:18:21Z

seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/jsonpath/FiledConfig.java

+    private String path;
+
+    private String srcField;
+
+    private String destField;


final could be better.

…modifier

FuYouJ · 2023-10-22T07:32:50Z

@ruanwenjun jun✌🏻 please review

ruanwenjun · 2023-10-23T03:30:36Z

CI failed, you need to check if the code has miss something.

FuYouJ · 2023-10-23T14:56:48Z

CI failed, you need to check if the code has miss something.

well,I finally fixed the problem. now, ci is successed

TyrantLucifer

Please add e2e test cases to parse array nest object

TyrantLucifer · 2023-10-24T13:24:04Z

...l-transforms-v2/src/main/java/org/apache/seatunnel/transform/jsonpath/JsonPathTransform.java

+    private JsonPathTransformConfig config;
+    private SeaTunnelRowType seaTunnelRowType;


final be better

…datatype-jsr310 to support parse localDateTime

…sion to seaTunnel sqlType

FuYouJ · 2023-10-29T12:28:04Z

@ruanwenjun @ruanwenjun @EricJoy2048
I refactored the code reference httpsource and fakesource to make it simpler and more powerful. Please review

FuYouJ · 2023-10-31T14:54:00Z

@ruanwenjun jun✌🏻 please review

FuYouJ · 2023-11-04T03:35:07Z

@ruanwenjun @TyrantLucifer It's been a long time. I miss you so much(✺ω✺)

Hisoka-X

Thanks @FuYouJ !

already add.

FuYouJ · 2023-11-07T06:37:43Z

Thanks @FuYouJ !

You're welcome!

FuYouJ and others added 5 commits October 15, 2023 19:57

[Transform][Jsonpath]add JsonPathTransform

5035ebb

[Transform][Jsonpath]add JsonPathTransform

[Transform][Jsonpath][Doc] add doc

b2342de

[Transform][Jsonpath][Doc] add doc

[Transform][Jsonpath][Doc]update release

5d462b2

[Transform][Jsonpath][Doc]update release

[Transform][Jsonpath]private variables may not use concurrent containers

f63699f

[Transform][Jsonpath]private variables may not use concurrent containers

[Transform][JsonPath] Clean dependent comment

13c5fd7

ruanwenjun requested changes Oct 16, 2023

View reviewed changes

FuYouJ added 6 commits October 16, 2023 20:48

[Transform][Jsonpath] jsonpath dependency use the same version with h…

980c9d1

…ttpbase [Transform][Jsonpath] jsonpath dependency use the same version with httpbase

[Transform][Jsonpath]doc

4947520

[Transform][Jsonpath]add com.jayway.jsonpath license

223b843

[Transform][Jsonpath]known-dependencies.txt add json-path-2.7.0.jar

a369b0c

[Transform][Jsonpath]known-dependencies.txt add jjson-smart-2.4.7.jar

02fe884

[Transform][Jsonpath]known-dependencies.txt jsonpath compile dependency

2cf66ad

FuYouJ requested a review from ruanwenjun October 17, 2023 13:33

ruanwenjun previously requested changes Oct 18, 2023

View reviewed changes

seatunnel-dist/release-docs/LICENSE Show resolved Hide resolved

seatunnel-dist/release-docs/licenses/LICENSE-jayway-jsonpath.txt Outdated Show resolved Hide resolved

ruanwenjun added dependencies Pull requests that update a dependency file feature New feature labels Oct 18, 2023

ruanwenjun assigned FuYouJ Oct 18, 2023

FuYouJ added 2 commits October 18, 2023 20:29

[Transform][Jsonpath]remove LICENSE-jayway-jsonpath.txt because exist…

d894b82

… anthor apache2.0 file

[Fix]accessors-smart is source from net.minidev not google,so fix it

55cd09a

FuYouJ requested a review from ruanwenjun October 18, 2023 12:38

TyrantLucifer reviewed Oct 20, 2023

View reviewed changes

FuYouJ added 3 commits October 22, 2023 13:20

Merge branch 'apache:dev' into jsonpath

998fa9e

[Transform][JsonPath]remove useless impl, config props use the final …

a8ce608

…modifier

[Transform][JsonPath]code style

18b8ed1

FuYouJ requested a review from TyrantLucifer October 22, 2023 07:03

[Transform][JsonPath]Fix possible multithreading errors

ae4bf7a

TyrantLucifer previously requested changes Oct 24, 2023

View reviewed changes

FuYouJ added 4 commits October 25, 2023 21:36

[Transform][JsonPath]add array and seatunnel row e2eTest,add jackson-…

d029afe

…datatype-jsr310 to support parse localDateTime

Merge branch 'apache:dev' into jsonpath

262a7c4

[Transform][JsonPath]refactor code,the parsed content supports conver…

122aa9f

…sion to seaTunnel sqlType

[Transform][JsonPath]fix e2e test

b46a6ac

FuYouJ requested a review from TyrantLucifer October 29, 2023 12:26

[Transform][JsonPath]rename dataType to destType

8d4d9ed

FuYouJ added 4 commits November 3, 2023 11:04

Merge branch 'dev' into jsonpath

f4a89cc

Merge branch 'apache:dev' into jsonpath

d35507b

[Transform][JsonPath]resolve jackson version conflict

66db2da

[Transform][JsonPath]resolve jackson version conflict

0a23647

hailin0 approved these changes Nov 6, 2023

View reviewed changes

Hisoka-X approved these changes Nov 7, 2023

View reviewed changes

Hisoka-X merged commit d908f0a into apache:dev Nov 7, 2023
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Transform] add JsonPath transform #5632

[Feature][Transform] add JsonPath transform #5632

FuYouJ commented Oct 15, 2023 •

edited

Loading

ruanwenjun Oct 16, 2023

FuYouJ Oct 16, 2023 •

edited

Loading

ruanwenjun Oct 16, 2023

FuYouJ commented Oct 17, 2023

FuYouJ commented Oct 17, 2023 •

edited

Loading

TyrantLucifer left a comment

TyrantLucifer Oct 20, 2023

TyrantLucifer Oct 20, 2023

FuYouJ commented Oct 22, 2023

ruanwenjun commented Oct 23, 2023

FuYouJ commented Oct 23, 2023

TyrantLucifer left a comment

TyrantLucifer Oct 24, 2023

FuYouJ commented Oct 29, 2023

FuYouJ commented Oct 31, 2023

FuYouJ commented Nov 4, 2023 •

edited

Loading

Hisoka-X left a comment

FuYouJ commented Nov 7, 2023

	<properties>
	<httpclient.version>4.5.13</httpclient.version>
	<httpcore.version>4.4.4</httpcore.version>
	<guava-retrying.version>2.0.0</guava-retrying.version>
	<json-path.version>2.7.0</json-path.version>
	</properties>

		private JsonPathTransformConfig config;
		private SeaTunnelRowType seaTunnelRowType;

[Feature][Transform] add JsonPath transform #5632

[Feature][Transform] add JsonPath transform #5632

Conversation

FuYouJ commented Oct 15, 2023 • edited Loading

related issue

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

ruanwenjun Oct 16, 2023

Choose a reason for hiding this comment

FuYouJ Oct 16, 2023 • edited Loading

Choose a reason for hiding this comment

ruanwenjun Oct 16, 2023

Choose a reason for hiding this comment

FuYouJ commented Oct 17, 2023

FuYouJ commented Oct 17, 2023 • edited Loading

TyrantLucifer left a comment

Choose a reason for hiding this comment

TyrantLucifer Oct 20, 2023

Choose a reason for hiding this comment

TyrantLucifer Oct 20, 2023

Choose a reason for hiding this comment

FuYouJ commented Oct 22, 2023

ruanwenjun commented Oct 23, 2023

FuYouJ commented Oct 23, 2023

TyrantLucifer left a comment

Choose a reason for hiding this comment

TyrantLucifer Oct 24, 2023

Choose a reason for hiding this comment

FuYouJ commented Oct 29, 2023

FuYouJ commented Oct 31, 2023

FuYouJ commented Nov 4, 2023 • edited Loading

Hisoka-X left a comment

Choose a reason for hiding this comment

FuYouJ commented Nov 7, 2023

FuYouJ commented Oct 15, 2023 •

edited

Loading

FuYouJ Oct 16, 2023 •

edited

Loading

FuYouJ commented Oct 17, 2023 •

edited

Loading

FuYouJ commented Nov 4, 2023 •

edited

Loading