[Feature][Connector-V2][HTTP] Use json-path parsing #3510

liugddx · 2022-11-22T14:29:11Z

close #3500

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist

liugddx · 2022-11-27T12:47:51Z

@hailin0 @Hisoka-X PTAL

Hisoka-X · 2022-11-28T03:00:56Z

docs/en/connector-v2/source/Http.md

+
+### new version
+
+- Increase jsonpath parsing.


Add PR link.

Please refer to the format of other changed logs. [Feature/Bugfix/Improve][XXX]

Hisoka-X · 2022-11-28T03:01:03Z

docs/en/connector-v2/source/Http.md

@@ -124,6 +125,18 @@ connector will generate data as the following:

 the schema fields of upstream data

+### json_field [Config]
+
+The Wildcards for jsonpath like


Add example json, and how to use json_field with example json, like parse data in list. And if we have json_field, do we need config schema or not? The answer of these question should be found in document.

Hisoka-X · 2022-11-28T03:20:55Z

By the way, this is a good feature.

EricJoy2048 · 2022-11-28T11:21:45Z

We have an issue discuss about this #3573. Can we wait for the discussion to finish before processing this pr?

Hisoka-X · 2022-11-29T02:13:59Z

docs/en/connector-v2/source/Http.md

+    }
+  }
+}
+```


Please add real response json example in here.

# Conflicts: # seatunnel-connectors-v2/connector-http/connector-http-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/config/HttpConfig.java # seatunnel-connectors-v2/connector-http/connector-http-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSource.java

liugddx · 2022-11-30T06:19:03Z

Please pending this pull request. Waiting for #3573 finished and decied this pull request is or not merged. cc @Hisoka-X @EricJoy2048 @liugddx

Hi, we have communicated about this issue. If you have other good suggestions, you can also put forward them. @Hisoka-X @EricJoy2048 @TaoZex

liugddx · 2022-11-30T06:19:22Z

Please pending this pull request. Waiting for #3573 finished and decied this pull request is or not merged. cc @Hisoka-X @EricJoy2048 @liugddx

Hi, we have communicated about this issue. If you have other good suggestions, you can also put forward them. @Hisoka-X @EricJoy2048 @TaoZex

@TyrantLucifer

TyrantLucifer · 2022-11-30T07:30:52Z

...se/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceReader.java

+            List<?> result0 = results.get(0);
+            List<?> result = results.get(i);
+            if (result0.size() != result.size()) {
+                throw new SeaTunnelException(


Use HttpConnectorException

TyrantLucifer · 2022-11-30T07:30:56Z

...se/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceReader.java

+    private void initJsonPath(JsonField jsonField) {
+        jsonPaths = new JsonPath[jsonField.getFields().size()];
+        final int[] index = {0};
+        jsonField.getFields().forEach((key, value) -> {


Use entrySet is better, final int[] index = {0}; make the code not clear

EricJoy2048

This is a very good feature. Thanks for you contribution.

TaoZex · 2022-11-30T09:46:02Z

docs/en/connector-v2/source/Http.md

 | name                        | type   | required | default value |
 | --------------------------- | ------ | -------- | ------------- |
 | url                         | String | Yes      | -             |
 | schema                      | Config | No       | -             |
 | schema.fields               | Config | No       | -             |
+| json_field                  | Config | No       | -             |


Please add content_json

Please add content_json

Thanks for your code review.

add content_json

TaoZex · 2022-11-30T10:03:44Z

...se/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceReader.java

+        List<List<String>> datas = new ArrayList<>();
+        for (int i = 0; i < results.size(); i++) {
+            List<String> result = results.get(i);
+            if (i == 0) {
+                for (Object o : result) {
+                    String val = o == null ? null : o.toString();


The code on line 170 is probably not needed because result is of type List<String>.

TaoZex · 2022-11-30T10:05:45Z

...se/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceReader.java

+                    Object o = result.get(j);
+                    String val = o == null ? null : o.toString();


Same as above.

Sometimes there is null data, which needs to be converted to a null string.It is necessary to do so

TaoZex

+1

liugddx marked this pull request as draft November 22, 2022 14:29

liugddx closed this Nov 23, 2022

liugddx reopened this Nov 23, 2022

add json-path dependency

97c10c4

liugddx force-pushed the json-path-parsing branch from 3869359 to 97c10c4 Compare November 25, 2022 06:09

liugddx added 5 commits November 27, 2022 10:41

Merge remote-tracking branch 'upstream/dev' into json-path-parsing

99a5eb3

add json path parsing

e505779

add e2e test

cc36e21

add e2e test

14ae43c

modify md

e38de3a

liugddx marked this pull request as ready for review November 27, 2022 12:47

Hisoka-X reviewed Nov 28, 2022

View reviewed changes

Hisoka-X added feature New feature connectors-v2 labels Nov 28, 2022

EricJoy2048 added the discuss label Nov 28, 2022

liugddx mentioned this pull request Nov 28, 2022

[Discuss][HTTP Connector] Add specified field function for all HTTP connector #3573

Closed

3 tasks

modify md

694f7e5

liugddx requested a review from Hisoka-X November 28, 2022 12:53

Hisoka-X reviewed Nov 29, 2022

View reviewed changes

docs/en/connector-v2/source/Http.md

}

}

}

```

Copy link

Member

Hisoka-X Nov 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add real response json example in here.

liugddx added 2 commits November 29, 2022 13:45

fix cv.

10800fc

remove unnecessary code

4c0f465

liugddx requested a review from Hisoka-X November 29, 2022 05:55

liugddx added 5 commits November 29, 2022 13:58

fix ci error

3594eeb

add content json

199a48f

fix some bug

91b7d07

fix md

d00448c

TyrantLucifer added the don't merge There needs to be a specific reason in the PR, and it cannot be merged for the time being. label Nov 30, 2022

TyrantLucifer removed the don't merge There needs to be a specific reason in the PR, and it cannot be merged for the time being. label Nov 30, 2022

TyrantLucifer reviewed Nov 30, 2022

View reviewed changes

github-actions bot added approved reviewed labels Nov 30, 2022

TyrantLucifer requested a review from EricJoy2048 November 30, 2022 07:32

fix cr problem

c30eef8

liugddx dismissed Hisoka-X’s stale review via c30eef8 November 30, 2022 08:10

github-actions bot removed approved reviewed labels Nov 30, 2022

EricJoy2048 previously approved these changes Nov 30, 2022

View reviewed changes

github-actions bot added approved reviewed labels Nov 30, 2022

TaoZex reviewed Nov 30, 2022

View reviewed changes

Update Http.md

aa47ec8

add content_json

liugddx dismissed EricJoy2048’s stale review via aa47ec8 November 30, 2022 09:50

github-actions bot removed approved reviewed labels Nov 30, 2022

TaoZex reviewed Nov 30, 2022

View reviewed changes

liugddx requested a review from EricJoy2048 November 30, 2022 10:37

EricJoy2048 approved these changes Nov 30, 2022

View reviewed changes

github-actions bot added approved reviewed labels Nov 30, 2022

TaoZex approved these changes Nov 30, 2022

View reviewed changes

liugddx requested a review from Hisoka-X November 30, 2022 10:51

Hisoka-X approved these changes Nov 30, 2022

View reviewed changes

Hisoka-X merged commit 1807eb6 into apache:dev Nov 30, 2022

liugddx deleted the json-path-parsing branch December 1, 2022 01:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2][HTTP] Use json-path parsing #3510

[Feature][Connector-V2][HTTP] Use json-path parsing #3510

liugddx commented Nov 22, 2022

liugddx commented Nov 27, 2022

Hisoka-X Nov 28, 2022

EricJoy2048 Nov 28, 2022 •

edited

Loading

Hisoka-X Nov 28, 2022

Hisoka-X commented Nov 28, 2022

EricJoy2048 commented Nov 28, 2022

Hisoka-X Nov 29, 2022

liugddx commented Nov 30, 2022

liugddx commented Nov 30, 2022

TyrantLucifer Nov 30, 2022

TyrantLucifer Nov 30, 2022

EricJoy2048 left a comment

TaoZex Nov 30, 2022

liugddx Nov 30, 2022

TaoZex Nov 30, 2022

TaoZex Nov 30, 2022

liugddx Nov 30, 2022

TaoZex left a comment

		Object o = result.get(j);
		String val = o == null ? null : o.toString();

[Feature][Connector-V2][HTTP] Use json-path parsing #3510

[Feature][Connector-V2][HTTP] Use json-path parsing #3510

Conversation

liugddx commented Nov 22, 2022

Purpose of this pull request

Check list

liugddx commented Nov 27, 2022

Choose a reason for hiding this comment

EricJoy2048 Nov 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hisoka-X commented Nov 28, 2022

EricJoy2048 commented Nov 28, 2022

Choose a reason for hiding this comment

liugddx commented Nov 30, 2022

liugddx commented Nov 30, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EricJoy2048 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TaoZex left a comment

Choose a reason for hiding this comment

EricJoy2048 Nov 28, 2022 •

edited

Loading