Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add json path operator #238

Merged
merged 2 commits into from
Dec 2, 2019
Merged

Add json path operator #238

merged 2 commits into from
Dec 2, 2019

Conversation

kairichard
Copy link
Contributor

@kairichard kairichard commented Dec 2, 2019

Why add this?

The default decode JSON operator requires the user to build a matching type that exactly mimics the schema in the JSON document. However, if the user only wants to extract a single value or a small subset she still has to construct the entire type.

The new operator JSON Path allows the user to specify multiple JSON path queries to extract data. It uses https://github.com/tidwall/gjson for parsing and extracting the data.

There is a noteworthy case where this operator might not work as expected.
When the path expression would return a value that is not a primitive but an object or an array it actually returns the raw JSON for the given expression. Reasons for implementing it that way were:

  • to handle such a case the user would again need to specify which of the expression are expected to return an object or an array, which would defeat the goal of this change.
  • when returning a string we can use the old decode JSON to work with a much smaller document

//cc @td5r
related to #237

@codecov-io
Copy link

Codecov Report

Merging #238 into master will increase coverage by 0.21%.
The diff coverage is 88%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #238      +/-   ##
==========================================
+ Coverage    41.7%   41.92%   +0.21%     
==========================================
  Files         102      103       +1     
  Lines        5361     5386      +25     
==========================================
+ Hits         2236     2258      +22     
- Misses       2946     2948       +2     
- Partials      179      180       +1
Impacted Files Coverage Δ
pkg/elem/manager.go 95% <100%> (+0.04%) ⬆️
pkg/elem/encoding_json_path.go 87.5% <87.5%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d07643c...258a934. Read the comment docs.

@kairichard kairichard merged commit 5085124 into master Dec 2, 2019
kairichard added a commit that referenced this pull request Dec 3, 2019
This PR improves upon #238 where the JSON paths where configured through properties - which led to unparseable port references in the resulting bundle with the message Message: "unknown port: bpi: jsonpath)bpi.USD.rate. That is because the properties would get translated into ports with names like bpi.USD.rate which confuses the parser.

I changed the operator to now accept the paths as inputs and map them to the outports via names given in the properties.

//cc @td5r
-- related to #237
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants