Adding jsonExtractScalar function to extract field from json object

[xiangfu0]

Right now we could put a JSON blob into a string field.
This udf leverages JsonPath (https://github.com/json-path/JsonPath) DSL to read from a JSON string.

1. Adding Transform function: jsonExtractScalar

Function Syntax:

jsonExtractScalar(JSON_STRING_FIELD, JSON_PATH, OUTPUT_FORMAT)

Sample queries:

Select jsonExtractScalar(myJsonMapStr,'$.k1','STRING') from myTable  where jsonExtractScalar(myJsonMapStr,'$.k1','STRING') = 'value-k1-0'";

Select sum(jsonExtractScalar(complexMapStr,'$.k4.met','INT')) from myTable group by jsonExtractScalar(complexMapStr,'$.k1','STRING')

2. Adding Transform function: jsonExtractKey to extract the paths for a given pattern of a json.
   E.g. for a given json:

{ "k1": "v1", "k2": "v2", "k3": "v3" }

The result of jsonExtractKey(jsonField, '$.*') will return a list a string of matched json paths pattern. In above example, the result is ["$['k1']", "$['k2']", "$['k3']"]

3. Normalized transform function to resolve both jsonExtractScalar and json_extract_scalar function names to the same function.

[codecov-io]

Codecov Report

Merging #4597 into master will increase coverage by 0.42%.
The diff coverage is 73.83%.

@@            Coverage Diff             @@
##           master    #4597      +/-   ##
==========================================
+ Coverage   66.08%   66.50%   +0.42%     
==========================================
  Files        1072     1077       +5     
  Lines       54668    54978     +310     
  Branches     8152     8213      +61     
==========================================
+ Hits        36125    36565     +440     
+ Misses      15895    15717     -178     
- Partials     2648     2696      +48     

Impacted Files	Coverage Δ
...ava/org/apache/pinot/common/utils/SchemaUtils.java	9.85% <0.00%> (ø)
...ger/realtime/Server2ControllerSegmentUploader.java	71.42% <ø> (ø)
...edicate/BaseDictionaryBasedPredicateEvaluator.java	54.16% <ø> (ø)
...predicate/BaseRawValueBasedPredicateEvaluator.java	87.87% <ø> (ø)
...e/operator/dociditerators/MVScanDocIdIterator.java	59.37% <14.28%> (-6.73%)	⬇️
...not/core/data/recordtransformer/PinotDataType.java	95.08% <33.33%> (-1.03%)	⬇️
...m/function/JsonExtractScalarTransformFunction.java	48.80% <48.80%> (ø)
...rg/apache/pinot/broker/routing/RoutingManager.java	80.91% <50.00%> (-0.24%)	⬇️
...e/pinot/common/function/TransformFunctionType.java	82.85% <50.00%> (-4.24%)	⬇️
...indexsegment/generator/SegmentGeneratorConfig.java	72.44% <50.00%> (+10.18%)	⬆️
... and 103 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 44b0a91...fa1bfdc. Read the comment docs.

[mcvsubbu]

Can you also add documentation on how json columns are to be used? I suppose the json columns cannot be used in filters (yet) right?

[xiangfu0]

Will add more documents, this could be used in selection/filtering/groupby, please refer to integration tests.

Current implementation requires the column to be a string column and the content is the json string in order to be used（e.g. the AVRO field needs to be String type）. I'm still investigating how to directly ingest a record/map type into the field.

[kishoreg]

Now that we have support for udf in filter predicates, this can be an amazing feature.
Can we change the function names to match with presto

• https://prestodb.io/docs/current/functions/json.html
• https://docs.aws.amazon.com/athena/latest/ug/querying-JSON.html

[xiangfu0]

Now that we have support for udf in filter predicates, this can be an amazing feature.
Can we change the function names to match with presto

• https://prestodb.io/docs/current/functions/json.html
• https://docs.aws.amazon.com/athena/latest/ug/querying-JSON.html

Rename function name to json_extract_scalar also added a new function to extract json_extract_key to extract the paths for a given json.

[kishoreg]

Can you also add documentation on how json columns are to be used? I suppose the json columns cannot be used in filters (yet) right?

We can know after we added the support for expressions in the filter predicates