[BUG] GetJsonObject sees a double quote in a single quoted string as invalid #10219
Assignees
Labels
bug
Something isn't working
Comments
revans2
added
bug
revans2
changed the title
|
@SurajAralihalli put in some help trying to debug this on the CUDF side and it looks like single quote support is off by default. And wouldn't you know it. The java API does not support configs so we get the default configs that do not support single quotes. We need to fix the JNI API to let configs be passed in. We need to enable single quote support in our operator, and we need to add more tests to verify that it is working properly with single quotes. |
3 tasks
rapids-bot bot
pushed a commit
to rapidsai/cudf
that referenced
this issue
Feb 12, 2024
Resolves [10219](NVIDIA/spark-rapids#10219) This PR introduces a new class named `GetJsonObjectOptions` that holds the configurations to control the behavior of the underlying `cudf::get_json_object` function. It incorporates this new class into the `getJSONObject` JAVA API as an additional argument but also keeps the previous API to maintain backwards compatibility. It also includes a test case, `testGetJSONObjectWithSingleQuotes`, validating the behavior of `getJSONObject` when single quotes are enabled. Authors: - Suraj Aralihalli (https://github.com/SurajAralihalli) Approvers: - Robert (Bobby) Evans (https://github.com/revans2) - MithunR (https://github.com/mythrocks) - Karthikeyan (https://github.com/karthikeyann) URL: #14956
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
With input data of
{'a':'A"'}, and a json path of$, Spark will output{"a":"A\""}, but we output null, meaning it was an error and we could not parse it. I'm not sure if it has something to do with it being a single double quote and quote matching is a problem or what because['a','b','"C"'], will fail because of #10218 not because it returned None.The text was updated successfully, but these errors were encountered: