[FEA] Add whitespace removal as a JSON reader preprocessing option

[GregoryKimball]

Is your feature request related to a problem? Please describe.
In cases where libcudf returns a JSON field as an unparsed string (see #14572), the string includes any unquoted whitespace present in the original JSON character data. This presents a challenge for downstream processing of mixed types, especially in the Spark-RAPIDS plugin where we match Spark's behavior to always remove unquoted whitespace. It also allows better control in the case where a user requests a hierarchical type as a string type.

@revans2 Would you please comment on other benefits of having this reader option available?

Describe the solution you'd like
We would like a JSON reader option and cuIO utility function that uses an FST instance to remove unquoted whitespace from input character data.

Describe alternatives you've considered
Document the whitespace differences as a GPU vs CPU difference and don't migrate customer queries that include mixed types and unquoted whitespace.

Additional context
We have a similar whitespace normalization challenge for the results of calls to get_json_object (see NVIDIA/spark-rapids#10218), and an FST-based whitespace normalization tool could be part of the solution for that problem as well.

[revans2]

The main thing for us is to try and match what Spark does with white space so we can return bit for bit identical results. This would happen any time that we ask for a column to be returned as a string, but they underlying data types are not actually quoted strings.

Here are a few examples of what Spark does with normalization of the white space.

+------------------------------------------------+-------------------------------------+
|jsonStr                                         |normalized                           |
+------------------------------------------------+-------------------------------------+
|{"a":50}                                        |{"a":50}                             |
| { "a" : 50 }                                   |{"a":50}                             |
|{" a ":50}                                      |{" a ":50}                           |
|{"a": [1, 2, 3, 4, 5, 6, 7, 8], "b": {"c": "d"}}|{"a":[1,2,3,4,5,6,7,8],"b":{"c":"d"}}|
|{"a":\t"b"}                                     |{"a":"b"}                            |
+------------------------------------------------+-------------------------------------+

Please note that in the above example \t represents the tab character and not the two character sequence '' and 't'.

Technically Spark will also strip out carriage return and line feed (\r and \n) characters, but that would have to be dependent on JSONLines being enabled as '\r' and '\n' are a hard separator for us when parsing JSON lines format.

I tied multiple unicode which space characters as well as \f and \v, which resulted in Spark saying that the result is invalid JSON. So I would want to be sure that we do not remove these from the result as we might turn something that was invalid JSON into valid JSON.

[shrshi]

Thank you very much for these examples @revans2!
Can the inputs be invalid JSON strings? I'm trying to understand what the normalized output would be in the following example

{"a" : "b }

[GregoryKimball]

Closed by #15033