Changes JSON reader's recovery option's behaviour to ignore all characters after a valid JSON record

[elstehle]

Description

Closes #14226.

The new behvior of JSON_LINES_RECOVER will now ignore excess characters after the first valid JSON record on each JSON line.

{ "number": 1 } 
{ "number": 1 } xyz
{ "number": 1 } {}
{ "number": 1 } { "number": 4 }

Implementation details:
The JSON parser pushdown automaton was changed for JSON_LINES_RECOVER format such that when in state PD_PVL (post-value, "I have just finished parsing a value") and when the stack context is ROOT ("I'm not somewhere within a list or struct"), we just treat all characters as "white space" until encountering a newline character. post-value in stack context ROOT is exactly the condition we are in after having parsed the first valid record of a JSON line. Thanks to @karthikeyann for suggesting to use PD_PVL as the capturing state.

As the stack context is generated upfront, we have to fix up and correct the stack context to set the stack context as ROOT stack context for all these excess characters. I.e., (_ means ROOT stack context, { means within a STRUCT stack context):

in:    {"a":1}{"this is supposed to be ignored"}
stack: _{{{{{{_{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{

Needs to be fixed up to become:

in:    {"a":1}{"this is supposed to be ignored"}
stack: _{{{{{{__________________________________

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[elstehle]

This is just a minor improvement that I piggy-backed into this PR. The delegate that implements translation_op has no benefit from the extra reference of *this, so we removed it from the arguments.

[bdice]

Just a couple small pieces of feedback. I think this makes sense. The overall complexity of the FST code is getting higher and higher, but I'm not sure if we can do much about that while continuing to add requested features.

[elstehle]

. The overall complexity of the FST code is getting higher and higher, but I'm not sure if we can do much about that while continuing to add requested features

Yes, agreed. Unfortunately, the Pushdown Transducer for a JSON parser is quite complex by nature already. For most of the remaining Spark requests on our radar we will be introducing extra pre-/post-processing steps rather than changing the Pushdown machinery.

[karthikeyann]

Great work! 🚀 Especially with the additional processing step without complicating the existing JSON FST table.

[GregoryKimball]

@andygrove would you please evaluate this solution?

[andygrove]

@andygrove would you please evaluate this solution?

@GregoryKimball I have confirmed that this resolves the issue. The plugin tests for this issue now pass when run against this PR.

[elstehle]

/merge