Get the span as a result of a match

[charles-paperman]

Given a string in RAM, it would be nice to collect all the span of the result so that they could be parsed and loaded with an external solution.

[github-actions]

Tagging @V0ldek for notifications

[V0ldek]

We already have matches which returns a Match that includes the MatchSpan, but it includes the copied bytes as well. I understand the request is about a zero-copy endpoint.

Unfortunately, we can't exactly get zero-copy, because of the additional work required to get the end of a match – starting indices are easy (hence indices exists), but to get the end we need to store the matched bytes and then trim the trailing whitespace; so the copy would happen anyway, even if we didn't return it to the caller.

However, if the purpose is for external to parse the results, then trimming the whitespace is probably not required. I'd expect any parser to just discard the trailing whitespace. Maybe we can expose this and explicitly document that it's an approximate span, where the ends might point to some whitespace after the actual match. Call it approx_spans? That would be significantly faster than the full matches function now.

[V0ldek]

To make the issue concrete, there are currently two real cases where a match includes whitespace that needs to be trimmed. One, if the JSON is weird and has spaces before commas delimiting items in an object or list:

{
  "a": 42   ,
  "b": 43
}

with query $.a would match 42␣␣␣. or

[ 42  , 43 ]

where $[0] would match 42␣␣␣.

This is a corner-case, since no sane person or formatter writes JSONs like that. The other happens all the time, however, and it's when the match happens at the end of an object/list and so it will include all whitespace until the parent's closing brace/bracket:

{
  "a": {
    "b": 42
  }
}

where $.a.b would match 42⏎␣␣.

In the last case, the correct span to report is 20-21, but to get that we need to trim the whitespace after the match. In the fast approximate span endpoint the report would be 20-25, which would include all the whitespace.

[V0ldek]

@charles-paperman Is there value in adding this to rq proper as an output mode or is the use case purely for the library?

[charles-paperman]

I believe it is purely for the lib. I don't think reporting the span in rq makes a lot of sense.

[V0ldek]

So the API here would be:

fn approx_spans<I, S>(&self, input: &I, sink: &mut S) -> Result<(), EngineError>
       where I: Input,
             S: Sink<MatchSpan>;

Internally, a different recorder that would eschew all the byte-copying and whitespace trimming and work with reported offsets only. It'd still need to use the stack to make sure the responses are ordered properly, we should follow a similar design to NodesRecorder.

The documentation MUST explicitly and prominently state that the spans will be inaccurate at the ends, but with the guarantee that only trailing JSON whitespace characters are included.

[charles-paperman]

Efficient approximate copy-free span would be neat!

[V0ldek]

Implemented in v0.8.1 as approximate_spans.