Support \r\n-style newlines in VAST

[lava]

Support both Unix-, Windows- and Mac-style line endings
for all input formats, either uniformly or mixed.

NOTE: I'd actually like to run benchmarks for this change before merging, because I'm not sure if the libc-readline has some "magic sauce" that makes it far more performant.

[mavam]

This works.

Another way to go after this would be to write a custom line_parser in our parser DSL. It's not as generic as this one, but way faster. For now, what you added is good enough because we will split the I/O path from the parsing soon, so that we only need to focus on buffers of bytes eventually. Then we no longer need this heavy-weight I/O stream machinery from the standard library.

[lava]

@mavam Where would the input to the line_parser come from in this scenario? It looks like the parser would still have to pull out the line byte-by-byte from the ifstream, so i'm not sure where the speedup could come from.

[mavam]

Where would the input to the line_parser come from in this scenario?

You'd always work on a contiguous block of memory. Better cache utilization. Also no virtual function calls.

The new API will forward a chunk that contains a block of lines to the format parser. Then we can put the \r\n logic simply at the end of an existing parser, put a kleene star around it, and have an efficient parser line-based parser.

[mavam]

Switching to this API will still take a bit. @dominiklohmann and I have talked about it extensively and we're looking forward to loop you in on this.

[dominiklohmann]

Switching to this API will still take a bit. @dominiklohmann and I have talked about it extensively and we're looking forward to loop you in on this.

JFYI: Said looping-in has taken place this morning.

On this PR, I think this is a good way to fix for our current API. I'll let @mavam have the last word on this, but all I'm missing here is a changelog entry.

[mavam]

Yep, I think we should mention that we now support \r\n and merge.

(Modulo failing checks.)