Incremental input

[Marwes]

It would be nice to support reading from streams where the input was produced incrementally to avoid needing to read the entire input into memory (such as from files). Since Stream types must be able to be cloned which makes it simple to support arbitrary look-ahead but it makes it impossible to use an iterator such as ::std::io::Chars.

Without arbitrary look ahead it would be trivial to just support LL(1) parsers by adding a peek function to Stream but I don't think its worth it given how useful the try parser can (even if it is inefficient).

Not quite sure how to implement this efficiently yet though.

[michaelsproul]

I'm interested in getting this working as I'd like to use parser combinators to parse XML from a serial stream...

One idea I had was to use an intermediate struct in place of the input - something that buffers input items as they're read.

struct BufferedStream<Input, Stream> {
    buffer: Vec<Input>,
    stream: Stream
}

Then parsers could read from the buffer directly if it were non-empty, or else pull a new item from the stream. It prevents the need to "reverse" changes to the stream if a pulled token ends up being unparseable, as it can simply be thrown onto the buffer (no stream cloning required). I will admit I haven't given this a whole lot of thought though... I'll play around with it a bit more.

[michaelsproul]

Or we could steal a bunch of ideas from nom, which seems to have streaming but isn't generic.

[Marwes]

I might be wrong but I don't think your BufferedStream handles parsers which can consume multiple tokens before failing. If you put the token in the buffer if it fails there might still have been tokens which has already been accepted which can't be recovered at the point that parsing is resumed.

I am not sure that the way nom handles streaming works well with recursive descent parsers (which is what parser combinators are, at least in this library). If a parser needs more input in nom I think it basically throws away everything it has parsed up to that point, requests more and then continues once it gets it (I might be wrong though) if you are deep within a parser when that point lots of work would be repeated.

[Marwes]

@michaelsproul I have a sketch for how this might be implemented in #37. I think it is rather close to what you had in mind, any comments?

[Marwes]

Closed by 2.0.0