streams as source for matchAll

[ThomasDeutsch]

I would like to use a node.js stream ( https://github.com/substack/stream-handbook ) as a source stream for the matchAll() source parameter.

Would this be possible?

With browserify ( https://github.com/substack/node-browserify ) this could also be used on the client.

[Page-]

I think it would be possible, but tricky and situational, to implement support for streams in OMeta, although it'd be great to support them. The issue is that OMeta expects to be able to backtrack at any given point (potentially all the way to the start), so whilst it could feasibly build up the input from a stream (and read as necessary) it would need to store the whole input in order to be able to backtrack which would defeat the purpose.

I think this could be solved for input streams to a point where it would useful by having support to drop any input which cannot be backtracked to (whenever we're not in an _or, _lookahead or _xor I believe would be fine). Of course for this to be useful it would also require the grammar to be written in such a way as to avoid going into one of these branches that can backtrack for very long at any point, but it would at least mean it would be possible to take advantage of streams given the appropriate care.

To return a stream however would require a major change to the OMeta syntax, to allow building up the return value of a rule gradually, so that whenever you're not in a branch then that could be funnelled back up the rule chain through streams. I think this would be possible, but it'd probably require rewriting the ometa-js core to build streams into every point.

[CMCDragonkai]

Is the advantage of using streams so that OMeta could potentially compile a huge source file without loading the file all into memory?

Your description about how we can drop input that can no longer be backtracked to, sounds very similar to memory management techniques such as garbage collection or RAII.

[Page-]

Yeah, not needing to load all the input at once would be the advantage, you could save memory by dropping the input when it's no longer needed and also start processing sooner (eg downloading a file and starting parsing instantly, rather than waiting for the download to complete

[CMCDragonkai]

Immediate processing sounds like you could then implement http://oboejs.com/ in OMeta! Or even streaming CSV. Or parsing binary files via streaming too.