Streaming interface to handle connections

[hadley]

Needs to cache first x lines for autodetection, then manage stream of updates. See xml2 for the libxml2 API

[dselivanov]

@hadley , @romainfrancois , is this in short or middle-term plans? or streaming API has very low priority? I'm wondering, because readLines actually becomes bottleneck in my package. Can I somehow help with this?

[hadley]

Relatively low priority currently because it's a lot of work. Not sure if it's possible for anyone to help out - it needs some fairly deep changes to the package.

[dselivanov]

@hadley , sorry for offtopic, but may be you can share some insights on how I can implement fast stream reading from connection? I found only this gist. Any links / ideas / whatever_will_help are very appreciated.

[dselivanov]

Gentle followup. Would really big deal.

[hadley]

This should be easier to tackle now that we have the callback interface. Ideally, we should just be able to pass a connection, and it'll read it chunk by chunk. The trickiness will be matching up bytes from the connection to lines in the reader.