Operations on file contents should be composable

[quintesse]

Right now we have a copyLines() method which works more or less as a map() for reading from one file and writing the mapped result to another file.
I was thinking about adding something like filterLines() where you can decide which lines to copy or leave out.

But when I thought about doing something that required both I realized that the current design is not composable at all. We really need a way to access a file's contents like an Iterable and have access to all it's useful functionality.

The problem of course is that right now Reader could not actually be an Iterable<String> because it's assumed that creating an Iterator is cheap and repeatable, which in the case of streams of data might/is not the case.

Maybe we need a different Iterable-like interface that implements similar methods for one-shot resource streams. Or maybe the implementation of Iterable con be extended to include one-shot resource streams. That should be investigated.

[gavinking]

We already have a proposal for closeable iterators, where the for construct would do what try does today.

Sent from my iPhone

On 29 May 2014, at 12:57 pm, Tako Schotanus notifications@github.com wrote:

Right now we have a copyLines() method which works more or less as a map() for reading from one file and writing the mapped result to another file.
I was thinking about adding something like filterLines() where you can decide which lines to copy or leave out.

But when I thought about doing something that required both I realized that the current design is not composable at all. We really need a way to access a file's contents like an Iterable and have access to all it's useful functionality.

The problem of course is that right now Reader could not actually be an Iterable because it's assumed that creating an Iterator is cheap and repeatable, which in the case of streams of data might/is not the case.

Maybe we need a different Iterable-like interface that implements similar methods for one-shot resource streams. Or maybe the implementation of Iterable con be extended to include one-shot resource streams. That should be investigated.

—
Reply to this email directly or view it on GitHub.

[quintesse]

What @gavinking refers to is ceylon/ceylon-spec#895

[tombentley]

While I think closeable iterators would work nicely for things like ResultSet (where there's a Transaction going on), I'm not so sure about stuff like iterating the lines read from a file or network socket. To work with for I need an Iterable, but the only way to do that for @quintesse's case would be to buffer what's read from the Reader, but since we've no idea how much data might be read that's probably a bad idea.

[luolong]

What about implementing Iteratees?

They are very composable and versatile...

Roland
29.05.2014 13:57 kirjutas kuupäeval "Tako Schotanus" <
notifications@github.com>:

Right now we have a copyLines() method which works more or less as a map()
for reading from one file and writing the mapped result to another file.
I was thinking about adding something like filterLines() where you can
decide which lines to copy or leave out.

But when I thought about doing something that required both I realized
that the current design is not composable at all. We really need a way to
access a file's contents like an Iterable and have access to all it's
useful functionality.

The problem of course is that right now Reader could not actually be an
Iterable because it's assumed that creating an Iterator is cheap
and repeatable, which in the case of streams of data might/is not the case.

Maybe we need a different Iterable-like interface that implements similar
methods for one-shot resource streams. Or maybe the implementation of
Iterable con be extended to include one-shot resource streams. That
should be investigated.

—
Reply to this email directly or view it on GitHub
#241.

[quintesse]

@luolong Don't know, reading about them now :)

[FroMage]

Note that while we still haven't properly unified the ceylon.file and ceylon.io modules, the Reader you get in ceylon.file will not work for binary files or files with a non-default encoding. This part is provided by ceylon.io and that's where any support for reading files line by line should be added. ceylon.file is about the file system, not about IO. Remember that you can also want to read sockets line by line.

[FroMage]

If ceylon.io and ceylon.file could have circular dependencies, we would move the Reader/Writer interfaces in ceylon.io where they belong.

[gavinking]

I guess I would prefer to remove Reader, and use a closeable iterator instead. But I don't see any way to get rid of Writer.

[quintesse]

Well even in the case of Reader there might be times that one would want more control over the way one want to read the input, right? The iterator would probably be the most used for the majority of cases (simply reading from beginning to end unless aborted before getting there), but the Reader might be used for those who need to be able to reset or jump to specific places etc. ?

[gavinking]

Well it doesn't offer any such facility now.