Non-empty pipes #64

Closed
wants to merge 3 commits into
from

Projects

None yet

2 participants

@aristidb
Contributor

Because I have a use-case where there are sources with always at least one output element, I've made this type, which allows safely folding without an initial element.

Some tests are included.

@snoyberg
Owner

I'm not sure how generally useful this is. Are there any examples where this would work without it being isomorphic to foldM1? Can you use it on a data source besides a list without introducing partiality?

@aristidb
Contributor

There is no foldM1 in Data.Conduit.List I think? Either way, that is somewhat beside the point.

These non-empty pipes give you a guarantee of returning at least one element or throwing an exception. Which is something which I think we can often guarantee. For example, consider ... well, that's exactly what I want to do, but I'm sure you can find other similar problems:

When making an AWS request for some list of objects or domains or whatever, AWS often imposes a size limit on the response, and returns a marker that can be used for subsequent requests. So I always know I can make at least one request and get a response (or an error), and after that there might be subsequent responses. If I create a Source of responses, then I throw away that information that there is at least one response. NonEmptySource lets me make that guarantee safely and have it be guaranteed by construction of the type.

(It isn't clear to me how useful non-empty pipes other than sources are, but they were just too much of an obvious generalization not to allow them.)

I have added a convenient unfoldM1 function that lets me generate such non-empty sources in exactly this pattern. I think there should also be no performance penalty the way it is implemented.

One thing that might not work fully yet is handling of leftovers. Which only makes sense for pipes that actually take input. But then when composing with other pipes one would probably first convert to a normal pipe with toPipe, and use the normal mechanism for dealing with that. I just haven't fully thought about that.

@snoyberg
Owner

By foldM1, I meant something similar to Control.Monad.foldM, e.g.:

foldM1 :: Monad m => (a -> a -> m a) -> [a] -> m a

The code I see in your module seems like it's not really doing anything that takes advantage of conduit. Generating a NonEmptySource seems to always consist of taking an existing list of an unfolder and then applying it to fold1. Maybe there are other use cases I'm not seeing.

I guess I'm looking for something more complex that demonstrates that this is a generally useful feature.

@snoyberg snoyberg closed this Jan 31, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment