Skip to content
100644 257 lines (190 sloc) 11.1 KB
9b19c7b @snoyberg Added conduit README
snoyberg authored
1 Conduits are an approach to the streaming data problem. It is meant as an
2 alternative to enumerators/iterators, hoping to address the same issues with
3 different trade-offs based on real-world experience with enumerators.
5 General Goal
6 ===========================
8 Let's start by defining the goal of enumerators, iterators, and conduits. We
9 want a standard interface to represent streaming data from one point to
10 another, possibly modifying the data along the way.
12 This goal is also achieved by lazy I/O; the problem with lazy I/O, however, is
13 that of deterministic resource cleanup. That is to say, with lazy I/O, you
14 cannot be guaranteed that your file handles will be closed as soon as you have
15 finished reading data from them.
17 We want to keep the same properties of constant memory usage from lazy I/O, yet
18 have guarantees that scarce resources will be freed as early as possible.
20 Enumerator
21 ===========================
23 __Note__: This is biased towards John Millikin's enumerator package, as that is
24 the package with which I have the most familiarity.
26 The concept of an enumerator is fairly simple. We have an `Iteratee` which
27 "consumes" data. It keeps its state while being fed data by an `Enumerator`.
28 The `Enumerator` will feed data a few chunks at a time to an `Iteratee`,
29 transforming the `Iteratee`'s state at each call. Additionally, there is an
30 `Enumeratee` that acts as both an `Enumerator` and `Iteratee`.
32 As a result, there are a few changes to code structure that need to take place
33 in order to fully leverage enumerators:
35 * The `Enumerator`s control code flow. This is an Inversion of Control (IoC)
36 technique.
38 __Practical ramification__: `Iteratee` code can be more difficult to
591fef4 @snoyberg Added clarifications/requirements to
snoyberg authored
39 structure. Note that this is a subjective opinion, noted by many newcomers to
40 the enumerator paradigm.
42 __Requirement__: Nothing specific, likely addressing the requirements
43 below will automatically solve this.
9b19c7b @snoyberg Added conduit README
snoyberg authored
45 * `Iteratee`s are not able to allocate scarce resources. Since they do not
46 have any control of the flow of the program, they cannot guarantee that
47 the resources will be released, especially in the presence of exceptions.
49 __Practical ramification__: There is no way to create an `iterFile`, which
50 will stream data into a file. Instead, you must allocate a file handle
51 before entering the `Iteratee` and pass that in. In some cases, such an
52 approach would mean file handles are kept open too long.
591fef4 @snoyberg Added clarifications/requirements to
snoyberg authored
54 __Clarification__: It is certainly *possible* to write iterFile, but there
55 are no guarantees that it will close the allocated `Handle`, since the calling
56 `Enumerator` may throw an exception before sending an `EOF` to the `Iteratee`.
58 __Requirement__: We need a solution which would allow code something like
aa4ec60 @snoyberg Typo correction
snoyberg authored
59 the following to correctly open and close file handles, even in the presence
591fef4 @snoyberg Added clarifications/requirements to
snoyberg authored
60 of exceptions.
62 run $ enumFile "input.txt" $$ iterFile "output.txt"
9b19c7b @snoyberg Added conduit README
snoyberg authored
64 * None of this plays nicely with monad transformers, though this does not
65 seem to be an inherent problem with enumerators, instead with the current
66 library.
68 __Practical ramification__: You cannot enumerate a file when running in a
69 `ReaderT IO`.
591fef4 @snoyberg Added clarifications/requirements to
snoyberg authored
71 __Requirement__: The following pseudo-code should work:
73 runReaderT (run $ enumFile "input" $$ iterFile "output") ()
9b19c7b @snoyberg Added conduit README
snoyberg authored
75 * Instead of passing around a `Handle` to pull data from, your code should
76 live inside an `Iteratee`. This makes it difficult and/or impossible to
77 interleave two different sources.
79 __Practical ramification__: Even with libraries designed to interoperate
80 (like http-enumerator and warp), it is not possible to create a proper
81 streaming HTTP proxy.
d7b38f7 @snoyberg Nicer ResourceIO
snoyberg authored
83 __Note__: This might actually be possible using the "nested iteratee"
84 technique. I would still posit that this is far too complicated a
85 solution to the problem.
591fef4 @snoyberg Added clarifications/requirements to
snoyberg authored
87 __Requirement__: It should be possible to pass around some type of producer
88 which will be called piecemeal. For example, the request body in Warp should be
89 expressible as:
91 data Request = Request
92 { ...
93 , requestBody :: Enumerator ByteString IO ()
94 }
96 Applications should be able to do something like:
98 bs <- requestBody req $$ takeBytes 10
99 someAction bs
100 rest <- requestBody req $$ takeRest
101 finalAction rest
103 Note that there may be other approaches to solving the same problem, this
104 is just one possibility.
9b19c7b @snoyberg Added conduit README
snoyberg authored
106 * While the concepts are simple, actually writing low-level Iteratee code is
107 very complex. This in turn intimidates users from adopting the approach.
591fef4 @snoyberg Added clarifications/requirements to
snoyberg authored
108 Again, this is a subjective measurement.
110 __Requirement__: Newcomers should be able to easily understand how to use
111 the package, and with a little more training feel comfortable writing their own
112 producers/consumers.
9b19c7b @snoyberg Added conduit README
snoyberg authored
114 Conduits
115 ===========================
117 Conduits attempt to provide a similar high-level API to enumerators, while
118 providing a drastically different low-level implementation. The first question
119 to visit is: why does the enumerator need to control flow of the program? The
120 main purpose is to ensure that resources are released properly. But this in
121 fact solved only *half* the problem; iteratees still cannot release resources.
123 ResourceT
124 ---------------------------
126 So our first issue to address is to create a new way to deal with resource
127 allocation. We represent this as a monad transformer, `ResourceT`. It works as
128 follows:
130 * You can register a cleanup action, which will return a `ReleaseKey`.
132 * If you pass your `ReleaseKey` to the `release` function, your action will be
133 called automatically, and your action will be unregistered.
135 * When the monad is exited (via `runRelease`), all remaining registered actions
136 will be called.
138 * All of this is provided in an exception-safe manner.
140 For example, you would be able to open a file handle, and then register an
141 action to close the file handle. In your code, you would call `release` on your
142 `ReleaseKey` as soon as you reach the end of the contents you are streaming. If
143 that code is never reached, the file handle will be released when the monad
144 terminates.
146 Source
147 ---------------------------
149 Now that we have a way to deal with resources, we can take a radically
150 different approach to production of data streams. Instead of a push system,
151 where the enumerators sends data down the pipeline, we have a pull system,
152 where data is requested from the source. Additionally, a source allows
153 buffering of input data, so data can be "pushed back" onto the source to be
154 available for a later call.
156 Sink
157 ---------------------------
a740377 @juhp correct a few small typos in
juhp authored
159 A `Sink` is the corollary to an `Iteratee`. It takes a stream of data, and can
9b19c7b @snoyberg Added conduit README
snoyberg authored
160 return a result, consisting of leftover input and an output. Like an
161 `Iteratee`, a `Sink` provides a `Monad` instance, which allows easy chaining
162 together of `Sink`s.
164 However, a big difference is that your code needn't live in the `Sink` monad.
165 You can easily pass around your sources and connect them to different `Sink`s.
166 As a practical example, when the Web Application Interface (WAI) is translated
167 to conduits, the application lives in the `ResourceT IO` monad, and the
168 `Request` value contains a `requestBody` record, which is a `Source IO
169 ByteString`.
171 Conduit
172 ---------------------------
a740377 @juhp correct a few small typos in
juhp authored
174 Conduits are simply functions that take a stream of input data and return
9b19c7b @snoyberg Added conduit README
snoyberg authored
175 leftover input as well as a stream of output data. Conduits are far simpler to
a740377 @juhp correct a few small typos in
juhp authored
176 implement than their corollary, `Enumeratee`s.
9b19c7b @snoyberg Added conduit README
snoyberg authored
178 Connecting
179 ---------------------------
181 While you can directly pull data from a `Source`, or directly push to a `Sink`, the easiest approach is to use the built-in connect operators. These follow the naming convention from the enumerator package, e.g.:
183 sourceFile "myfile.txt" $$ sinkFile "mycopy.txt"
184 sourceFile "myfile.txt" $= uppercase {- a conduit -} $$ sinkFile "mycopy.txt"
185 fromList [1..10] $$ (+ 1) =$ fold (+) 0
187 Trade-offs
188 ===========================
190 Overall, the approach achieves the goals I had hoped for. The main downside in
191 its current form is its reliance on mutable data. Instead of having an
192 `Iteratee` return a new `Iteratee`, thereby provide an illusion of mutability,
193 in conduit the sources and sinks must maintain their state internally. As a
194 result, code must live in IO and usually use something like an IORef to keep
195 track of the current state.
197 I believe this to be an acceptable trade-off, since:
199 1. Virtually all conduit code will be performing I/O, so staying in the `IO`
200 monad is reasonable.
201 2. By using `monad-control`, conduit can work with any monad *based* on `IO`,
202 meaning all standard transformers (except `ContT`) can be used.
203 3. Enumerator experience has shown that the majority of the time, you construct
204 `Iteratee`s by using built-in functions, such as fold and map. Therefore,
205 the complication of tracking mutable state will usually be abstracted from
206 users.
208 Another minor point is that, in order to provide an efficient `Monad` instance,
209 the `Sink` type is complicated with tracking two cases: a `Sink` which expects
210 data and one which does not. As expressed in point (3) above, this should not
211 have a major impact for users.
213 Finally, since most `Source`s and `Sink`s begin their life by allocating some
214 mutable variable, both types allow some arbitrary monadic action to be run
215 before actual processing begins. The monad (et al) instances and connect
216 functions are all built to run this action once and then continue operation.
218 Status
219 ===========================
221 This is currently no more than a proof-of-concept, to see the differences
222 between enumerators and conduits for practical problems. This may serve as a
223 basis for WAI and Yesod in the future, but that will only be after careful
224 vetting of the idea. Your input is greatly appreciated!
d9914af @snoyberg Added some notes
snoyberg authored
226 Notes
227 ===========================
229 This is just a collection of my personal notes, completely unorganized.
231 * In enumerator, it's relatively easy to combined multiple `Iteratee`s into
232 an `Enumeratee`. The equivalent (turning `Sink`s into a `Conduit`) is
233 harder. See, for example, chunking in http-conduit. Perhaps this can be
234 improved with a better `sequence`.
236 * Names and operators are very long right now. Is that a feature or a bug?
22d9341 @snoyberg More README notes
snoyberg authored
238 * Should we use Vector in place of lists?
240 * It might be worth transitioning to RegionT. Will the extra type parameter
241 scare people away?
243 * Perhaps the whole BSource/BConduit concept doesn't need to be exposed to
244 the user. Advantage of exposing: it makes it obvious at the type level that
245 a source/conduit can be reused, and possibly more efficient implementations
246 (no double buffering). Disadvantage: more functions to implement/user to
247 keep track of, so harder to use.
4b60dca @snoyberg MonadIO instance
snoyberg authored
249 * I dislike the travesty which is `type FilePath = [Char]`, so I'm using the
250 system-filepath package. I've used it for a lot of internal code at work,
251 and it performs wonderfully. If anyone is concerned about this approach,
252 let me know.
ccb4652 @snoyberg A bunch of renames
snoyberg authored
254 * Should we rename ConduitM to Conduit (et al), and then give Conduit a name
255 like ConduitRaw? After all, users interact with the current "M" versions
256 more often than anything else.
Something went wrong with that request. Please try again.