A port to Scala of the Conduit library by Michael Snoyman
Scala Shell
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
benchmark/src/main/scala/conduits/benchmark
bytestring/src
conduits/src
examples
http/src
network
project
resourcet/src
text
.gitignore
.sbtrc
.travis.yml
README.md
sbt

README.md

This is a direct port to Scala of Micheal Snoyman's Conduit library. See Conduits are an approach to the streaming data problem. It is meant as an alternative to enumerators/iterators.

Build Status

The most up-to-date documentation is available as an appendix of the Yesod book, at: http://www.yesodweb.com/book/conduits. Below follows a brief description of the core concepts.

Summary

Streaming data allows us to process large datasets without pulling all values into memory. Conduits allow us to process large streams of data while retaining deterministic resource handling They provide a unified interface for data streams, whether they come from files, sockets, or memory.

A Monad Transformer, ResourceT, can be used to safely allocate resources, knowing that they will always be reclaimed- even in the presence of exceptions.

Core classes

  • Source. A producer of data. The data could be in a file, coming from a socket, or in memory as a list. To access this data, we pull from the source.

  • Sink. A consumer of data. Basic examples would be a sum function (adding up a stream of numbers fed in), a file sink (which writes all incoming bytes to a file), or a socket. We push data into a sink. When the sink finishes processing it returns some value.

  • Conduit. A transformer of data. The simplest example is a map function. Like a sink, we push data into a conduit, but instead of returning a single value at the end, a conduit can return multiple outputs every time it is pushed to.