New design for Producers and Consumers

[Geal]

The actual design for producers and consumers is not very flexible, and very slow. For the 1.0 version, they should be modified to allow for better streaming patterns.

List of current problems:

• pusher!, Stepper, AccReader and Consumer all have their own buffer to partially consume data and accumulate more bytes. This results in a lot of clearing and filling of buffers
• the pusher macro is not really useful
• the Consumer pattern is confusing, since it handles data consumption in a loop. What most people ask for currently is the Stepper.
• the Producer's buffer size is fixed, it cannot change once we know more about the format
• the signals between Producer and Consumer could make it easier to indicate more data, seeking needed, etc

Ideas for the new design:

• I really like how Haskell's pipes and conduit work, the yield and await calls are useful patterns. Unfortunately, they are not doable in Rust without either a coroutine implementation, or a syntax extension writing a complete state machine. But we can still take inspiration from those projects
• the Producer's buffer should handle data accumulation and consumption. The new API could look like AccReader's, with a a BufRead implementation to indicate how much data has been consumed. This would avoid a lot of data copy in the rest of the pipe
• the Producer's buffer size should be changeable. This implies that data must be copied, of course
• there could be a "middle" component, like a pipe, and a way to combine them like pipes or conduit do it, such as Producer + Pipe == Producer, Pipe + Pipe == Pipe, Pipe + Consumer == Consumer
• more error reporting for Stepper and Consumer

any other ideas?

[Geal]

Fixed in 1.0