Looking for advice in trying to decide on a streaming solution

[david-janssen]

I recently asked a question on reddit's r/haskell (https://www.reddit.com/r/haskell/comments/18nkdt1/streaming_libraries_and_adhoc_pipelines_question/) and was recommended dunai and invited to ask for more clarification here. I have since tried dunai and the first impression is really positive. However, there are 4-5 reasonable pipes/streaming solutions for haskell, and some of the devs behind those libraries are vehemently convinced that their solution is clearly better, so it's been a bit daunting settling on a solution. I would like to make use of your offer to help me figure out the right solution here, so let me describe what I'm doing.

I'm the author of kmonad (https://github.com/kmonad/kmonad), which I wrote as a sort of first challenge to myself to make a useful haskell app work (after a lot of little toy examples). It ended up doing unexpectedly well. However, since it's release I learned a lot about managing software projects (particularly the kinds of desires/demands that users might have). Additionally, I ended up falling more and more ill, and have basically been unable to focus on anything for the last 3 years. Recently, my health-situation has started slowly to improve, and I've moved from a few hours of focus a month, to a few hours a couple of days a week.

I've started orienting myself to writing an app that tries to fill the same niche as kmonad (keyboard remapping) but with a couple of serious improvements that all revolve around moving some of the decisions/implementations from inside the compiled haskell into the user config files. Basically, I want to define a number of basic event-stream computations that can be composed by users into event-stream transformations so as to define input-event remappings. I am happy to explain more, but I also don't want to ramble on, so I stop the general description here.

Concretely, what I am trying to build is:

• a stream of user-configurable events that have
  • a fixed time at which they occur
  • a configurable 'payload' (i.e. users define event-sources in the config, and say what values they consist of and assign labels to these values. known only at runtime, not compile-time, but won't change after parsing the configuration.)
• a collection of stream transformation combinators that users can use to construct more complicated transformations. (since these are configurable, the network of stream-transformation functions is also not known at compile time, but will be static after configuration parsing is completed).

The streams need to be able to 'stretch' and 'compress', i.e: there are situations where multiple input stream elements are used to produce 1 output element, and conversely, sometimes single input-stream elements are used to produce multiple output elements. Additionally, the streams sometimes need to recurse. I found the feedback function in dunai, so the latter seems doable. I'm not sure what the 'synchronized' means in the MSF-description as "synchronized stream functions with side effects.", it makes me think the former might be more difficult.

Finally, I am concerned about performance. For most event-remappings the volume of events is actually quite low and I could get away with some inefficiency, but I've also tried to remap mouse-movement events, and that can actually be quite high-volume; not problematically so, but I also don't think people would want to dedicate a sizable percentage of their CPU to their input-remapper.

Sorry if this is all a bit rambly and mixed-up. That's currently the state of my mind :-). I look forward to any feedback around my situation, the problem I am sketching, or other general advice. Please feel free to ask for clarification.

Best,

D

[turion]

Hi @david-janssen , glad you found your way here! I think the right choice has a lot to do with the relation and synchronicity between output and input, as you've already observed.

• dunai's MSFs are fundamentally synchronous, which means that they consume one input for every output.
• In contrast, pipes and conduit are "internally" asynchronous, they never output or input at the same time, instead the pipe internally decides whether to accept input or produce output next.
• There is a third option: In my library rhine (which is a compatible extension to dunai), there are "externally" asynchronous components where a clock or event source decides whether output or input must happen. See https://hackage.haskell.org/package/rhine-1.1/docs/FRP-Rhine-ResamplingBuffer.html for components which can consume and produce data at different, external rates.

You can emulate such a resampling buffer with feedback, but it is a bit cumbersome. It would look something like this:

processBuffer :: MSF m (Maybe Event, [Event]) (Maybe Output, [Event])
processBuffer = arr $ \case ->
  -- We have received a new event, but we can't process it right now
  (Just event, events) -> (Nothing, event : events)
  -- No event, but external trigger to produce output
  (Nothing, events) -> (Just $ produceOutput events, [])

process :: MSF m (Maybe Event) (Maybe Output)
process = feedback processBuffer

You would then have to drive this MSF by feeding it Nothing when output is required, or Just Event when an event arrives. It's possible, but error-prone, and I've developed rhine to deal with such situations in a principled way. There you have it, yet another streaming library author promotes their library as the best solution ;)

[david-janssen]

Hello Turion,

Thank you very much for your advice. I'm afraid I won't be trying out your library anytime soon, since I ended up experimenting more with pipes and it ended up being really quite satisfactory. If I do end up running into a problem that just seems unfixable with the pipes solution, then rhine will be the first thing I explore :-)

All the best,

D

[david-janssen]

As mentioned above, I ended up experimenting more with the various libraries, and ended up getting very satisfactory behavior from pipes. I will still keep my eye on this thread and am very happy to hear input and ideas, but I am not currently stuck anymore.