New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Push-Pull proposal: Signals to complement Streams #581

Closed
staltz opened this Issue Apr 23, 2017 · 147 comments

Comments

@staltz
Member

staltz commented Apr 23, 2017

TL;DR: create a Signals library besides xstream, use it as the expected sink in some drivers


Problem

Over the weekend in CycleConf 17 we started discussing how to better accommodate usage of inherently pull values like Math.random() or window.outerHeight into Cycle.js apps in a way that follows the framework's way of working.

Currently you would have to do something nasty like click$.map(() => Math.random()) which is impure but most importantly, makes testing a bit worse since it's non deterministic and we're not into doing invasive mocking of globals.

I noticed a similarity between these values, is that they all look like () => T, e.g. Math.random(): number. Then I drafted this on a paper:

  • PURE: f(x: A): B
  • PUSH: f(x: A): void (used for sending events e.g. observer.next(ev): void)
  • PULL: f(): B (used for reading values)
  • SINK-ONLY DRIVER: f(x$: Stream<A>): void
  • SOURCE-ONLY DRIVER: f(): Stream<B>
  • MAIN (or what @TylorS calls component): f(x$: Stream<A>): Stream<B>

Most of these bullet points are covered in Cycle.js, except PULL. That's how I noticed it was a missing piece. After discussing with others, watching a talk by Erik Meijer, revising the Push-pull FRP paper by Conal Elliott, I noticed we just need to have a functional ES6 Iterable library. For a while I investigated Dominic Tarr's pull streams and came to the conclusion it's basically AsyncIterable as per Erik Meijer. For a while we also got quite worried about polling given what Conal Elliott described as the biggest problem with pull collections to represent continuous values over time. We thought if there should be a push-and-pull collection (e.g. to represent the virtual DOM over time, which is a "continuous" value which emits updates), and noticed that perhaps Derivable.js by David Sheldrick or Menrva could fill that gap.

Then, @Widdershin suggested we might not even need a push-and-pull hybrid for the DOM Driver if we have an entirely pull-driven approach with requestAnimationFrame polling. Nick suspected it would be even more better than the current reactive approach.

We started hacking and came with a proof of concept here: https://github.com/staltz/experiment-push-pull-cyclejs

function main(sources: Sources): Sinks {
  const vdomS = sources.windowHeight.map(height =>
    div('.foo', 'Height: ' + height),
  );

  return {
    DOM: vdomS,
  };
}

run(main, {
  windowHeight: () => Signal.from(() => window.outerHeight),
  DOM: makeDOMDriver('#main-container'),
});

Notice the new convention x$ is a Stream and xS is a Signal.

The example works with rAF and demonstrates that we could essentially model values-over-time as signals, with the same type of functional composability with operators as in xstream or rx.

What next

Having Streams and Signals in Cycle.js would be a revolutionary change that requires a lot of careful planning. We don't want to find a show-stopper and be embarrassed after launching this change. For now we have no guarantee whatsoever that this will actually be a thing in Cyclejs.

Publish the signal library as a standalone ysignal in release candidate or v0.x for wild experimentation. ysignal should be "hot" in the same sense as xstream because we need the assumption of "one execution per stream" and "one execution per signal" so that the DevTools dataflow graph displays something that makes sense in people's mind. A hot Signal (implements Iterable) would mean there is a shared Iterator which is lazily created and destroyed with refCounting.

We need to consider should we remove MemoryStreams from xstream because so far it has been used to represent values over time, and with Signals we wouldn't need MemoryStreams anymore. Currently MemoryStreams are pull on subscribe, push afterwards, so they have a pull ability. With Signals dedicated for pull, we might not need to mix pull and push. We can have Streams only for push, and Signals only for pull.

xstream and ysignal could be tightly interrelated "sibling" libraries if they have conversion operators from one to the other and vice-versa. Not sure if they should do that, or if they should just assume conversion with Iterables and ES Observables.

On the topic of conversion, I noticed Push-to-Pull then Pull-to-Push is not an isomorphism. I don't know what to call that in category theory but all I'm saying is that when you go from Push to Pull, the Pull producer can choose what and how much data to replay, and when you go from Pull to Push, the Push producer can choose when to emit data. This liberty of choice means information is lost during conversion.

We need to investigate further if we need a hybrid type in Menrva/DerivableJS style. Related to that, since those libraries do intelligent computation of derived values in the signal graph, we might need that ranking algorithm (maybe we don't need the Push part of being hybrid).

We could map what drivers would expect Signal sinks and what drivers would expect Stream sinks.

DOM Driver isolation and event delegator event Streams based on Element Signals may be a challenging problem.

Need to also consider how to support stream lib Diversity.

Nick Johnstone could comment on his ideas how to use this with cycle/time.

@staltz

This comment has been minimized.

Member

staltz commented Apr 24, 2017

Relevant content being cooked by Matt Podwysocki https://gist.github.com/mattpodwysocki/34e76e199442971885a4f48c0730aa40

@SteveALee

This comment has been minimized.

Contributor

SteveALee commented Apr 24, 2017

Can you define a signal in this context for those who missed the discussion?. I've come across in various RTOSs when it is often an event without any value. It seems you mean something that is triggered when an internal value changes? Ie something not declared as a stream

@SteveALee

This comment has been minimized.

Contributor

SteveALee commented Apr 24, 2017

I think you are saying that signal.from() uses mechanisms like polling to create a reactive 'stream' from a value that changes over time but has no natural push semantics? Signals would do this in a generic way so we wouldn't just great a driver each time.

From Could use different 'schedulers' for the example it could poll based on a dom event.

Is that close?

@aronallen

This comment has been minimized.

Contributor

aronallen commented Apr 25, 2017

I can see the example with Math.random() (the consumer fetches a random number)
But I think window.outerHeight is more push based, there is a resize event on the window whenever the size changes.

@staltz

This comment has been minimized.

Member

staltz commented Apr 25, 2017

But I think window.outerHeight is more push based, there is a resize event on the window whenever the size changes.

That's true, but try to consider the general case of some property that changes over time but there is no available event that notifies that.

@borisirota

This comment has been minimized.

borisirota commented Apr 25, 2017

That's true, but try to consider the general case of some property that changes over time but there is no available event that notifies that.

@staltz as far I understand you are talking about integrating properties that changing over time into the current push mechanism of cycle.

But how does Math.random() which is a pull based action (or any other Question & Answer type of side effect as you described in this comment) fit in the proposed signal mechanism ?

Is this what you was after when described

The goal in Cycle.js is not necessarily to be as concise as possible. It's instead to make the dataflow explicit, for readability and predictability. How to do this properly for any effects is what I'm after.

in this comment ? :)

Thanks !!

@SteveALee

This comment has been minimized.

Contributor

SteveALee commented Apr 25, 2017

I can only think that the Signal factory has to specify how and when sampling is done - eg some trigger generated in a stream, or a regular poll, or a driver.

Possibly analogous to RxJS schedulers?

@staltz

This comment has been minimized.

Member

staltz commented Apr 25, 2017

But how does Math.random() which is a pull based action fit in the proposed signal mechanism ?

 run(main, {
-  windowHeight: () => Signal.from(() => window.outerHeight),
+  windowHeight: () => Signal.from(() => Math.random()),
   DOM: makeDOMDriver('#main-container'),
 });

But how does ... any other Question & Answer type of side effect as you described in this comment ... fit in the proposed signal mechanism ?

QA type of side effect is a different type of problem which (I believe) Pull Signals have nothing to do with. I'd be glad to be proven otherwise.

I can only think that the Signal factory has to specify how and when sampling is done

Not how and when. Only how.

@borisirota

This comment has been minimized.

borisirota commented Apr 25, 2017

@staltz so if I'm getting it right, the edited version of windowHeight will cause infinite flow of random numbers into main ?
And if I want to get individual random number in a specific point in time (Question & Answer type of side effect), Random driver is the solution ?

@SteveALee

This comment has been minimized.

Contributor

SteveALee commented Apr 25, 2017

Not how and when. Only how.

ah, how defines when :)

@staltz

This comment has been minimized.

Member

staltz commented Apr 25, 2017

so if I'm getting it right, the edited version of windowHeight will cause infinite flow of random numbers into main ?

Yes

And if I want to get individual random number in a specific point in time (Question & Answer type of side effect), Random driver is the solution ?

randomNumS.sample(x$) returns a stream that gets a random number only at the times when x$ stream emits. Then, given that stream, you could do startWith(0) to get a signal, and pass that to the DOM driver expecting a signal as sink.

ah, how defines when :)

Well, not really. With signals, the producer is not in charge of when data will be produced. The consumer is in charge.

http://reactivex.io/rxjs/manual/overview.html#pull-versus-push

What is Pull? In Pull systems, the Consumer determines when it receives data from the data Producer. The Producer itself is unaware of when the data will be delivered to the Consumer.

What is Push? In Push systems, the Producer determines when to send data to the Consumer. The Consumer is unaware of when it will receive that data.

@staltz

This comment has been minimized.

Member

staltz commented Apr 25, 2017

And if I want to get individual random number in a specific point in time (Question & Answer type of side effect), Random driver is the solution ?

PS: I wouldn't call that a QA type of side effect because this type of pull system is still synchronous. We are using an Iterable, not an AsyncIterable. With AsyncIterables you would be able to do HTTP request and response.

@whitecolor

This comment has been minimized.

Contributor

whitecolor commented Apr 25, 2017

How all this Signal thing would be better/different than a stream based driver with that can pull the value from source, as it was discussed before for example for HTTP case:

HTTP.pull(xs.periodic(100).mapTo('google.com'))

What will it allow to do what is not possible now, how it would really improve/change the current state of things?

@borisirota

This comment has been minimized.

borisirota commented Apr 26, 2017

randomNumS.sample(x$) returns a stream that gets a random number only at the times when x$ stream emits. Then, given that stream, you could do startWith(0) to get a signal, and pass that to the DOM driver expecting a signal as sink.

@staltz can you please define the meaning of signal as you see it ?

I wasn't familiar with it and after some reading, from what I understand, the meaning is representation of properties that changing over time. Now, Math.random() is not a property which changes over time so why implement it as signal ? Its a little bit confusing.

I do understand the benefit of this as integrating properties that changing over time into the current push mechanism of cycle though (like the windowHeight example).

Maybe what confuses me is the relation between signal and pull based system.

PS: I wouldn't call that a QA type of side effect because this type of pull system is still synchronous. We are using an Iterable, not an AsyncIterable. With AsyncIterables you would be able to do HTTP request and response.

So how would you call it ? What is the difference between these 2 types of side effects from the main function perspective ?

@aronallen

This comment has been minimized.

Contributor

aronallen commented Apr 26, 2017

@whitecolor I really like that HTTP interface, it seems much more explicit, and there is no need for categories to filter the request objects.

@aronallen

This comment has been minimized.

Contributor

aronallen commented Apr 26, 2017

I am not a particle physicist, but I really see a need for us to inject Math.random. With @cycle/time we can control time, with reactive bindings we control space, and with @cycle/random (I think we should call it @cycle/chaos) we control chaos.
Space, Time, Chaos are dimensions which all particles (data) can flow through, if we can directly control all three, we can control anything.

@cluelessjoe

This comment has been minimized.

cluelessjoe commented Apr 26, 2017

A side note regarding this bit:

Currently you would have to do something nasty like click$.map(() => Math.random()) which is impure but most importantly, makes testing a bit worse since it's non deterministic and we're not into doing invasive mocking of globals.

One can use "pure functional pseudo random number generator", which consists of carrying a seed along with each value. Then the next random value a new seed, deduced from the first one, is given back. This new seed has to be used afterwards for the next random value and so on.

The best explanation I've seen of it so far is in Functional Programming in Scala, but this one might do as well, hopefully.
Looks like there's also some JS impl, pure random.

I'm sorry I don't have enough skills to provide a short & concise explanation, however the gist of my comment is to say that generating random number in a pure & functional way is possible and done by others. This in turn allows us, I think, to discuss only the "periodic triggering of events", either through Streams or Signals.

@whitecolor

This comment has been minimized.

Contributor

whitecolor commented Apr 26, 2017

I really like that HTTP interface, it seems much more explicit, and there is no need for categories to filter the request objects.

This API is kind of arguable, while it may seem suitable for GET requests if it is used for example for updating something we may lose track and control over this requests as they don't go all the path down/up through the sink and the source.

Though it is still functional in terms of separation of side effects from pure stuff, but it is not so cyclic.

@jvanbruegge

This comment has been minimized.

Member

jvanbruegge commented Apr 26, 2017

AFAIK, the push part would be kept in place. But the API would be nicer, because if you want to GET something, you get your value "directly" without having to make request first (API-whise). When you want to do something (POST) you would still send that to your sinks.
Like this:

function app(sources) {
    const changedID$ = sources.state$.map(s => s.id).compose(dropRepeats());
    const data$ = sources.HTTP.pull(changedID$.map(idToRequest));
    const update$ = data$.map(dataToAPIUpdate);
    return {
        HTTP: update$
    };
}
@SteveALee

This comment has been minimized.

Contributor

SteveALee commented Apr 26, 2017

Now that's an interesting extension on simply polling a changing value.

I'm not saying it's wrong but at a system level the pull is pushing a message and waiting for an async response. That's more complex what are the error flows going to be. You could certainly make a tomeoutcpart of the pull spec.

@SteveALee

This comment has been minimized.

Contributor

SteveALee commented Apr 26, 2017

An interesting featuring is you a pushing the correlation of request and response to the driver. That would probably be great in most cases but you'd want to avoid extension to a stream version of RPC:)

@whitecolor

This comment has been minimized.

Contributor

whitecolor commented Apr 27, 2017

For things like HTTP, pull api also could look like, :

xs.periodic.mapTo(HTTP.pull('google.com'))
  .flatten().map(res => res.body)

here pull would take RequestInput and send to driver always lazy request, that will be executed only on subscription.

This thing resembles examples in https://github.com/pull-stream/pull-stream

@staltz

This comment has been minimized.

Member

staltz commented Apr 27, 2017

@whitecolor

How all this Signal thing would be better/different than a stream based driver with that can pull the value from source
HTTP.pull(xs.periodic(100).mapTo('google.com'))

I see how that is an appealing API, and to some extent it isn't much different to the API of something like sources.windowHeight or sources.random, except now we have an argument.

That said, this is different because we would require an AsyncIterable (should we support both Iterable and AsyncIterables? I don't know). Notice that with a normal Iterable like for random numbers, we can sample that Signal at times defined by x$ and we will get our sampled values at the same time as x$ emitted. With AsyncIterables, when we sample, we spawn an async execution which will later yield a value. So with this, we also need to think about parallelization and cancellation, just like we have flatten/flattenSequentially/flattenConcurrently for streams.

Then, it's also different due to the semantics of pull and how that affects the devtools (or in general visual programming, which is something we want to heavily invest in Cycle.js, as we discussed a lot in CycleConf).

With a push-driven HTTP API (the normal Cycle.js), you have a dataflow graph of request going down, leaving the system, and then the external world responding back with a response. In visual programming, this mental model makes a lot of sense and is easy to teach.

With a pull-driven API, I don't know what the accurate visual metaphor is. A sampling of a signal with a signal would look like a "combine" but when the combined thing emitting in order to start pulling down asynchronously the data from the sources. Nothing "leaves" the system, we only get data in. It might look weird. Or might not, this is why we need to discuss.


@cluelessjoe

One can use "pure functional pseudo random number generator"

The original motivation for this issue was not "how do we put Math.random() in Cycle.js" but more like "there are dozens of pull-based APIs, such as Math.random() and window.outerHeight, how do we accommodate any pull-based API?"


@borisirota

can you please define the meaning of signal as you see it ?

I defined it in the beginning

a functional ES6 Iterable library

xstream and ysignal could be tightly interrelated "sibling" libraries if they have conversion operators from one to the other and vice-versa.

ysignal should be "hot" in the same sense as xstream because we need the assumption of "one execution per stream"

I know my first comment was quite dense in information, but this isn't a beginner tutorial on Signals, it's a proposal and a theoretical experiment. There is also example code if people have more questions.

Now, Math.random() is not a property which changes over time so why implement it as signal

You could consider it a property that is constantly changing over time, it's just the actual implementation of Math.random() happens to be lazy and state-driven. Consider if there would be an implementation of Math.random() based on a sensor of the actual temperature in a room, and the minor fluctuations of that value. It's a property changing over time, and achieves the purpose of Math.random().

@whitecolor

This comment has been minimized.

Contributor

whitecolor commented Apr 27, 2017

So with this, we also need to think about parallelization and cancellation, just like we have flatten/flattenSequentially/flattenConcurrently for streams.

@staltz This was my first question. Because async HTTP.pull and alike sync pull source sources.windowHeight can be actually implemented currently without introducing Signal:

Instead of:

run(main, {
  windowHeight: () => Signal.from(() => window.outerHeight),

we could just do:

run(main, {
  windowHeight: () => {
    sample(sampler$: Stream<any>) => sampler$.map(() => window.outerHeight)
  },

What is the actuall technical/conceptual reason to impement new paradigm and there reimplement there part of current Stream functionality?

@staltz

This comment has been minimized.

Member

staltz commented Apr 27, 2017

In the snippet above, the source would still be a stream, composed with stream operators. In the original example, the source would be a signal, composed with signal operators, and eventually consumed in a driver which expects signals.

What is the actuall technical/conceptual reason to impement new paradigm and there reimplement there part of current Stream functionality?

staltz: ... how to better accommodate usage of inherently pull values

Specially in a way that acknowledges the fundamental difference between pull and push. Note that you can't convert from pull to push without losing some information. (Also vice-versa). Users are getting too much confusion around synchronous cases of streams, such as xs.of() + xs.combine() or some other usage e.g. with MemoryStream (which is half pull, half push).

@whitecolor

This comment has been minimized.

Contributor

whitecolor commented Sep 24, 2017

@staltz

This comment has been minimized.

Member

staltz commented Sep 25, 2017

I have, many months ago, but I should take a second look at it. I liked the ideas of sample and snapshot:

sample :: Event a -> Behavior b -> Event b
snapshot :: (a -> b -> c) -> Event a -> Behavior b -> Event c

@jvanbruegge jvanbruegge added this to To do in the next months in jvanbruegge's pipeline Oct 3, 2017

@brucou

This comment has been minimized.

brucou commented Oct 13, 2017

Wanted to share with you guys two articles (connected to our subject, bear with me) :

The summary of those two articles is the following :

  • the true dual of Observer is Iterator (and vice versa, as logical)
    • that is about the most clear article I have read on the subject. @staltz this complements the talk you mentioned by Eric Mejer, where he shows duality by expliciting type signatures
  • a good API for iterator is moveNext and current (I used get for current and pull for moveNext in my partial-synchronous-streams) library. Naming apart, the idea is to isolate the 'nexting' side-effect from the observation of the current value.

On a sidenote, another lesson I retain from this is that API design is a tricky business if even a team of PhDs can't get it right the first time. Indeed, it is hard to imagine all the future consequences of a design decision, given it is hard to imagine all the future use cases...

@brucou

This comment has been minimized.

brucou commented Oct 13, 2017

I will get back to the subject in a few days. Short summary is as of now, I tend to favour three related concepts : events, behaviours, and iterators ; over the two concepts of streams and signals, though both formulations would probably be pretty much equivalent in fine, the first one just seems more clear to me to reason about. But enough of that, the point I want to experiment on in the next few days is the following :

  • iterators can end up through miscellaneous combination either in yet another iterator, or in a stream
    • iterator to iterator is through the ystream operators that @jvanbruegge is working on
    • iterator to stream can be achieved TWO WAYS (cf. my previous post about moveNext and current):
      • withCurrentFrom as in stream.withCurrentFrom(signal) (cf. Rxjs, withLatestFrom, which encompasses both the sample and snapshot mentioned in most-behave
        • no side-effect, i.e. can be called multiple times, will return the same iterator values. That would allow for the equivalent of multicast we have with the push model, by which a pushed value makes its way to several stream pipelines. Here a pulled value could be integrated into several stream pipelines.
    • withNextFrom, as in stream.withNextFrom(signal)
      • same but with 'next' side-effect, will possibly return a different value every time
  • stream to iterator would be
    • necessarily behaviour to iterator, because there is no way a stream event would be simultaneous to a next call. And if you store that event for next to produce later, then you have a behaviour in disguise.
    • already covered I believe as one of the iterator factory in ystream (can't find the name right now)
  • stream to stream, well, that was we have already

If there are only two concepts, then those sould be the four possibilities of COMBINING them.

For CONSTRUCTING them from one another, the possibilities are :

  • stream to signal, as in Signal.fromStream(stream)

    • there is a large number of ways to build a signal from a stream. The most useful is probably to pull the latest value emitted by the stream (i.e. similar to remember or behaviours). But you could also pull any number of past values from the stream in any order (say, the first and the latest).
    • The good news here is that this pull any number of past values ... can be subsumed into a stream -> stream function (stream -> stream.scan(getFirstAndLatest)), dealt with the aforedescribed most useful case of emitting the latest value of stream in Signal.fromStream(stream). So here just one API seems to suffice.
  • signal to stream, as in Stream.fromSignal(signal)

    • here you are forced to pull the signal, i.e. to specify a pull function. Obvious choice is polling at a given frequency : Stream.fromSignal(signal, pollingFrequency). Or pulling when a given event occurs, i.e. sampling based on a stream i.e. we go back to the previous combining case of stream.withNextFrom(signal). Those two cases are actually the same, so here, there is no need for a new API except the convenience of having it.
    • The general case however seems large : you can aggregate any number of past values pulled at any moment in time from the iterator in any order AND emit that at any moment in time.... That is a lot of degrees of freedom and I don't see here how to reduce that back to a previous use case. You need two streams here, one to tell you when to pull from the iterator (sampling stream), one to tell you when to emit (scheduling stream), and then an aggregating function to combine the pulled values into an output value.
@staltz

This comment has been minimized.

Member

staltz commented Oct 17, 2017

Good insights, @brucou, I'll think about these.

@staltz

This comment has been minimized.

Member

staltz commented Dec 25, 2017

I found some new insights and briefly discussed this with Jan. I'll report here as well.

Insight 1: Push is lower level than pull

This can be demonstrated easily: you could replace Math.random() with two push messages: (1) a void-content request message (in other words ()), and (2) a response message of type number. It might sound contrived, but it's certainly possible to support Math.random use cases in Cycle.js by having a randomness driver, where sinks are request messages each with a specific ID, then source is a stream of responses that have that ID attached to it so the app can know which response matches which request.

Note that this insight is only theoretical so far, I'm not saying that we're going to force all drivers to look like that. It's just important to realize that pull is not fundamentally complementary to push. Pull is just a common and convenient API for Q&A-style push messages.

As an example, the chosen API for WebWorkers is quite low level, and it's entirely push-based. postMessage is a push, and onmessage is push as well.

Conclusion: push is better suited as the low level building block, it's probably better to build pull APIs on top of push, than to bake opinionated pull inside core parts of Cycle.js.

Insight 2: Wrapper utilities abstract low-level details

A utility like Onionify doesn't perform I/O to peripherals, but instead provides nicer APIs to manage state. We could consider pull in the same light: given that pull is an abstraction over push, we can provide pull collection APIs through a wrapper. Let's name this, for now, pullify.

As another example (and this could be built today, without changing the framework), people have been asking for a better API for Q&A-style I/O like the HTTP driver has. Instead of using the low-level push-based API (with categories), we could provide a pull-based API through a component wrapper, and the wrapper would take care of low-level push messages with the HTTP driver. Let's call this, for now, pullifyHTTP.

This starts to hint towards pull being an integral part of the main-side, but not of cycle/run or drivers.

Insight 3: Drivers could be rewritten to be lower level

Currently the DOM driver has some special methods attached to the source object, and they perform some pull-like side effects, e.g. domSource.events() actually registers event listeners based on the arguments.

The DOM driver could be rewritten to be simply a function stream -> stream, where the sink would be a stream of commands instead of stream of VDOM. Commands could be of multiple types, where the most common should be {type: 'VDOM', value: vdomTree} but should also support new command types like {type: 'ADD_EVENT_LISTENER', ...} which would affect what the source stream would output.

In other words, most complex drivers would do Multiplexing on sinks and also on sources. This would be much lower-level than what we have today, but before I explain its benefits, let's talk about:

Insight 4: Driver wrappers would provide nice APIs

If the DOM driver would be rewritten like explained above, we could create a wrapper for the DOM driver in the same style as onionify or pullify, and this wrapper would take care of interpreting all those low level commands and responses, and providing the convenient API we have today of domSource.events(), domSource.elements() etc, and probably more or better APIs. The point here is that the DOM driver could be split into two parts: one is a function Stream<LowLevelReq> => Stream<LowLevelRes> and the other is a wrapper function which replaces the DOM channel with DOMSource and Stream<VNode> as sink.

Insight 5: Plugins are pairs of driver & wrapper

Since the DOM driver would be split into two parts, many similar drivers also would be split, where the low-level part does the actual side effects, and the wrapper part simply provides nice APIs. So as a redefinition:

  • Driver is a function Stream<LowLevelReq> => Stream<LowLevelRes>
    • Takes a sink stream of low-level commands and returns a source stream of low-level responses
    • Operates on some channel, Foo
  • Wrapper is a function from high-level component to low-level component
    • It's job is to interpret a component written in high-level style with nice APIs and return a component that operates on the low-level with the driver
    • Takes as input a main function with nice-API source Foo and nice-API sink Foo, and returns a wrapped main function which uses low-level streams on channel Foo
  • Plugin is a pair [driver, wrapper]

Note now, that a plugin may have a dummy driver or a dummy wrapper.

Then we can redefine onion state as a plugin [identityFn, onionify], with no driver (or basically, an identity function instead of a driver) and a valid wrapper.

Most of the basic drivers out in the wild would be simply converted to [driver, identityFn] but complex drivers like the DOM driver would be [newDOMDriver, wrapperForDOMAPI] (bad names but just to illustrate the idea).

run(main, {
  onion: makeOnionPlugin(),
  DOM: makeDOMPlugin('#app'), // return [driver, wrapper]
  HTTP: makeHTTPPlugin(),
);

Plugins would also help avoid the ordering problem involved when chaining wrappers (e.g. modalify with onionify). And there are more creative ways we can make use of plugins.

Insight 6: Core packages become fully stream agnostic

With plugins as [driver, wrapper] pairs, we could also put stream adaptation (for xstream / RxJS / most) on the wrapper side. E.g. imagine makeDOMPluginForRxJS('#app') if you want to use the DOM plugin with RxJS. This would also help us to get rid of setAdapt since adapt was mostly used in driver-land.

This would remove the need for most-run and rxjs-run which would be a good riddance, while still supporting Most and RxJS through other means.

cycle/run would be stream agnostic, would not even require xstream. We could just use callbacks, much like how WebWorkers are push-based but use raw callbacks. Callbacks would be enough to send low-level request objects and low-level response back and forth through channels.

Insight 7: DevTools is 100% on the main side

With visual programming in mind, we noticed that today the DevTools is already mostly useless for Cycle.js users who use Most or RxJS. The main purpose of the DevTools is to understand how the application works, not the driver side, so the DevTools should be aware of the stream library being used. It's hard to reliably visualize all stream libraries in the same style because they have different semantics specially regarding multicast and laziness.

With the addition of pull ("signals"), the DevTools would have to be rethinked as well. Notice that since pull is merely a main side thing for convenient APIs, we can't add pull on main without thinking how would we visualize pull collections.

So I propose we acknowledge the truth that DevTools are stream-library specific, so they should at least be named accordingly, e.g. "Cycle.js xstream DevTool" or "xstream visualizer".

Insight 8: Enforcing low-level drivers would enable fine-grained logging

If we do this rewrite, then we could constrain all drivers to be strictly of type Stream<LowLevelReq> => Stream<LowLevelRes>. This hard constraint would then become a guarantee, which can be useful for building things on top of such guarantee. Type safety would be one benefit, but also we would get fine-grained low-level logging for free.

Since the new API would be run(mainFn, pluginsObj), each plugin row already has a wrapper for nice API, but we can also add a catch-all-channels wrapper for logging: run(mainFn, logify(pluginsObj)) where logify :: obj -> obj and intercepts low-level messages on each channel, to add them to a log.

Remembering the big picture

The proposal is:

  • Plugins as [driver, wrapper] pairs
  • main-side pull collection APIs provided by wrappers
  • run-side is callback-based, stream agnostic
  • Potentially multiple DevTools (probably one official and other community)
  • Wrappers provide xstream (push) + ysignal (pull) ... OR ... some hybrid of both

Let's remember why we're discussing all this in the first place.

  • We need a dataflow-friendly way of handling pull-based APIs:
    • Math.random
    • UUID
    • DOM element sampling
  • A better API for element.focus()
  • A better API for preventDefault
  • Visual programming

But also these new insights allow us to address previous gripes that left us scratching our heads:

  • How to accommodate a nice API for Q&A-style I/O like HTTP requests and responses?
  • Driver sources are usually complex objects (classes!) with methods
  • Drivers had mixed responsibilities: perform the I/O effects, and provide a nice API for querying results of effects
  • How to seamlessly accommodate WebWorkers?
  • WASM?

Open questions ahead

  • How do we visualize pull versus push in DevTools?
  • Which of the parts would run in WASM and which would run in the main JS thread, which would run in WebWorkers?
  • Where to draw the line between WebWorker and main thread?
    • It would be useful that the WebWorker sends the low-level push-based requests and responses from/to the main app, while drivers and run are on the main thread
    • So we would have
      • main and wrappers: in WebWorker
      • run and drivers: in main thread
    • BUT with the proposed new run(main, plugins) API, wrappers and drivers are put together, so it's inconvenient to draw the line between them
    • Maybe we need a new API to replace run(main, plugins) which makes it trivial to choose to apply a WebWorker
  • How would Cycle Time fit into this picture (given that it has high-level API for sources)?
  • Would this new proposal still fit well with a requestAnimationFrame-driven DOM driver (which is currently in the experimental-push-pull-cyclejs)?
@geovanisouza92

This comment has been minimized.

geovanisouza92 commented Dec 25, 2017

My 2 cents: "wrapper" in this proposal seems to be more like an Adapter, if I understood its role as a API translator (from high-to-low and low-to-high level)

@geovanisouza92

This comment has been minimized.

geovanisouza92 commented Dec 25, 2017

If this rewrite will be considered, I think that's a good opportunity to include changes to achieve other pain-points in the current restrictions, like #727, #729 and #432

@staltz

This comment has been minimized.

Member

staltz commented Dec 26, 2017

@geovanisouza92 About the name, you're right that Adapter would be more accurate than Wrapper.

Also about those DOM issues, we don't need to fix all at once. For instance, here is a possible release roadmap:

  • Cycle Run and Plugins are released as a new version of Cycle.js
  • The current APIs for all drivers and stream libraries is almost entirely the same as before
  • A new stream library that supports pull is built
  • One by one we update drivers to support the pull APIs and fix a few issues
@staltz

This comment has been minimized.

Member

staltz commented Dec 26, 2017

plugins

Blue is I/O. Green is logic. Red is the cycle.

@whitecolor

This comment has been minimized.

Contributor

whitecolor commented Dec 26, 2017

@staltz Drawn perspectives seem to be more open-minded and universal, I like this.

"wrapper" in this proposal seems to be more like an Adapter,

Yes, it seems like returning to diversity version principles =)

@staltz

This comment has been minimized.

Member

staltz commented Dec 26, 2017

Indeed, adapters like before. But this time it would be generic adapters, not stream adapters only.

@brucou

This comment has been minimized.

brucou commented Dec 26, 2017

I feel like this is a complex discussion, handling various issues at the same time. Not so easy to properly discuss this in a github issue format.

But first of all, a note about the review and comment process. It could be beneficial to divide this into separate subjects, and discuss them separately. For instance :

  • architecture
    • driver
    • plugin
  • Interface
    • plugin API
    • core API
  • dev tool
  • cross-cutting concerns
    • wasm
    • webworkers
    • logging

Now, about the specifics (I will follow your insight breakdown here) :

  • Insight 1

    • ''postMessage is a push'' : what makes it so? I was a bit lost in the pull/push discussion in that section. The on event, do something i.e. onMessage and the do something now, i.e. postMessage seems to be of different nature to me.
    • in any case yes, you can have the do something now passed through a driver and get that back. The caveat is that you should strive to keep equal semantics to a function call. In particular that means synchronicity (otherwise it is not a do something NOW anymore, right?). To replicate a function call, you also need atomicity of the execution. That means that ideally after function invocation, you should not do anything else than computing the result, and returning it. It is not obvious to me immediately that this can be guaranteed with a stream/channel API/architecture, specially in the absence of schedulers. I would be super careful here because concurrency issues such as this one, are ridiculously hard to figure out and remediate after the fact
    • Independently of possible ordering or concurrency issues, I am strongly supportive of having ALL effects in dedicated effect handlers. It has the positive benefits you mentioned of having a consistent stream signature for components functions. That is a TREMENDOUS benefit, and you also explained that somewhat, so I won't do it here. I see two issues :
      • API duplication
        • in the current model, you take a sources.dependency object, and you execute a dependency.command function. The injected dependency already comes with its API set. For what I understand, the proposed model is that you emit requests {command, params} on a dedicated dependency sink. To keep equal expressive power than the current model, you then have to, in the worse case, fully duplicate the dependency API. I know that you can always do a 80/20 approach, and only duplicate the most useful API, but it will always come a time when you need the remaining 20. So for instance, you would have to duplicate the DOM API which has a huge surface.
    • addressing (i.e. isolation)
      • reconciling request and responses is an issue you mentioned already. You said that you can do it with wrapper. I confess my ignorance of how wrapper actually do this. But I would wonder how this works in the context of componentization, i.e. how to route a response deep down a component hierarchy. I understand that this should all work through the isolation mechanism, but I don;t know how isolation works deep down. But if it is possible, then great, because this is no trivial issue.
  • Insight 8

    • logging/tracing is linked to visualization in the sense that on can draw a dataflow graph from a log of events and requests. If somehow that log would contain hierarchy information (which component at which level is receiving the input, or emitting the request), it would also be possible to draw the component graph from it. This would possibly be the stream-independent way to draw the dataflow.
@geovanisouza92

This comment has been minimized.

geovanisouza92 commented Dec 26, 2017

@staltz By mentioning that issues, I mean that, it's a good opportunity to design those API's with that features in mind: 1) Don't tie plugins with specific ways to do things; 2) make the creation lazy.

@aronallen

This comment has been minimized.

Contributor

aronallen commented Dec 26, 2017

If DOM is a Stream<LowLevelReq> how would tree composition work?

@staltz

This comment has been minimized.

Member

staltz commented Dec 26, 2017

@aronallen in that case the Req would be an object with type "VDOM" and value is the VDOM tree

@bloodyKnuckles

This comment has been minimized.

Contributor

bloodyKnuckles commented Dec 28, 2017

If the cycle/dom interface is rewritten then I suppose the cycle/html interface would also need to be rewritten to maintain "isomorphicity".

@VasilioRuzanni

This comment has been minimized.

VasilioRuzanni commented Dec 30, 2017

@staltz @jvanbruegge On the architecture part, would it make sense to consider "the other side" (i.e., the run/drivers) being a special kind of function on its own instead of opinionated plugin system configurable at run(...) call? That would provide full flexibility on how to organize the effectful part of the app (plugin system might just reside there in some form), similar to how motorcycle.ts does that: https://github.com/motorcycle/motorcycle.ts/blob/master/examples/sokoban/src/bootstrap.ts#L16-L28

Or is it fundamentally different from what Cycle.js is moving towards (or maybe there are some subtle issues)? What do you guys think, anyway?

P.S. I'm seeing Cycle as a dataflow pattern in the first place, but like its emphasis on being functional (in addition to reactive).

@staltz

This comment has been minimized.

Member

staltz commented Dec 30, 2017

@VasilioRuzanni could you comment on the newly created #760 issue? :)

@VasilioRuzanni

This comment has been minimized.

VasilioRuzanni commented Dec 30, 2017

@staltz Sure thing man, posted a little bit more elaborate version of the comment there :)

@goodmind

This comment has been minimized.

goodmind commented Apr 21, 2018

any info about signals?

@staltz

This comment has been minimized.

Member

staltz commented Apr 23, 2018

Hey @goodmind, hard to say if we're going to do Signals (as in the ysignal idea) or if we're going to allow Callbags as one of the stream libraries (which would then allow push&pull using Callbags) or if we're going to do something with xstream. The plan so far is to #760 with Callbags and build support for push&pull from inside out. That's been the theoretical plan, I just need to find time to focus on building #760. Most likely a rewrite of xstream would also occur. But that comes later.

@VasilioRuzanni

This comment has been minimized.

VasilioRuzanni commented Apr 27, 2018

@staltz Andre, haven't plans for making Cycle Callbags-based been established yet? Curious if there are any advancements on this and experiment-plugins implementation. Maybe some help is needed there (even conceptual/architecting)?

@staltz

This comment has been minimized.

Member

staltz commented Oct 29, 2018

At CycleConf18 we discussed further what to do about this issue, and @jvanbruegge and I figured out that simply utilizing source class methods like sources.http.get(url) should be enough to support pull, in a backwards compatible way, and we could determine a protocol for source classes to follow so that they can be easily visualizable.

Here's a decision tree for how we got to that conclusion, presented at the conf:

future

@staltz staltz closed this Oct 29, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment