Cycle.js coding style puzzle: sink-HTTP-source, or just wrapping ajax promise to a metastream and then flatMap?

[beeplin]

In the traditional RxJS programming, when doing Http request, we just use fromCallback or fromPromise to wrap a new stream, then map the request stream with it, building a meta-stream, and then flatMap it.

But cycle.js recommends to put the Http thing out of main() and into the HTTP driver, just like what we do with DOM.

I understand the drivers mean 'outside world' or 'human user'and main() means 'computer'. But, yes, DOM events are from human users, while HTTP should still be on the 'computer' side.

So what's the point to put HTTP operations into drivers? Is it just because Http operations introduce side effects? Does that mean main() is not just 'computer' bu also pure, and all side effects must go to drivers, no matter they are from humans or from HTTP data fetching?

[staltz]

But, yes, DOM events are from human users, while HTTP should still be on the 'computer' side.

HTTP is not just in the computer. It crosses the network and talks to other computers. Drivers are for external communication, and network is really just that.

Is it just because Http operations introduce side effects?

Yes.

Does that mean main() is not just 'computer' bu also pure, and all side effects must go to drivers, no matter they are from humans or from HTTP data fetching?

Yes.

That said, I do understand your point of view that it's not ergonomic to use the Stream I/O API for req/res. I totally get that. A monadic API is better (don't worry about the jargon, the point here is that there is theoretical foundation behind these issues). We are discussing in these issues about alternative HTTP driver APIs:
#278
#272

[beeplin]

@staltz thanks for the clear reply. Actually I have digged for hours into the previous issues you mentioned, and here are my thoughts:

I totally agree with you that, if we hold the side effects standards, then all I/O operations have to go to drivers (even your new plans about the monadic API are still putting HTTP in drivers, not main(), right? ), and the main() function (and component functions) must be pure, which makes testing very easy. I think the testability is the primarily (if not the only) benefit by doing so.

I understand you hold this standard quite firmly, so in #204 you are not interested in putting state data into drivers. And I agree with you on this point.

But I have some further ideas that, not only state data, but also HTTP, websocket, localStorage... all these things may be moved out of drivers, into main(), if we can loose the side effects standard. In other words, drivers should only deal with human I/O through DOM, and all other I/O should be inside main().

I can name three main benefits of doing that:

1. clearify cycle.js' philosophy;
2. fit the MVI pattern better;
3. make data flow graph more clear.

Let me elaborate it.

Firstly, I do feel some subtle incompatibility inside the philosophy of cycle.js. Sometimes the cycle means the cycle between logic and side effects, sometimes it means the cycle between computer and human users. So the line between app and external world is always vague. You mentioned:

You need to define what is the "external world" and what is the "application". It's pretty clear that graphics and mouse inputs are from the external world, but so is a server (in relation to a client app). Other not-so-obvious distinctions are e.g. storage. It's not actually external to the computer, but it is external to the application, so we can consider it part of the external world.

On the other hand, we don't in Cycle.js consider e.g. the memory heap as part of the external world. That's why state in memory is not a side effect in Cycle apps.

OK, so database on the remote server is external, and localStorage on the client itself is still external. So what about client-side in-memory database (like client-side minimongo of meteor)? In fact I am doing a project that will make use of this sort of in-memory database on client-side. This database uses an async API (callback or promise) to fetch data from both localstorage and remote server(by ajax or socket.io), combining and syncing the two sources.

Of course I don't need to write a driver for this in-memory database, because as you said, this is within my own client-side app. But, in fact the API of the in-memory database is exactly the same with that of the localStorage database and that of remote database I am using. I designed all the three databases. All of them are part of my app and my logic. Nothing external. And I don't see why I have to put the other two inside drivers and leave the in-memory one in main(), especially when I don't care the "logic - effect" division and can mock a database to do testing.

Secondly, I feel the "logic-effect" standard is also somehow contradictory with your MVI pattern. Let's see, if the DOM is the only source passed from driver to main(), and the vtree$ is the only sink passed from main() to driver, then the main() function does fit the MVI pattern: you extract user intent (dom events) from source, process it into model, and then construct a virtual DOM view object to return.

But with HTTP and other I/O stuffs mixed in the sources, the MVI pattern is kind of broken. "Intent" by its name means user operation to the DOM, not the HTTP or websocket response, but at the top part (supposingly the intent part) of main() we have to do HTTP response processing. And "view" by its name means the manipulation of the vdom, not the HTTP request, but at the bottom part of main() we have to return not only vdom, but also HTTP request etc. So when we do HTTP req and res through driver loops, in main() we see two parallel lines, one for DOM's 'MVI, one for HTTP. Of couse it's not a big deal, but traditionally, fetching data from server and posting data back to server according to users click events are always treated as the "model" part.

Lastly, just as you mentioned here:

HTTP interaction is very different to DOM interaction because in the former you have a 1-to-1 correspondence between sink event and source event, which you want to keep. On the DOM however, you don't have any direct correspondence between DOM sink message (a VTree) and DOM source event (click, etc). For each rendered vtree, you may have zero or 1 or infinite click events related to that vtree. It's not a 1-to-1 correspondence.

What I learn from this statement is that, unlike DOM source events which are quite diverse and unpredictable and hence be better to use the driver loop to handle, the HTTP request-response pairs are just traditional 1-to-1 callbacks or promises that are very easy to be transformed into flatMapped streams. So if we give up the HTTP driver loop and just embed the HTTP operations into main(), we won't break the data flow graph -- actually we are making the data flow graph even clearer, like:

userClick$ --(map)--> HTTPRequest --(flatMap)--> HTTPResponse --(map)--> VDOM$

cycle.js's current way, by dividing the http req and res into two parts, actually makes this data flow a little bit hard to handle.

The websocket push event stream, if used, can be similarly handled in main() without breaking the data flow graph, say, can be merged into HTTPResponse stream and then mapped to VDOM$.

That being said, I agree that this is basically a "philosophical" or "architect" issue rather than technical. Both ways could deliver what developers require when building large apps. And the current way of course has its merits: testability, driver as the only place to do supscription...etc. The only thing I want to say is, there is still some vagueness and incompatibility inside cycle.js's design philosophy. We must make decisions and make them clear.

Anyway, thank you for the wonderful design!

[beeplin]

sorry I got some network connection issue here and wrongly posted the above comment multi times just now. all cleared.

[beeplin]

And after reading #272, I do more prefer the design that there is no universal HTTP driver: each component makes its own HTTPRequest$ and handles its own HTTPResponse$ by flatMap; we can still put multiple response$ together by merge/combineLastest etc. if we need to; and so there's no isolation issue then. I don't see much benefit from having the potential to do pre- and post-processing for the HTTP I/O -- components can do that by their own. And I do hate filtering HTTP responses again and again.

We do need a universal DOM driver because there is only one DOM root there, and all components' vdom$ have to be put together finnally to render the real one DOM; but we possibly don't need a universal HTTP driver because unlike DOM, HTTP is not a bus, not by its nature public.

[staltz]

Ok, I understand your point of view.

Sometimes the cycle means the cycle between logic and side effects, sometimes it means the cycle between computer and human users. So the line between app and external world is always vague.

The line isn't that vague because side effects are essentially "changing something in the external world". There is always a correlation between "doing real-world changes" and the concept of "side effect".

I think the more sensible distinction is

• Question & Answer type of side effect (QA)
• Real-time interaction type of side effect (RT)

HTTP is QA, localStorage is QA, etc. But DOM Driver is RT and also Web Socket is RT. Notice that the side effect doesn't need to be related to the user in order to be RT, because the Web Socket is not user interaction.

Stream I/O (what you see in Cycle.js) is appropriate for RT. Monadic I/O or traditional impure I/O (what you want) is appropriate for QA.

userClick$ --(map)--> HTTPRequest --(flatMap)--> HTTPResponse --(map)--> VDOM$

So, if you believe this is a good idea and you don't care about losing pure testability for the HTTP effects, then go ahead and make the HTTP requests inside main(). You are acknowledging that main() is now not pure anymore, but doing so doesn't require new tools to be built in Cycle.js.

But with HTTP and other I/O stuffs mixed in the sources, the MVI pattern is kind of broken. "Intent" by its name means user operation to the DOM, not the HTTP or websocket response
The only thing I want to say is, there is still some vagueness and incompatibility inside cycle.js's design philosophy.

There is nothing incompatible really. MVI is meant for DOM concerns only, it's not meant for every effect concern. Check this component for instance where there is an http function which handles HTTP concerns, while V and I only handle DOM concerns. https://github.com/futurice/power-ui/blob/master/src/components/pages/PeoplePage/index.js#L67-L83

MVI is an onion-architecture, but the more effects you add, you'll have to add more pre and post processing functions before Model. E.g. Intent is pre and View is post, for the DOM. For other concerns you build other pre and post functions as required, but the Model can take inputs from multiple concerns.

but traditionally, fetching data from server and posting data back to server according to users click events are always treated as the "model" part.

That doesn't mean that the traditional way is good. Model should be just about pure logic, because that's a good idea for the "onion-like" features of the architecture. Here is a small sample of what I'm talking about: https://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html

[beeplin]

@staltz thank you for your patience. Yes now I realize my previous proposal is actually reducing Cycle.js into merely a wrapper of RxJS with a v-dom plugin. That's clearly not what you want. Cycle.js is more about a beautiful programming paradigm.

[3n-mb]

@staltz when programming logic requires at a certain point of data flow diagram a transformation that needs to do a request (QA type) to an outside (HTTP), then a requirement to pass request into sink and wait for reply from a source will complicate (nuke?) a nice data flow diagram.
Programing involves reading. And I want to start reading it from looking at a cycle devtool diagram first. If diagram can be clean, its a win.

If transformation function does HTTP request, is it automatically impure? It is a pure function, if QA was a read. It just happen to transcend boundaries, that's all. But, it is a pure function.

If use of HTTP inside of a function makes testing a bit more complex, may be it is better to transfer complexity to testing, away from programs' structural wiring diagram. Test phase may have unittest stage, integration at different levels stages, etc. But, if original program's structure is simpler, may be, there will be less tests to attend?

May be, this visualization, this help in seeing what program does is a baby in the water.
In my experience, typescript was like a bread for the starving, cause computer started to help me, developer, in my work. Cycle dev tool diagrams is a sliced bread!

@beeplin a circuitry of observables, with sinks and sources of RT events, is already that beautiful programing paradigm, visual, simple to grasp, maintainable in a long run. And yes, v-dom is a mere plugin on both sinks and sources sides for some RT events. And may be, further purity requirements are not as essential?

About purity. An observable that accumulates incoming values has an internal state! Some of fundamental Rx elements have state! So, purity questions can be not particularly pure.

Secondary note. In electrical diagrams, components with electric or magnetic capacity are called reactive (direct translation from ru, that is). What if you have to deal with a diagram that connects to a battery? Think of that battery as big chunk of state, data, etc. Somehow, electrical diagrams are used with complex elements in them -- no problem. The problem arises only when a diagram is unnecessarily complex.
And testing of a circuitry with complex elements is by definition complex.

[3n-mb]

@staltz have you seen these people who say that Promises will be dropped in favour of observables? They are mixing QA and RT styles!

Promise is a mere function call that happens to complete on a next tick. In langs with blocking, call blocks. When someone in a blocking lang starts to use promise/future it is done for performance reasons.

Observable, on another hand, is a source of events.

Function call <-> QA
Observable <-> RT

[jvanbruegge]

@3n-mb accessing anything outside your program (even accessing non-const variables in your program, or getting the current time) makes your function impure
The problem is that pure functions have to be referencial transparent. This means you can replace the function with it's result and not changing anything. This is not possible if you make a HTTP request, because the result of this is not fixed

[3n-mb]

@jvanbruegge that's right, the deeper we go along purity argument, the more difficult it is to see the worth preserving it.

When I look at a diagram, generated by cycle's devtool, I can see that some incoming event from dom, like click, starts an activation in a wiring of observables. RT (real-time) event drives that activation, event starts it. This is a uni-directional data flow (good).
When my vdom is sunk into other side of dom (sink), it is also a uni-directional flow.

HTTP, or QA, or Request-Reply is a bi-directional single time interaction. Bi-directional has no place neither among uni-directional sinks, at the bottom of a data-flow diagram, nor among uni-directional sources, at the top of a data-flow diagram.
Can someone explain why Request-Reply HTTP must be in spots perfectly fitted for sources and sinks, giving a compelling enough argument to nuke data-flow diagram with such placement?

What if we say the following. Folks, there may be times when in a certain section of your data-flow a request is needed, somewhere just a read, somewhere a write. Make those places smaller, and mark them as impure.

In tiny examples of cycle.js, state is contained in little collectors. But it is a trivial example of state in a diagram. What if domain requires keeping a bigger state?
Guess what, I want cycle.js dev tool to help me deal with this state! It means that circuitry with model/state may consume lots of input for change, and output consistent total, or consistent focused part (lense?), besides events that say if state has been serialized (assumes HTTP talk), if state is consistent. Id est, all outgoing events are meaningful to the rest of the circuitry. Having a low level detail of HTTP on top and bottom, that serve internal concern of a state circuitry, only brings noise to a diagram.
Most importantly, since all of state circuitry is made visual by cycle tool, finally computer helps me in tuning the last bit which cannot be made pure. Conversely, if we shovel state into non-circuitry parts, we, literally, loose visibility of it.

This hybrid route hasn't been explored, yet.
The purity requirement for every circuit has been cultivated and the place is full of questions about what to put in driver, why split HTTP requests into in and out one-time streams, yada, yada, yada ...
This is a signal from reality, from nature. Should we ignore it? Everything else in cycles is digested with ease, but this.

In some talk, Alan Kay was asked about functional programing, and the answer was simple: "Show me function in physical world, cause I cann't. Yet, I can show you lots of objects." This is a good observation. And it may not be a stretch to suggest that an evolution must've attuned our brains to recognize objects, inferring their state, etc.

Let's recall What are we doing here? We have a Turing (state?) machine on one hand, higher primates brains on another, and all of programming languages, tricks, and beautiful paradigms in between. And we want to arrange pieces so that those brains a more efficient at creating programs. Pushing functional purity down their throats is not in my job description. And a hybrid approach is helpful here. Let folks see and appreciate the difference between pure and impure circuitry literally displayed by cycle.js dev tool.

How can I highlight more, that this auto-magic visual show of what program is, done by cycle's dev tool, is a discovery in itself.

[jvanbruegge]

Purity is a good thing, it's not something forced onto us, we chose it. It makes it easoer to reason about your code. You can be sure the function does not mess with the rest of the program. You can easily reuse it, because it does not interfere with anything in the reat of your app.

We also dont have to limit us in any way, just because this is how the real world works like this. We dont have that limitation inside a computer.

This is why the cycle architecture is great. As JS is not a functional language, so we have at some point to use imperative, side-effectful code. With cycle we can collect all those side effects in a single place (the driver). This way the rest of your code stays pure.

[3n-mb]

@jvanbruegge unfortunately, when I introduce cycle.js into my team, they start to feel forced into the purity, cause when they open material on the net, this thing splashes into the face. And now they have no choice, but to follow my lead into cycles.

It would be easy, if we explicitly talk about having some impure circuits, yet, easily grokable, thanks to visualization done by cycle dev tool, the whole pure/impure story stops being a boogeyman mystery.

There is no good reason to hide state into driver. There is a good reason to keep it in a separate circuit, and, here is a diagram, as an example, thank you very much, kinda thing.

It will also settle all of questions about deforming point http request into driver cyclic dance. <joke> You know, they haven't found any evidence for Super Symmetry at LHC. Bye, bye Super Symmetry. Hence, bye, bye Super Strings. We are stuck with point particles, not loops! </joke>

[jvanbruegge]

Cycle is a functional, reactive framework. If you never used FP or FRP, you should make yourself familiar with those concepts.
Angular forces you to learn its templating syntax, react has it's own pitfalls. Cycle forces you to learn about FP/FRP. The advantage of this is that those are concepts useful beyond cycle, so you will learn for laster. If you dont want to invest this time, you might be better with another framework

[3n-mb]

Here is this language:
Some transformation/computation in a circuitry may depend on HTTP request, or async call, etc., but it is not driven by it. It is circuit's internal that start (drive) and consume results of the request. Thus, HTTP should be at a point.
Drivers should be producing events that drive circuitry.
What do you think? @beeplin @jvanbruegge @staltz

[3n-mb]

@jvanbruegge this is what cycle.js today.
With the discovery of a visual tool, it can become a more powerful tool, even if less pure. And it seems that we shouldn't have to re-code much to start using it in a liberating way.

Notice how I talk about visual dev tool all the time. Its a mix that is powerful.

If we do not try to hand inescapably needed state as a visualizible circuit, we won't have a chance to discover patterns, now visual in a diagram, that will tame state beast.

Let's exercise logical purity here. If we push anything non-trivial into driver, it still becomes a circuit, it is half a circle, but on the outside (!), thus invisible to our wonderful cycle dev tool.
Advice to hide state into driver blinds. Notice that before having visual tool (eyes), this advice wasn't subtracting anything much. Not now.

[jvanbruegge]

Noone is hiding state in a driver btw, in the simple case you fold it somewhere inside your app. For more complex cases like onionify you wrap your app, leaving it pure.

[3n-mb]

@jvanbruegge we always have to learn something. May be, with all do-pure requirements there even will be parity in a cognitive effort for both cycle and angular (as you brought it up).
But, with a cycle visual dev tool we may pave a road for gentle and a bit subliminal learning, utilizing a driving force of a visual cortex of our brain.

[3n-mb]

@jvanbruegge onionify -- cool! Does it have event that says that state has been safely saved on a server? No. Why should it. It is a general tool. One may need to have ones own state circuit. But, prevailing wisdom, you know, tells that circuits should stay pure, instead of saying that if these common patterns, like onionify, are not enough, go on, do a little impure circuit yourself, and here is a visual tool for you to debug this state handling.
Permissive things are liberating.

[3n-mb]

@jvanbruegge don't get me wrong, I get @staltz 's analogy with horses and cars made in a talk. I just happen to question edges that look a little sharper than needed for a smooth introduction of cycles to a 14-year old. There are 14-year olds that fly planes, you know. Activity that doesn't require pure math thinking. Just practical.