Adopt a Flux-like approach to let information flow through an application

[almarklein]

This is a proposal for some pretty profound changes to how building apps with Flexx works. Though Flexx is still marked alpha, there are people using Flexx already, and they'd need to make some serious changes if we go this route. We thus have to think carefully whether the proposed changes are worth it.

Background

We've found Flexx to be working pretty well, even for larger applications, though the complexity of an app increases too, and we've found that newcomers to our app have trouble understanding the data-flows etc. I feel that #359 is a symptom of this, and that there is a deeper cause.

Flexx uses events. A lot. Everything is tied together using events. This is the case for both user input (mouse and key events) and property changes. The system works very nice (if I may say so myself), but a problem is that information flows all over the place, which makes it difficult to reason about the state of the application.

Flux and friends

At Facebook they had similar problems with their classic MCV approach; it scales badly because its hard to predict the state of an application based on stuff that happens in the app. Their core insight was that if information flows in one direction, an app becomes much more predictable. Here is a video that explains it very clearly (thanks @Korijn) and a comic.

At Facebook they came up with a pattern (and framework) that they call Flux. The same pattern has been used in different places, in different variants, e.g. Redux and Veux. The core principal is that information flows like this:

Actions --> Dispatcher  --> One ore more stores --> Views
                /\                                    |
                 - - - - - - - - - -  Actions - - - - -

The store(s) represent the state of the application. The views listen to changes in the state and update the UI accordingly. Views (and other parts of an app) can create actions (kind of events) that are handled by the store and eventually give rise to changes in the state.

An important concept is that any actions that are dispatched (e.g. from views) are only handled after the view is done processing stuff. That's why we can legitimate saying that data flows in one direction, even though we see an arrow going to the left. Also important is that updates to the store are atomic, "commiting mutations" in Veux-terminology. The store handles actions, and then either applies mutations, or it may e.g. call out to an API, schedule some work, and eventually (asynchronously) apply mutations. Or both. This page explains it in more detail.

The state in the store is a higher-level representation of the application state; instead of having a label for which its text property represents the username, the app would have a "username" entry in the store, and a view would be responsible for showing it.

Adopting this pattern in Flexx

My initial feeling is that with this model, Flexx can look like this:

Py                             |                  JS
                               |
                        -------------------
     Server stuff <--   | Stores / models |  --> Views / widgets
         |              -------------------               |
         -- actions -->        |             <-- actions --
                               |

Where a store is a bit like our current app.Model, having properties that are synchronised between Python and JS. Properties are not set directly, but via actions that can come either from JS or Python. The views/widgets live completely in JS (they have no representation in Python).

Advantages

• Business logic is separated from UX.
• Its easier to reason about how actions change state and how state affects the UI.
• Js and Python can no longer have conflicting states, which can lead to hard-to-predict behavior.
• Due to the above points, larger applications become much easier to maintain.
• It becomes possible to set the state of an app, log actions, log the history of the state, etc. which can
  be awesome during developing and debugging.
• The server is only concerned with stuff that matters (app state, and server specific tasks), and is unaware of all the properties of all the widgets as is the case now. This should make the server more lean for web apps.
• If Widgets are JS-only, they are lighter, and we can instantiated them at runtime from JS. This also avoids restrictions such as discussed in Tree item in JS #301.
• In-line callbacks for widgets are now in JS, which can help writing in a more declarative style (e.g. more like Enaml, Use with Enaml #359). Callbacks in widgets will typically also be smaller (emit a certain action).

Disadvantages

• Ppl will have to rewrite their apps
• Simple apps become slightly more complex? Maybe not, if a store is optional. Widgets can always have internal state.

I'll make another post to give an idea of how the code would look like. This is something that I need to work on some more, and which I want to compare against current examples and apps.

[Korijn]

I'm missing one element: Mutations. See here:

https://vuex.vuejs.org/en/mutations.html

Specifically the relationship to Actions and the store is very important and should not be skimmed over.

[almarklein]

Thanks, updated the description.

[almarklein]

Some more observations about "mutations" ...

Setting properties on the current app.Model are not atomic, but "eventually synchronous". The reasoning is that when setting mymodel.foo = 3, one expects mymodel.foo to be 3 immediately; the property is "mutated" at the JS/Py side, and then synchronized. This means that Python and JS can apply conflicting changes to the store. A good example is "flickering" when a value is changed a lot over a short period of time (from Python). The states will eventually be synchronized, but in the mean time, other parts of an app can react on the state changes, which can lead to behavior that is difficult to predict.

By letting go of this settable-properties-approach, and thinking in mutations, changes to a model/store can be made atomic by always applying the change in JS first, and then syncing with Py. Technically, there is a brief moment when the states in Py and JS are not the same, but this state is Python being behind JS, not conflicting with it.

edit: also important: a mutation can set multiple values in one "commit", e.g. append a todo item and increase the counter.

[Korijn]

Something that also came to mind: for some application state you should be able to specify if it is server or client state only, to disable syncing.

For example, a Sign In action that retrieves an authorization token for an external API should be able to commit that token to the store for later use in queries; but that token belongs to the user, and should never be synced to the server. That would be a serious security risk.

I'm wondering if that should be specified per property or per store though.

---

In reply to your further thoughts about mutations:

changes to a model/store can be made atomic by always applying the change in JS first,

Is that feasible though? Can't the server react to state changes and commit mutations as well?

[almarklein]

for some application state you should be able to specify if it is server or client state only, to disable syncing.

That crossed my mind as well. Another use-case is when the client modifies a state very fast (e.g. during mouse interaction). One idea for this case could be to temporarily shut down syncing of the store, and when turning it back on, the store has remembered (a squashed version) of all changes and syncs that with Py. For other cases (like the one you mentioned) I'd say we have stores that don't sync at all. Perhaps they should be JS-only; I'd not want to put sensitive data in a store that is capable of sending data to the server, and only prevented to do so by a flag :)

Can't the server react to state changes and commit mutations as well?

The problem with that is that if server and client commit a mutation to the same property, then both sides of the store will have conflicting states. Plus you need a mechanism to resolve the conflict eventually, which comes down to always roundtrip from Python to JS (or vice verse), which can cause "flickering". All this can be resolved by only committing mutations in JS, at the cost of not being able to apply a direct change from the server. I can imagine that the server can "submit" mutations that are first communicated to JS and then applied, but I think you'd call these "actions".

edit: Thinking of this more, the server could apply mutations to store attributes as long as these attributes are not mutated from JS. So it could be determined per attribute/store which side does the mutations.

[Korijn]

I totally agree and like the idea. I'm just wondering what would be left to do for the server at this point. If it won't react to state changes, what else will it do at runtime?

[almarklein]

The server can still react to state changes, it's just that the state can't be mutated directly; any change submitted by the server will first be applied in JS and then synced back to Py. How this will work exactly I don't know yet, perhaps by dispatching a new action for which there is a handler at the JS side.

I am working on a sorts of prototype, based on the existing Model class, to get a feeling of how this can look. I will post more when it starts to take shape.

[almarklein]

I found this an interesting article: http://www.christianalfoni.com/articles/2015_08_02_Why-we-are-doing-MVC-and-FLUX-wrong It explains how classic MVC used to be a good model when the model represented the state of the server. When we moved everything to the front-end, some separations were removed, leading to a tighter coupling between model and view, making it harder to scale. The purpose of tools like Flux is to introduce a decoupling, by queuing actions and handling them after the views are done. It is somewhat critical on Flux though, not sure yet what to think of that.

edit: it does make clear to me, that there are different kinds of MVC, MVC is not what it used to be, and Flux/Veux et. al. can easily be considered just another flavor of MVC.

[Korijn]

There is some strong commentary in that article. You can tell that Vuex incorporated some of that feedback in its design, when comparing it to Flux.

I'm starting to feel that this issue's main change to flexx would be to separate view components from models.

[JohnLunzer]

I can't say I follow this too closely, I've not gotten too much into the guts of Flexx or other frameworks to understand the pros/cons. That said there appear to be two components that should help drive your decision. Looking at your description and looking through the conversation history it seems like the proposed model has clear advantages and minor disadvantages. If your goal is to create the best framework possible, given the alpha state of the software, it seems like the "right" decision would be to adopt and implement the improved model.

If you don't do it now then you'll likely never get the chance. Those users consuming your alpha software are capable enough to deal with a major architectural change and if the stated advantages are as you've described, they will be grateful for the changes in the long run.

[JohnLunzer]

Additionally here is the bottom of the page from the intro to Veux that I found salient:

When Should I Use It?

Although Vuex helps us deal with shared state management, it also comes with the cost of more concepts and boilerplate. It's a trade-off between short term and long term productivity.

If you've never built a large-scale SPA and jump right into Vuex, it may feel verbose and daunting. That's perfectly normal - if your app is simple, you will most likely be fine without Vuex. A simple global event bus may be all you need. But if you are building a medium-to-large-scale SPA, chances are you have run into situations that make you think about how to better handle state outside of your Vue components, and Vuex will be the natural next step for you. There's a good quote from Dan Abramov, the author of Redux:

Flux libraries are like glasses: you’ll know when you need them.

I think if Flexx could incorporate a Flux model without forcing the use of a Flux model that might be the optimal solution. If Flexx forced a Flux model without added complexity for simple applications that might also be optimal.

[almarklein]

@JohnLunzer I agree; good to hear at least one user would not mind a change like this :)

I've been thinking a lot about how Flexx could like like when ideas like this are included. It's hard though. We also have the Python-JS connection/separation to take into account. I agree with @Korijn that making Widgets JS-only should be part of the solution. The idea of a store-like model that can only be manipulated via actions also seems like a good idea, but it makes the event system be/feel different from how to observe changes in the UI. I don't want users to learn two systems, they'll get confused. Work in progress ...

[Korijn]

In react/vue widgets have their own observable state. It's when your application grows that the store becomes useful and you can add it to the mix. That makes it less confusing.

[almarklein]

A quick message to let ppl subscribed to this issue know that I am still working on this (every day). I'm working on a proposal of changes and documentation for the newly proposed Model class. I've found that writing things down leads to new insights and realizations of having overlooked something. Also I want to match the new ideas to existing code to see whether things actually improve. Things are certainly converging, but it needs a bit more work ...

[almarklein]

A proposal is waiting for reactions at #367

[Korijn]

I like the example shopping cart. What would the widgets look like to display the products and the cart?

---

I'm also wondering why the store is self.root? It's true that you'll be working with a hierarchy of Models, but really, everything except the top will be a Widget, if I understand correctly. So it feels like self.model or self.store would be more appropriate.

[almarklein]

What would the widgets look like to display the products and the cart?

Depends on how you'd render them, but they'd react to the store's product property to update the quantity.

I'm also wondering why the store is self.root?

It does not have to be. It's up to the user to design the hierarchy of models in an appropriate way. I updated the shopping cart example to have a widget as the root model, which has a store and cart submodel. So the cart can access the store via self.root.store. But other approaches are possible as well.

[Korijn]

Depends on how you'd render them, but they'd react to the store's product property to update the quantity.

Can you show what that would look like? It's really important to me that that becomes simple. :)

[almarklein]

Can you show what that would look like?

added

[Yardanico]

@almarklein offtopic, but:
is there any IRC channel or chat about flexx?

[almarklein]

is there any IRC channel or chat about flexx?

Not yet. We could think of a gitter channel ...

[Yardanico]

@almarklein yes please! it's very easy to setup and maintain

[almarklein]

https://gitter.im/zoofIO/flexx