General architecture & patterns for status react

[janherich]

Preamble

Idea: <to be assigned>
Title: General architecture & patterns for status react
Status: Draft
Created: 2017/11/22

Summary

Define and adhere to architecture vision for status react. Have all the best practises clearly defined in one place.

Vision

Our architecture (shape and organisation of data + code) is not optimal, what's worse it's nowhere clearly defined and discussed about. The purpose of this Idea is to improve it by identifying all the problematic parts of our current architecture, capture them as user stories, fix them and properly document it for future reference.

Swarm Participants

• Lead Contributor: @janherich
• Testing & Evaluation:

Requirements

1. Re-frame database shape is reflecting the the app access patterns (writing for handlers, reading for subscriptions) by correctly indexing for fast lookups and normalised storage for fast, consistent updates
2. Subscriptions form efficient signal graph, with minimal amount of Level-1 subscriptions and highest possible re-use of all subscriptions
3. All the events in application are related to outside inputs and there are no artificial intermediate events anymore
4. Co-effects and effects are isolating the side-effects everywhere and we re-use them to highest possible degree

Goals & Implementation Plan

Minimum Viable Product

Goal Date: 2017-12-08
Description:

• Database is refactored according to point 1

Iteration 1..N

Goal Date: 2017-12-08
Description: Requirement 1 should be done

Goal Date: 2017-12-24
Description: Requirement 2 should be done

Goal Date: 2018-01-14
Description: Requirement 3 should be done

Goal Date: 2018-02-07
Description: Requirement 4 should be done

Supporting Role Communication

Post-Mortem

Copyright

Copyright and related rights waived via CC0.

[janherich]

User Stories

1. Re-frame database shape is reflecting the the app access patterns (writing for handlers, reading for subscriptions) by correctly indexing for fast lookups and normalised storage for fast, consistent updates

When designing the data-structure of re-frame app-db, always have in mind how the data will be accessed and how it will impact performance and consistency.
For example whenever you need random access to any piece of data which will be red much more often then wrote, put it in map and construct the key in such way that simple and fast lookup (without additional filtering) is guaranteed, even if it means that construction of key is little bit more complicated/slower.
If something has to be unique, put it in set.
Any derived data shouldn't be in database at all and should be computed from source data on fly by default (in super rare cases, we can revisit that when it would hurt performance).

Current situation

Some derived data are stored in db and recalculated explicitly (tags for discovery statuses). But the most problematic thing is proliferation of data indexed by message-id (:message-data, :message-id->transaction-id, :message-status, :unviewed-messages, :handler-data), which all are just workarounds around the fact that the primary path where messages live (:messages) is not indexed in any way.

2. Subscriptions form efficient signal graph, with minimal amount of Level-1 subscriptions and highest possible re-use of all subscriptions

There is not much to elaborate on this, whenever some subscriptions needs input, always check if the input is not already provided by another subscription, so you won't be creating another Level-1 subscription when absolutely not necessary.
Also, whenever you use some function to compute subscription values, make sure that the function is only taking what's necessary to produce result.

Current situation

Some subscriptions are essentially duplicated, and there are cases where subscription function takes whole app-db as argument, even when not necessary - https://github.com/status-im/status-react/blob/develop/src/status_im/chat/subs.cljs#L109-#L114

3. All the events in application are related to outside inputs and there are no artificial intermediate events anymore

This is maybe the most ambitious task of the whole idea. The vision is to only have such events, which naturally maps to asynchronous input from "outer world" into our application. Good examples are user actions (typing, swiping), network callbacks, callbacks from phone modules (camera, permissions, etc.).
If any operation the events perform needs to be re-used (like reloading contacts) it will be exposed as a plain (pure) function which operates with re-frame cofx map on input and produces fx map as output.
This approach, when implemented consistently everywhere has huge potential benefits:

• We will have much less events exposed and they will completely describe the "surface" area of our application (think about role of commands in CQRS event driven systems)
• We will reduce the the amount of subscription re-computations, as for each distinct action, :db effect will be produced and swapped into app-db only once
• Most of the system logic will be captured in pure functions, which will be hooked to minimal possible amount of events - dream to test, play with in repl, etc.
• Ugly non idiomatic events like [:set ...] or [:set-in ...] will disappear

Possible problems are that when you try to combine data from multiple coeffects consuming and effects producing functions, it will get non-trivial, for example sometimes you have to check for :db both in coeffects (if no function produced :db effect yet) and effects, you have to make sure that plural effects like saving something into realm are properly accumulated, etc.
But it's still just data munging and Clojure is wonderful language to work with data, so we will eventually evolve some helpers/patterns which will make such compositions a breeze.

Current situation

There are some namespaces where the approach above was already started (for example status-im.chat.events.input and status-im.commands.events.loading) but we still have to go a long way to implement it everywhere, the most problematic are in my opinion events in the contacts module and application startup events.

4. Co-effects and effects are isolating the side-effects everywhere and we re-use them to highest possible degree

Co-effects but most importantly effects only capture the essence of the side-effect operation (realm op, network call, etc) and nothing else, they are not highly specific and don't contain much logic.
The upside of coding them that way (always striving to make them absolutely minimal) is that more of our logic will live in the pure functional domain, which is much easier to test (you don't have to mock so much logic) and reason about, and additionally, as the effects will be fairly generic, they could be re-used to much higher degree.

Current situation

Although we moved to cofx/fx based handlers almost completely, there is very little in term of coeffects/effects standardisation, some of them are duplicated and registered under different keywords in different namespaces, some of them are really complex, specific to just one use case and containing a lot of logic, for example here - https://github.com/status-im/status-react/blob/develop/src/status_im/ui/screens/contacts/events.cljs#L145-#L159 (which should be just :http effect with callback logic not owned by effect)

[yenda]

I can pledge some of my time that I spend in idea 9 since the two seems closely related

[naghdy]

Closing this as it hasn't been touched in ~3 months. Feel free to re-open if this is something that we are going to tackle soon