Optimize the client-server communication

[petar-qb]

Description

By merging some callbacks and rewriting some of them into the clientside_callback form (pure javascript code processed on the user's client browser machine), the rendering time (when someone opens a new page or applies some action like filter, export_data...) is significantly reduced.

Progress:

• Implement action loop trigger_to_global_store as clientside_callback.
• Merge gateway , set_remaining and executor into one single callback.
• Rewrite the merged (new gateway) callback as clientside_callback.
• Merge after_action and cycle_breaker callbacks into one clientside_callback.
• Overwrite RangeSlider, Slider, ThemeSelector and other callbacks in the system to clientside_callback.
• Try to merge two callbacks where input is theme_selector. (IMPOSIBLE because Patch object isn't supported in the clientside_callback) - actually it this fix isn’t needed.
• Add tests for client-side callbacks written in Node.js tool called jest (run with command: hatch run test-js)

Speed-up

	opening a new page	filer action	export_data action
Current system	4050 ms	3100 ms	3300 ms
Optimised system	850 ms	490 ms	470 ms
Speed-up	4.75x	6.3x	7.0 x

Screenshot

Action loop architecture for the current system:

Action loop architecture for the optimised system (after we rewrite the gateway callback into the client-side one):

Consequences

• There's no more logs on the server-side that show which component is triggered and what actions chain will be executed.
• Since many callbacks have been moved to javascript code, we can no longer unit test them using the pytest.

To be discussed (not the part of this PR):

• It's probably possible to save the vizro.themes on the frontend side as well? Then to convert the update_graph_theme callback into the clienside_callback too.
• Try to get rid of dash.pages / or wrap (decorate) it.
• At the user's request, enable keeping original (or filtered) data on the frontend, instead of in the DataManager.
• Put some kind of @lru_cache on the server functions.

ToDo:

• Handle when a new actions chain is started before the previous one has finished.

Checklist

• I have not referenced individuals, products or companies in any commits, directly or indirectly
• I have not added data or restricted code in any commits, directly or indirectly
• I have updated the docstring of any public function/class/model changed
• I have added the PR number to the change description in the changelog fragment, e.g. Enable feature XXX ([#1](https://github.com/mckinsey/vizro/pull/1)) (if applicable)

Types of changes

• Docs/refactoring (non-breaking change which improves codebase)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

[huong-li-nguyen]

Took a quick look - the example dashboard definitely feels quicker! 🚀 🐎 Great job, Petar!

I have a few open questions, some suggestion but would have to take a detailed look when I am back! If the PR is ready by this week, I'll leave the review to Max or Lingyi :)
Otherwise i'll take a look when I'm back

[lingyielia]

Another future use case relevant to this topic: when a dashboard needs to connect to the database and pull data from it,

Clientside callbacks are not possible if you need to refer to global variables on the server or a DB call is required.

https://dash.plotly.com/clientside-callbacks

How should we construct the action chain if there is a DB call involved

[petar-qb]

@lingyielia

How should we construct the action chain if there is a DB call involved

Only the internal mechanism (the mechanism that allows the action loop to work) is implemented using clientside-callbacks. If someone needs a DB connection, they can create a custom action or we can release a predefined action for that (either way it's an Action). Remember that Action callbacks are blue rectangles in the architecture sketch (look Description) which means that they are not client-side callbacks, but regular dash.callbacks that "live in the server".

[maxschulz-COL]

Thanks for the optimizations! Started a discussion on a few routes we can take to move forward

[maxschulz-COL]

LGTM!

Have a few very small suggestions that we should do before merging. Anything marked as (for later) please ignore until we have merged.

[lingyielia]

LGTM! See great speed improvement

[maxschulz-COL]

Lgtm now!