Trying to make the resampler work with dynamic graphs

[prokie]

So I made this minimal example but I can not figure out why I can't get the callbacks to work.

`

"""
Minimal dynamic dash app example.
"""

import numpy as np
import plotly.graph_objects as go
import trace_updater
from dash import Dash, Input, Output, State, dcc, html
from plotly_resampler import FigureResampler

x = np.arange(1_000_000)
noisy_sin = (3 + np.sin(x / 200) + np.random.randn(len(x)) / 10) * x / 1_000

app = Dash(__name__)

fig = FigureResampler(go.Figure(go.Scatter(x=x, y=noisy_sin)))


app.layout = html.Div(
    [
        html.Div(
            children=[
                html.Button("Add Chart", id="add-chart", n_clicks=0),
            ]
        ),
        html.Div(id="container", children=[]),
    ]
)


@app.callback(
    Output("container", "children"),
    Input("add-chart", "n_clicks"),
    State("container", "children"),
)
def display_graphs(n_clicks: int, div_children: list[html.Div]) -> list[html.Div]:
    """
    This function is called when the button is clicked. It adds a new graph to the div.
    """
    figure = fig
    figure.register_update_graph_callback(
        app=app,
        graph_id=f"graph-id-{n_clicks}",
        trace_updater_id=f"trace-updater-id-{n_clicks}",
    )

    new_child = html.Div(
        children=[
            dcc.Graph(id=f"graph-id-{n_clicks}", figure=fig),
            trace_updater.TraceUpdater(
                id=f"trace-updater-id-{n_clicks}", gdID=f"graph-id-{n_clicks}"
            ),
        ],
    )
    div_children.append(new_child)
    return div_children


if __name__ == "__main__":
    app.run_server(debug=True)

`

[jvdd]

Great to see you toying around with our package! 😃

Your code does not work because of a (large) limitation of dash; it is not possible to dynamically add callbacks in a dash app. In your example, you try to do this in the display_graphs function when you call the figure.register_update_graph_callback function.

To circumvent this, you should register the callbacks statically (and not dynamically in another callback function). A quick (but kinda dirty) fix for this is to create several dummy figures (i.e., FigureResampler() objects without any trace) and register the callbacks for these figures with unique ids.
You can still add the data to the figure dynamically via either the .add_trace method er the .replace method.

See this working version of your minimalistic example

"""
Minimal dynamic dash app example.
"""

import numpy as np
import plotly.graph_objects as go
import trace_updater
from dash import Dash, Input, Output, State, dcc, html
from plotly_resampler import FigureResampler

from typing import List

x = np.arange(1_000_000)
noisy_sin = (3 + np.sin(x / 200) + np.random.randn(len(x)) / 10) * x / 1_000

app = Dash(
    __name__, 
    suppress_callback_exceptions=True,  # surpress unused ID warnings
)

# Create empty figures
max_nb_figures = 10  # NOTE, 10 is arbitrarily chosen
figs = [
    FigureResampler()
    for _ in range(max_nb_figures)
]

# Add the resampling callback to these empty figures
for n_clicks, f in enumerate(figs):
    f.register_update_graph_callback(
        app=app,
        graph_id=f"graph-id-{n_clicks}",
        trace_updater_id=f"trace-updater-id-{n_clicks}",
    )


app.layout = html.Div(
    [
        html.Div(
            children=[
                html.Button("Add Chart", id="add-chart", n_clicks=0),
            ]
        ),
        html.Div(id="container", children=[]),
    ]
)


@app.callback(
    Output("container", "children"),
    Input("add-chart", "n_clicks"),
    State("container", "children"),
)
def display_graphs(n_clicks: int, div_children: List[html.Div]) -> List[html.Div]:
    """
    This function is called when the button is clicked. It adds a new graph to the div.
    """
    print(n_clicks)
    global figs
    # Select the n_clicks figure (with its already registered callbacks)
    figure = figs[n_clicks]
    # Add the data to the figure
    figure.add_trace(go.Scatter(x=x, y=noisy_sin))
    # NOTE, you could also use figure.replace(go.Figure(go.Scatter(x=x, y=noisy_sin)))

    new_child = html.Div(
        children=[
            dcc.Graph(id=f"graph-id-{n_clicks}", figure=figure),
            trace_updater.TraceUpdater(
                id=f"trace-updater-id-{n_clicks}", gdID=f"graph-id-{n_clicks}"
            ),
        ],
    )
    div_children.append(new_child)
    return div_children


if __name__ == "__main__":
    app.run_server(debug=True)

I hope this answer helps you!

(@jonasvdd can you think of another / cleaner solution?)

[prokie]

Okay, thanks for a quick response. But then it should be possible to use pattern matching callbacks? I will look into it and see if I can make that work.

[jonasvdd]

@prokie, I suppose it should! I will also look into this a little! Keep you posted! :)

[prokie]

It probably needs some work done on the TraceUpdater. I assume that register_update_graph_callback should work out of the box with something like this
figure.register_update_graph_callback( app=app, graph_id={"type": "graph", "index": ALL}, trace_updater_id={"type": "graph", "index": ALL}, )

But the TraceUpdater I don't really understand and it gives an error when you supply it a dict.
trace_updater.TraceUpdater( id={"type": "trace-updater", "index": n_clicks}, gdID={"type": "graph", "index": n_clicks}, ),

Error:
Invalid argument gdID passed into TraceUpdater with ID "{"index":0,"type":"trace-updater"}". Expected string. Was supplied type object. Value provided: { "type": "graph", "index": 0 }

[jonasvdd]

Working on it! Already have a p.o.c. which does what I think you intend, I think we will have a new release at the end of this week with a Dash example for this use-case.

I indeed had to make some changes to the TraceUpdater component! 😃

[jonasvdd]

WIP: see #35

[jvdd]

Is now supported in v0.3.0 🎉

[prokie]

One issue that I have found (using the example) is that if I actually try to deploy the server it won't work as intended since we store the FigureResampler objects globally in the dictionary graph_dict. Thus all users will share FigureResampler objects, however, the users want to work individually. I instead tried to store the objects using dcc.Store but that serializes the data to a dictionary. Do you know any solution that would work without storing the objects globally? @jonasvdd @jvdd

[prokie]

Maybe you can use dash-extensions and the function ServersideOutputTransform. This would allow storing objects without serializing them. Will look into this and come back.

[jonasvdd]

Hey @prokie,
Quick idea: is it possible to create a new UUID per user/session and know which user/session session makes a request? Because then, you can have a graph_dict which looks like:

{ 
   "<user_uuid_str>": {
          "<graph_uuid_str>": FigureResampler(...), 
          ... 
    },
    ...
}

I will give this later some more thought!

[prokie]

Yes that should work aswell.

[prokie]

@jonasvdd Tried using ServersideOutputTransform but the issue is the same. It pickles the FigureResampler object and when I pack it up it is no longer an object but a dictionary. I guess your quick idea is the way to go. It just seems a little "iffy" to store everything in a global variable.

[jonasvdd]

@prokie,
Mhmm interesting, I acknowledge that my proposed solution indeed is a bit "iffy".
Let me know whether you are able to get it working or find a better way to do so!

[prokie]

Have been thinking about this for some time now. And I have not been able to store the FigureResampler object without it being serialized in the backend. I guess the way I would want it to work was that you could send the figure to the callback and then create a FigureResampler object from it each time you have to resample the trace.

@app.callback(
    Output({"type": "dynamic-updater", "index": MATCH}, "updateData"),
    Input({"type": "dynamic-graph", "index": MATCH}, "relayoutData"),
    Input({"type": "dynamic-graph", "index": MATCH}, "figure"),
    prevent_initial_call=True,
)
def update_figure(relayoutdata: dict, fig):
    return FigureResampler(fig).construct_update_data(relayoutdata)

But this does not work and I guess I will have to dig in the TraceUpdater and FigureResampler components to figure out why exactly.

FYI:

With a flask backend, it is possible to use your selected approach by retrieving the username using the request function. @jonasvdd

from flask import request

# And grab the name in callback you want by
request.authorization['username']

[prokie]

Is it still necessary to save the Figure objects globally with the latest plotly release?
@jonasvdd

[jonasvdd]

@prokie! This is still necessary!

As for now this is a low-prio issue for us as it somehow works, and we are mostly using it for data exploration and not production ready apps. Always feel free to further dive into this or update this with additional insights!

[jonasvdd]

Hi @prokie,

We are currently looking into serializing plotly-resampler objects (in #87 ); which would enable the use of ServersideOutputTransform to store these objects more cleanly. Could you provide a minimal example where you use plotly-resampler & this ServersideOutputTransform component?

Cheers,
Jonas

[prokie]

Hi Jonas!

That would be awesome. I can make one!

[prokie]

Hi again,

I guess something like this should work. @jonasvdd

"""
Minimal dynamic dash app example.
"""

import numpy as np
import plotly.graph_objects as go
import trace_updater
from dash_extensions.enrich import (
    MATCH,
    DashProxy,
    Input,
    Output,
    ServersideOutput,
    ServersideOutputTransform,
    State,
    dcc,
    html,
)
from plotly_resampler import FigureResampler

x = np.arange(1_000_000)
noisy_sin = (3 + np.sin(x / 200) + np.random.randn(len(x)) / 10) * x / 1_000

app = DashProxy(__name__, transforms=[ServersideOutputTransform()])


app.layout = html.Div(
    [
        html.Div(
            children=[
                html.Button("Add Chart", id="add-chart", n_clicks=0),
            ]
        ),
        html.Div(id="container", children=[]),
    ]
)


@app.callback(
    ServersideOutput({"type": "store", "index": MATCH}, "data"),
    Input({"type": "dynamic-graph", "index": MATCH}, "figure"),
    prevent_initial_call=True,
)
def store_fig(fig):
    return fig


@app.callback(
    Output("container", "children"),
    Input("add-chart", "n_clicks"),
    State("container", "children"),
    prevent_initial_call=True,
)
def display_graphs(n_clicks: int, div_children: list[html.Div]) -> list[html.Div]:
    """
    This function is called when the button is clicked. It adds a new graph to the div.
    """
    figure = FigureResampler(go.Figure(go.Scatter(x=x, y=noisy_sin)))

    figure.register_update_graph_callback(
        app=app,
        graph_id=f"graph-id-{n_clicks}",
        trace_updater_id=f"trace-updater-id-{n_clicks}",
    )

    new_child = html.Div(
        children=[
            dcc.Graph(id={"type": "dynamic-graph", "index": n_clicks}, figure=figure),
            trace_updater.TraceUpdater(
                id={"type": "dynamic-updater", "index": n_clicks}, gdID=f"graph-id-{n_clicks}"
            ),
            dcc.Store(id={"type": "store", "index": n_clicks}),
        ],
    )
    div_children.append(new_child)
    return div_children


@app.callback(
    Output({"type": "dynamic-updater", "index": MATCH}, "updateData"),
    Input({"type": "dynamic-graph", "index": MATCH}, "relayoutData"),
    State({"type": "store", "index": MATCH}, "data"),
    prevent_initial_call=True,
)
def update_figure(relayoutdata: dict, graph: dict):
    return graph.construct_update_data(relayoutdata)


if __name__ == "__main__":
    app.run_server(debug=True)

[jonasvdd]

Hi @prokie,

This example seems to start, but I get the following error.

Traceback (most recent call last):
  File "/home/jonas/.cache/pypoetry/virtualenvs/plotly-resampler-aW2rMSDw-py3.8/lib/python3.8/site-packages/dash_extensions/enrich.py", line 1112, in decorated_function
    args[i] = serverside_output.backend.get(args[i], ignore_expired=True)
  File "/home/jonas/.cache/pypoetry/virtualenvs/plotly-resampler-aW2rMSDw-py3.8/lib/python3.8/site-packages/dash_extensions/enrich.py", line 1216, in get
    filename = self._get_filename(key)
  File "/home/jonas/.cache/pypoetry/virtualenvs/plotly-resampler-aW2rMSDw-py3.8/lib/python3.8/site-packages/cachelib/file.py", line 202, in _get_filename
    return os.path.join(self._path, bkey_hash)
UnboundLocalError: local variable 'bkey_hash' referenced before assignment

It seems like it is the error from this issue: emilhe/dash-extensions#185, but as I'm not that acquainted with that codebase, it is rather hard to find out what I'm doing wrong.

fyi: I'm working with the plotly-resampler code on the master branch, which normally has serialization support (not yet released in a PyPi version)

Are you able to get this running?

[prokie]

I can test it out tomorrow. I did not know that serialization support was done!

Yes that error is related to flask_caching, I think the issue was fixed in 0.1.4 for dash_extensions, before you had to run with cachelib==0.6.0. But I will test tomorrow and get back to you.

[jonasvdd]

Hey! @prokie

Just wanted to mention that I got it working! 🔥
No global variable needed, using the ServerOutput to have independent graphs for each user / session.

Will share later on today my working example (It did require some small adjusting of the dash-extensions because I could not get the default lib working on my machine).

[prokie]

Great looking forward to see it. I also made an example that works. I put it in your issue over at dash-extensions.