🩺 Process status

[fonsp]

At any point (especially in the first minutes), we want to know what is currently loading/running/compiling, and show it to the user with #2376

Schermopname.2022-11-30.om.15.45.59.mov

New internal API

This PR adds new API that we can use in the backend to report a status item, e.g.:

Status.report_business_started!(notebook.status; name=:export)

generate_export(notebook)

Status.report_business_finished!(notebook.status; name=:export)


# Alternatively, you could do:

Status.report_business!(notebook.status; name=:export) do
	generate_export(notebook)
end

This will:

• Create a "child task" of notebook.status, which is the "root task" of this notebook: all business is reported there
• Record the starting & finish times
• Update all connected clients, with a throttled update function

Subtasks

Tasks form a tree structure (of subtasks), e.g."

export_status = Status.report_business_started!(notebook.status; name=:export)
Status.report_business_started!(export_status; name=:html)
Status.report_business_planned!(export_status; name=:jl)

generate_html(notebook)

Status.report_business_finished!(export_status; name=:html)
Status.report_business_started!(export_status; name=:jl)

generate_jl(notebook)

Status.report_business_finished!(notebook.status; name=:export) # will automatically finish all child tasks

The current notebook.status will be included in the statefile, making it easy to build a frontend GUI!

MVP TODO

• Timings have the potential to be a security risk, because they give insight into a computer. (This is why JS timers have a small amount of random noise.) I don't think these risks apply to us, but let's think about it, and maybe add noise or rounding.
  • Conclusion: is does not create an additional security risk for these situations:
    • Pluto in normal use: no risk, this data is only available if the entire notebook is available.
    • Pluto running as public API with PlutoREST or PlutoSliderServer: code execution is already timed, these extra timings don't give additional precision into timing user code.
• Drift between browser time and server time can be noticable. Putting the current time in the statefile is not a good idea, so maybe lets add a new dynamic request endpoint to get the time and calculate offset.
• Mirror pkg terminal in pkg progress? Show Pkg logs in Process Status tab #2498

Future work

This is also automatically adding lots of benchmarks to everyone's Pluto installation, cool! This is useful to us, as it can help with debugging when people send us an HTML export. Maybe there are other cool uses?

We definitely want user-friendly descriptions of these task (rather than pkg_analyze), and maybe we could also write a short sentence that is shown when that task is stuck. E.g. if pkg_analyze takes a long time, we can show info about Julia Pkg downloads and precompiling.

Maybe we also want report_business_failed! for when a task fails.

For some tasks, we could give an "expected time". If it's more than that time, don't show it (like the 1ms save). BUT: comment from panagiotis and michiel: we should show it, otherwise it's scary that it suddenly disappeared.

resolve_topology should go in workspace setup

Even more!!! report_progress! lines, to give super detailed timings. We don't show those by default, but you can turn on "dev mode"

Maybe: the Pluto notebook is always a "snapshot": it is what it currently is, past has no effect. But maybe this new tab should be the way out of this perfect world: it should actually show history, execution order, etc.

Maybe instead of a global map (that overwrites itself), it could be an append-only list.

Special-case cells that do an import or using

Add requests like:

• interrupt
• doc query 100ms
• autocomplete 100ms
• state update?
• resync operational tranforms (after the PR)

[github-actions]

Try this Pull Request!

Open Julia and type:

julia> import Pkg
julia> Pkg.activate(temp=true)
julia> Pkg.add(url="https://github.com/fonsp/Pluto.jl", rev="status")
julia> using Pluto

After loading Pluto, open a notebook, and then press F12 to open the web developer tools. Go to the JavaScript console, and type:

PLUTO_TOGGLE_PROCESS_TAB()

to enable the feature.

[fonsp]

@disberd We could use this new tab for extra Pkg GUI like the additional GUI for #2245

[fonsp]

After yesterday's Pluto Community Call (join us on zulip to learn more), we found that showing each cell evaluation is a bit confusing/misleading, because it does not mirror the full list of cells.

So I decided to collect all numbered tasks, and display them all as a single progress bar instead. I think it looks super nice!

Before

Schermopname.2022-11-28.om.21.03.26.mov

After

Schermopname.2022-11-30.om.15.45.59.mov

[vangberg]

Hi @fonsp

First, I think this looks super nice, and trying it out on a heavy notebook, it gives a really good overview of what is going on. I imagine this will be super helpful when debugging when a notebook is stuck, as hinted at in another comment:

maybe we could also write a short sentence that is shown when that task is stuck

One suggestion I have, is to think about another scenario, which is what lead me to this issue in the first place: I open a notebook, and I want to know when everything has been loaded, before I start interacting with the notebook.

This tab does not directly solve that, as I have to click on it, and open it. It feels more like actively debugging/investigation than passively waiting for the "green light".

I realize that the narrow progress bar in the very top of the screen indicates this (I think), but to me there are two problems with that:

1. It is very unnoticeable. It took me some time to even realize it was there.
2. The meaning of it is overloaded. Often it is used to indicate "loading page" (e.g. on GitHub). Mentally "loading page" did not include "… and running cells" for me.

However much I try to restrain from going into "solution mode", I can't help it, so here is a suggestion: what if the status bar included the "sum status", something like this:

Hope this was in some way helpful 😊

[disberd]

@disberd We could use this new tab for extra Pkg GUI like the additional GUI for #2245

You mean also have the extra GUI for specifying custom package version/location inside the status tab, or as a separate tab within the docs/status folder

[fonsp]

@vangberg Are you on the last commit? Right now, this is what the status tab looks like while the notebook is starting.

Schermopname.2022-12-06.om.18.11.37.mov

[vangberg]

@vangberg Are you on the last commit? Right now, this is what the status tab looks like while the notebook is starting.

I guess I wasn't. That's perfect, though. 😊

[fonsp]

Releasing this feature behind the PLUTO_TOGGLE_PROCESS_TAB flag!

After loading Pluto, open a notebook, and then press F12 to open the web developer tools. Go to the JavaScript console, and type:

PLUTO_TOGGLE_PROCESS_TAB()

to enable the feature.