Mechanism for Python webserver to send progress bar updates to Electron

[simonw]

This is going to be quite a fun challenge: for various operations such as loading a CSV (#68) or downloading a CSV from the internet (#107) the application needs to be able to display a progress bar.

But... the progress is happening inside the Python process. So there needs to be a way to communicate that progress back to the Electron app in order for it to be displayed to the user.

[simonw]

Some idea:

• Use stdout - the server process could print out structured log entries to stdout which the Electron process could pick up on
• Polling: the Electron app could poll some endpoint on the Dataesette server for progress reports (probably easiest to implement)
• WebSockets: establish a WebSocket connection between the server and Electron and communicate this stuff that way (most exciting to build, but probably not worth the added complexity)

[simonw]

I think polling is the most sensible way to do this.

API calls like POST /-/open-csv-from-url could return a {"poll": "/-/progress?token=xyz"} response which Electron can then poll for progress reports.

This will be very easy to implement and could be used for all kinds of other interesting things too.

[simonw]

Could even have a bit of code inside the Electron wrapper that knows how to poll for update and turn changes in poll status into EventEmitter events that can be subscribed to within the app.

Also makes it easy to write JavaScript in browser windows that polls for /-/progress?token=xyz and updates a progress bar directly on the page, without any involvement of the Electron shell at all.

[simonw]

The _desktop database from #101 could include a table that is dedicated to progress tracking, as previously seen in plugins like datasette-upload-csvs, see https://github.com/simonw/datasette-upload-csvs/blob/191458c5d50d8faafca168cac9d23b20452e33ce/datasette_upload_csvs/__init__.py#L74-L88

[simonw]

... or that progress table could live in temporary depending on if we want it to survive application restarts (which _desktop does but temporary does not).

[simonw]

And then how to display them?

I custom smaller-sized BrowserWindowwith a progress bar could work well: it could show a title, the progress bar, an optional "what I'm doing right now" message and know how to do the fetch() polling against the progress API - then when progress completes it could call back to an IPC API to say "OK close me and open this other path instead".

[simonw]

Demo:

progress-bar-demo.mov