Add persistence support

[PardhavMaradani]

Changes made in this Pull Request:

• Added persistence support for dashboard
  • All widget instances state (inputs) saved in state manager
  • State saved (and read) as a JSON file
  • All widget instanes re-created on server startup
• Added a new arg to mdadash to specify custom state file if needed
• Refactored sio handlers and core managers in main.py to separate Class
• Prefixed widget lifecycle handler methods with on_ for consistency
• Tests for all new functionality

PR Checklist

• Tests?
• Docs?
• CHANGELOG updated?
• Issue raised/referenced?

---

Hi @orbeckst, @jeremyleung521, @amruthesht, @HeydenLabASU,

This PR adds support for persistence. Persistence across multiple browsers (users) always existed from the beginning - this was as long as the server was running. With this PR, even the server process can be restarted and it will restore any widgets added - along with their configured inputs, settings, layout etc.

Here is a short video that shows the server being restarted while an existing simulation was running. You can also see the dashboard showing a disconnected state and automatically reconnecting to the dashboard server once it is available and restored to the previous widgets:

mdadash-persistence-reconnect.mp4

Here is another short video that shows the UI launched from scratch after the server starts (and restores the previous state):

mdadash-persistence-scratch.mp4

The saved state file is a JSON file and has all the data required to restore the widgets to their configured state. Here is an example of the state file for the videos shown above:

Show example of the saved JSON state file

{
    "running_state": {
        "connected": true,
        "message": "",
        "pending": false,
        "running": true
    },
    "settings": {
        "dashboard_config": {
            "n_jobs": 2,
            "show_energies": true,
            "show_session_info": false,
            "ui_request_timeout": 5
        },
        "universe_configs": [
            {
                "batch_size": 10,
                "buffer_size": 10000000,
                "continue_after_disconnect": null,
                "kwargs": [],
                "socket_bufsize": null,
                "step": 1,
                "timeout": 5,
                "topology": "/tmp/start.gro",
                "total_steps": 500000,
                "trajectory": "imd://localhost:8889"
            }
        ]
    },
    "version": 1,
    "widgets": {
        "88c4b966-7476-11f1-8934-deda9e2ed9f4": {
            "class_name": "COMDistance",
            "inputs": [
                {
                    "attribute": "selection1",
                    "error": null,
                    "value": "resid 1"
                },
                {
                    "attribute": "selection2",
                    "error": null,
                    "value": "resid 129"
                },
                {
                    "attribute": "periodic",
                    "error": null,
                    "value": true
                },
                {
                    "attribute": "updating",
                    "error": null,
                    "value": false
                },
                {
                    "attribute": "custom_title",
                    "error": null,
                    "value": null
                },
                {
                    "attribute": "maxlen",
                    "error": null,
                    "value": 100
                },
                {
                    "attribute": "max_distance",
                    "error": null,
                    "value": 5.0
                },
                {
                    "attribute": "max_distance_alert",
                    "error": null,
                    "value": false
                },
                {
                    "attribute": "x_type",
                    "error": null,
                    "value": "time"
                }
            ],
            "uid": 0
        },
        "954f0b64-7476-11f1-8934-deda9e2ed9f4": {
            "class_name": "Total Energy",
            "inputs": [
                {
                    "attribute": "maxlen",
                    "error": null,
                    "value": 100
                },
                {
                    "attribute": "title",
                    "error": null,
                    "value": "Total Energy"
                },
                {
                    "attribute": "x_type",
                    "error": null,
                    "value": "step"
                }
            ],
            "uid": 0
        },
        "98470358-7476-11f1-8934-deda9e2ed9f4": {
            "class_name": "ROG",
            "inputs": [
                {
                    "attribute": "_run_frequency",
                    "error": null,
                    "value": "per-frame"
                },
                {
                    "attribute": "_run_mode",
                    "error": null,
                    "value": "serial"
                },
                {
                    "attribute": "selection",
                    "error": null,
                    "value": "resid 2"
                },
                {
                    "attribute": "periodic",
                    "error": null,
                    "value": true
                },
                {
                    "attribute": "updating",
                    "error": null,
                    "value": false
                },
                {
                    "attribute": "custom_title",
                    "error": null,
                    "value": null
                },
                {
                    "attribute": "maxlen",
                    "error": null,
                    "value": 100
                },
                {
                    "attribute": "x_type",
                    "error": null,
                    "value": "time"
                }
            ],
            "uid": 0
        }
    },
    "widgets_layout": [
        {
            "description": "Distance between two COMs",
            "h": 9,
            "i": "88c4b966-7476-11f1-8934-deda9e2ed9f4",
            "moved": false,
            "name": "COMDistance",
            "w": 4,
            "x": 0,
            "y": 0
        },
        {
            "description": "Plot of Total Energy",
            "h": 9,
            "i": "954f0b64-7476-11f1-8934-deda9e2ed9f4",
            "moved": false,
            "name": "Total Energy",
            "w": 4,
            "x": 4,
            "y": 0
        },
        {
            "description": "Radii of Gyration of a selection",
            "h": 9,
            "i": "98470358-7476-11f1-8934-deda9e2ed9f4",
            "moved": false,
            "name": "ROG",
            "w": 4,
            "x": 8,
            "y": 0
        }
    ]
}

The project proposal and the timeline mentioned adding an import / export button to the GUI. Adding an export button to return this JSON state file is pretty straightforward. However, it isn't very clear how the import path will work because the dashboard server needs to be restarted. For this reason, I skipped adding that part to keep it simple. The JSON file also makes it human readable, easy to share, source control friendly, etc.

This persistence support has already helped me with some manual testing during the development process as I didn't have to re-create different scenarios each time.

Thanks

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (8d2c09f) to head (4f9aa93).

Additional details and impacted files

Components	Coverage Δ
frontend	100.00% <ø> (ø)
backend	100.00% <100.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[read-the-docs-community]

Documentation build overview

📚 mdadash | 🛠️ Build #33385483 | 📁 Comparing 4f9aa93 against latest (8d2c09f)

  🔍 Preview build  

15 files changed · ± 15 modified

± Modified

• genindex.html
• autosummary/mdadash.backend.analyses.com_distance.html
• autosummary/mdadash.backend.analyses.energies.html
• autosummary/mdadash.backend.analyses.rog.html
• autosummary/mdadash.backend.kernel.core.html
• autosummary/mdadash.backend.kernel.manager.html
• autosummary/mdadash.backend.state.manager.html
• autosummary/mdadash.backend.widgets.base.html
• _modules/mdadash/backend/analyses/com_distance.html
• _modules/mdadash/backend/analyses/energies.html
• and 5 more...

[orbeckst]

Great addition. I agree with not implementing import.