Workspace persistence is fragile on localStorage; move graph/session storage to IndexedDB with migration + recovery

[avinxshKD]

We currently autosave full graph payloads in localStorage. This is brittle for larger workflows and can silently drop history when writes fail. Session restore is also partial (open tabs, active tab, and file metadata are not reliably recovered).

Based on discussion, this will follow a hybrid storage model:

keep filesystem storage for file/binary artifacts (imports/exports)
move graph/session metadata persistence to IndexedDB

This keeps binaries out of DB, while fixing reliability for autosave + restore.

Scope

• add a versioned IndexedDB storage layer for graph/session metadata
• run one-time migration of existing localStorage metadata on first successful init
• wire autosave/load/session-restore to the new metadata store
• handle corrupted entries by skipping bad records and continuing load
• keep a sane fallback path when IndexedDB is unavailable
• remove silent history-loss behavior during normal saves
• add automated tests for migration, restore, corrupted-record skip, and IndexedDB-unavailable fallback
• add docs for schema, migration flow, fallback behavior, and manual verification steps

Acceptance criteria

• reload restores previously open graphs and active tab
• large workflows save/load reliably without quota-related loss
• existing users are migrated automatically
• one corrupted record does not block full workspace restore

[avinxshKD]

Hey @pradeeban what do you think about adding this? It should make saves more reliable for bigger workflows and stop the random session/history loss we still see with localStorage.

[pradeeban]

Fair points about the local storage. But then the counter point is, often filesystem is better than database to store binary data. If you are thinking of indexing and retrieval, perhaps a hybrid approach of storage of the files in the local file system and the metadata in a database could be the winner approach.

[avinxshKD]

@pradeeban Thanks, that makes sense. I agree with the hybrid direction. We can keep filesystem storage for files/binary exports, and use IndexedDB for autosave/session metadata + recovery (graph state, history, open tabs, active tab). That should fix the localStorage reliability issues without moving binaries into DB. If you’re aligned with this scope, I’ll update the issue and start implementation :p

[pradeeban]

I had a similar discussion when I started implementing Niffler back in 2020. I went with local file-system based storage for the binary data (DICOM images) with metadata extracted into csv formats which itself then stored into a MongoDB instance as a metadata archive. I am seeing parallels.

[avinxshKD]

Thanks for the context. Got it, that lines up well. I’ll update this to make the hybrid split explicit: filesystem for binaries/exports, IndexedDB for graph/session metadata with migration + fallback. I’ll open a PR for the storage layer and migration path.

[avinxshKD]

@pradeeban Updated the issue with the hybrid scope.

[pradeeban]

@avinxshKD perfect.