Dialog stores data very inefficiently in localStorage

[curiousdannii]

It was reported here that Counterfeit Monkey can have trouble saving/restoring save files, but it can happen to any storyfile: i7/counterfeit-monkey#191. More discussion on the forum.

The issue is that the localStorage is getting too full to store the save file, which is about 150KB, out of the total 5MB that the browser can store. If you've been playing lots of things in Parchment then even if you haven't been making lots of save files there could be lots of autosaves taking up space. Then when you try to restore, it fails because the save file is empty.

Parchment/Dialog could be storing data in localStorage much more efficiently, and it could be using IndexedDB, which has much bigger limits (over a GB I think), and it could also be evicting autosaves to make space. I have plans to make those sorts of changes (#11), but they're not short term plans, it will take a lot of work to get everything ready.

What can we do about this?

• Without changing the whole Dialog system, we could make it store files more densely. Each byte currently takes up about ~3.5 characters, but it could be compressed using base32768 into ~0.5 characters/byte. (I was researching this literally 2 days ago.) That's an improvement of 7 times!
• Autosaves should be split into two localStorage files - one for text and one for the ram. Both could be compressed using base32768.
• Emglken currently stores data as latin-1 texts. These should be ported and accessed as Uint8Arrays, which means bumping the version of Emglken.
• We could make it remove autosaves to free up space. It currently doesn't store dates for when autosaves were created/used, so we'd probably just have to delete all of them. Or we could add a date for each.
• The various users of Glk need to check whether files were written successfully. I've reported this as a bug to the I7 tracker, but the other users all need to as well.

[curiousdannii]

Here are all the calls to file_write and file_read:

• file_write

  • emglkenfs.js
    • close calls with raw data converted to an ASCII string
    • open calls with raw ""
  • glkapi.js
    • gli_stream_flush_file calls with Array<number>
    • glk_stream_close calls with Array<number>
    • glk_stream_open_file/glk_stream_open_file_uni calls with raw ""
    • restore_allstate calls with raw ""

• file_read

  • emglkenfs.js
    • open calls with raw: true
  • glkapi.js
    • glk_stream_open_file/glk_stream_open_file_uni calls with raw: false
    • restore_allstate calls with raw: false

This is a good sign: aside from emglkenfs.js, raw mode is only ever used to set an empty file. So I'm thinking I'll consider uses of file_write that write a raw text other than "" to be deprecated.