Rethink about the storage model

[kinu]

Old APIs are disappearing and new APIs/usages are emerging, it looks we're getting to a situation where we might want to redefine the API with a set of simpler, low-level storage primitives.

I imagined we could start over with a minimum API which has something like requestStorageDurability, queryUsage and storage pressure events, but there seem to be also several discussion points that are discussed at length in https://github.com/slightlyoff/StorageDurability#2.

A few common popular requests are:

• A simple API to 'promote' an app's entire storage to persistent
  • Storage Durability and Storage API proposal both trying to address this
  • Ability to request/set per-origin persistent quota (i.e. requestPersistentQuota) does not seem really necessary
  • Whether we want to have partial storage durability (in terms of space or time) is still in question, but it could be too complex
• Ability for websites to indicate that some data are important but some are not (so that not entire origin data is always deleted during eviction for non-durable data)
  • Currently two options are being discussed: storage pressure events and named storage areas possibly with priorities

Filing this issue on Quota API too for tracking the discussion and showing interest on collaboration-- I'm willing to integrate these new changes or collaborating on a new API whichever smoother.

/cc @annevk @sicking @davidsgrogan @jakearchibald

[annevk]

I revised https://wiki.whatwg.org/wiki/Storage significantly to be a bit more clear about multiple storage boxes per origin and how that could work.

The main goals I have are to define the underpinnings of all storage APIs, enable persistent storage in a way that is competitive with native, and provide more storage options for sites so that clearing is no longer an all-or-nothing for them.

[sicking]

The big thing that have changed in my thinking since we last worked on the quota API is that I think that we should make it so that when a website requests persistent storage, and the user grants that, that it should change the storage policy for all data currently stored by that website.

So I don't think that the page should have to first request access to persistent storage, and then use some new API or new API syntax to write to that storage.

Instead, getting access to persistent storage should make all data written using the default syntax in IDB/localStorage/WebSQL/CacheAPI/Appcache suddenly become persistent.

This seems like the simplest model for pages.

I do think that we can additionally we can add new storage API syntax to allow pages to explicitly declare that certain data should be stored in a temporary storage area. Which it can do before or after having gotten access to persistent storage.

[kinu]

@annevk
I see, the proposal explains why it wants to be called ‘Storage’ rather than quota or something. Boxes stuff is interesting, allowing developers to explicitly declare some storage portion is discardable would be probably useful. I assume we'd want to spend a bit more on these ideas as it adds and implies a lot. (And as mentioned elsewhere we're still seeing possibility for pressure events thing)

[kinu]

@sicking

The big thing that have changed in my thinking since we last worked on the quota API is that I think that we should make it so that when a website requests persistent storage, and the user grants that, that it should change the storage policy for all data currently stored by that website.

Yeah I basically agree with this, or I've started to feel the same. Setting a per-origin quota limit was ok but asking apps to set the amount wasn't really practical, and requiring changes for storage APIs to use persistent storage was blocking us from going further. So origin's default storage mode could be either temporary (best-effort) or persistent, and to use the storage in the default mode no special API or syntax is required-- sounds simple.

A few things we could do in this spec for that direction would be:

• Deprecate requestPersistentQuota(bytes)
• Instead add requestStorageDurability() or something similar. (Or at least make quota API work with the new API if it wants to be in a separate spec)
• Also maybe add a storage type "default" and make queryInfo("default") (or just queryInfo() without parameter) return the origin's current storage situation with its default storage mode. (This way it would still give modest backward compatibility for old API users, i.e. FileSystem API users)

[annevk]

@kinu yeah, it's a strawman for where some of us at Mozilla think we should go. Note that if you think boxes is too much, we could start with v1. Expose requestPersistent(), persistentPermission(), getEstimate(), and getPersistentEstimate().

[davidsgrogan]

Note that if you think boxes is too much, we could start with v1. Expose requestPersistent(), persistentPermission(), getEstimate(), and getPersistentEstimate()

That's roughly what we (chrome) are planning on doing. Though we'd only have getEstimate() because v1 will consist of marking the entire origin as "durable" or not.

Speaking of, to start bikeshedding, I propose we switch to using "durable" for the new capabilities we're talking about - "persistent" was already used in the existing quota api to mean something a little different. I don't want to overload the term.

Back to real stuff, the Storage wiki page proposes exposing a requestPersistent() method to code running in a worker. We plan on showing a permission prompt at least some of the time, which might not be possible from a worker. Is FF going to have a similar problem?

I signed up for the extensible web summit in San Francisco this approaching monday (April 20) mostly to talk about durable storage. Anyone else going and want to talk in person? @wanderview?

[wanderview]

We plan on showing a permission prompt at least some of the time, which might not be possible from a worker. Is FF going to have a similar problem?

@benturner, do we currently prompt to expand quota for IDB in workers?

I signed up for the extensible web summit in San Francisco this approaching monday (April 20) mostly to talk about durable storage. Anyone else going and want to talk in person? @wanderview?

I wish I was going, but cannot. I'd be happy to dial in if you want to do a video call. No need if I'm the only remote person, though.

I think @annevk is going to be there, though.

[annevk]

@davidsgrogan just having getEstimate() would not account for future expansion and the distinction between site and origin. I also don't really like switching to durable just for the sake of an API that was not adopted cross-browser. Persistent is a much clearer term. We should just make sure we agree on the details. I'll be at EWS to discuss.

[annevk]

(I agree that the requestPersistent() API should be restricted to Window, fixed.)

[benturner]

We never prompt for expanding quota now, on the main thread or on workers. There's a prompt if you request persistent storage (though we haven't yet turned that on for workers, that will start in the next few weeks), but thereafter there are no prompts ever. We allow you to keep using more space until you run out of space. Then you just get QuotaErr thrown at you.

[davidsgrogan]

There's a prompt if you request persistent storage (though we haven't yet turned that on for workers

Can you explain what persistence means? FF doesn't implement "persistent" storage as discussed by the quota api, right?

[wanderview]

Can you explain what persistence means? FF doesn't implement "persistent" storage as discussed by the quota api, right?

I believe this means that the browser will not evict the origin under storage pressure. Currently the only way to get this in gecko is with an installed firefoxos app.