App designated archive for intenal storage.

[Gozala]

Overview

I have being writing multiple exploratory apps and have noticed recurring pattern:
App creates content which needs to be stored somewhere. And it has following options, which unfortunately have different downsides (that I'll describe inline below)

Ask user to select an archive to write into.

• Often implies that data will be shared across multiple apps, which is for some kind of data is undesired.
• Implies that there might be multiple archives. Which makes UX suffer and complicates the application architecture.
• Good UX suggest to not ask questions prematurely, which forces you to either risk loosing data user created before you ask for permission to save, or ask permission sooner than it would make sense.
• Requires app to preserve selected archives somewhere like indexDB or localStorage. But it also means hat app should be able to gracefully handle case that get's cleared and have even worse UX when that happens.
• Raises question how frequently should content be saved into chosen archive. Each write has overhead in creating version communicating update etc so in case user content is a text document it's becomes non trivial choice.

Create a designate archive for writing into.

Has all the same issues as above, except you can have more subtle permission request that does not requires user to take immediate action.

Use IndexedDB instead

This addresses most UX concerns but has engineering burden

• IndexedDB as far slower (at least on my machine) sometimes ridiculously so.
• IndexedDB API is very different so when it's time to actually make user created content sharable moving it from IndexedDB into DatArchive is fairly inconvenient.
• There is virtually no pass data from IndexdDB to the other app. In theory combination of some RPC mechanism and ArrayBuffers might allow that but as things stand now it's not.

Disclaimer: Last claim isn't exactly true I end up hacking around using combination of hidden iframe, new tab, BroadcastChannel and Blob URLs. But I don't think it's fare to say it's impractical.

Proposal

I propose to assign each app designated DatArchive instance with a limited space. Maybe something like DatArchive.loadStorage(location):Promise<DatArchive>. In a way it is similar to self-mutating site pattern except:

1. Loaded archive will be archive designated to write data into rather than archive of the site doing a call.
2. There will be 1x1 mapping between calling archive and designated storage one (so that app won't have to remember).
3. Loaded archive will have some designated storage capacity (I think it would be reasonable to let app request increase in quota).

This would address use case describe nicely as apps will be able to:

1. Store draft content in the own archive.
2. Store as frequently as it makes sense.
3. Would not need to prompt user ahead of time.
4. Would allow app to pass data to another app just by passing a URL. (For example navigate to the app that does publishing and pass query argument for data it wishes to be published).

Considerations

1. If app is able to write data that in turns is an app that gets own writing space it becomes possible to overcome space limit. I propose to treat archive returned by DatArchive.loadStorage(location):Promise<DatArchive> differently in that same call from with-in that archive would just return same archive. That would prevent hack to avoid space limit, but at the same time would not limit space archive in any other way.

2. Should space archives be replicated or stay local ? There is a benefit to allowing replication as it would allow simple backup and with multi-writer might allow sync across devices as well. That being said it might make sense to prompt user first time other peer will attempt to fetch data from it as it would mitigate accidental data leak while isn't going to be too much burden for backup / sync.

3. Should space archive be versioned / should quota take versioning into account ? One one side you'd want to write there often and not care about keeping the history but on the flip-side some apps might benefit from having revisions this could provide, but versioning without taking that into quota considerations could be abused too. I'm inclined to suggest that dropping versions and have a way to manually pin / unpin specific versions might be the best. If that is technically not viable, I'd suggest no versions as at the end of the day app could create own versioning mechanism if that is desired.

4. Is storage mapped to a domain name or an archive key ?
   I'm not sure what would be a good choice as on one hand I can see wanting domain name in case I map a different dat to it in the future. On the other hand I also see not wanting to makes users loose data in case I end up changing domain name.

   Ideally store could survive both domain name changes and mapped archive changes but I do not know how it can be pulled off. Another option would be to detect those migrations somehow and prompt user when store is being requested to provide a choice.

5. Should site forks also fork corresponding store ?

   I think one forks site to make changes to it so it would make sense to also also fork store along or maybe share the same store ? Furthermore not sure what the user of a fork should get. My instinct is it might be best to just create separate stores and have some mechanism for user to to migrate data from one to other.

[pfrazee]

Good proposal! Let's make this one happen. A couple thoughts:

• I suggest we do something like navigator.requestSiteStorage() or some variation of that. The key point is, we should put the getter navigator instead of DatArchive because DatArchive gets used outside of Beaker, and it's easier if we keep environment-specific functions off of it. (Related: DatArchive.selectArchive() was a mistake, and should probably be replaced with navigator.showArchiveDialog() or similar
• For consideration 1, can you clarify your suggested fix a bit? Are you saying the "storage dat" would not have its own storage dat?
• For consideration 2, I'm thinking the same things about the private dat, which is just going to be offline for now. We can treat these storage dats the same way (offline for now). In fact I think it'd make sense for the storage dats to be mounted in the private dat under something like /appdata, eg /appdata/fritter.hashbase.io for fritter.
• For consideration 3, "dropping versions and having a way to manually pin/unpin specific versions" is exactly where I want versioning for all dats to go. We're going to introduce a way to pin versions (possibly called "tags" since it's similar to git) and we'll use that UX to determine which historic files should be retained. In the short-term, I'm +1 to just not keep history in storage dats.

[Gozala]

For consideration 1, can you clarify your suggested fix a bit? Are you saying the "storage dat" would not have its own storage dat?

Let's assign names to things so it's easier to capture what I mean:

Lets say we have a site with dat://site-origin/. Call navigator.requestSiteStorage() from that origin will return Promise<DatArchive> where archives .url is let's say dat://site-origin-storage/. Any call navigator.requestSiteStorage() from the dat://site-origin-storage/ will return Promise<DatArchive> where archive's .url will be dat://site-origin-storage/.

Or in other words storage dat's storage is itself.

[Gozala]

• For consideration 2, I'm thinking the same things about the private dat, which is just going to be offline for now.

Do you have any thoughts in regards to backup / sync for private dat's ?

We can treat these storage dats the same way (offline for now). In fact I think it'd make sense for the storage dats to be mounted in the private dat under something like /appdata, eg /appdata/fritter.hashbase.io for fritter.

In that case would navigator.requestSiteStorage() still return a Promise<DatArchive>, or would it be something else ?

One counter argument I'd have is I would like app A to be able to pass URL to some resource in own storage to an app B. If I'm not misunderstanding private dats spec that would require app B to request read access to that resource which would not be ideal. I app B can read without user prompts that you can discard this argument.

I do like name mapping! It would be great to have a intuitive way to figure out what storage corresponds to what app.

[Gozala]

• For consideration 3, "dropping versions and having a way to manually pin/unpin specific versions" is exactly where I want versioning for all dats to go. We're going to introduce a way to pin versions (possibly called "tags" since it's similar to git) and we'll use that UX to determine which historic files should be retained. In the short-term, I'm +1 to just not keep history in storage dats.

👍

Would it be still possible to get notifications for dats that are synced with local dir ?

[Gozala]

Posted few more considerations as @pfrazee's feedback got me thinking about it.

[pfrazee]

I'm tied up with some other design work so I'll get back to this thread in a bit.

Would it be still possible to get notifications for dats that are synced with local dir ?

Should be