Set and respect storage quota per user collection.

[tublitzed]

We need to set a quota server side (2GB) and return a error to clients that will cause them to fail to sync (and log details) in this instance. This is to support spanner limitations.

Quota should be configurable server side to avoid need to do a train release to make changes there. Ideally it's some config var that will also prevent us from needing to do a server side release as well.

We also need to add tracking for this on the server side so we can see how many users this is affecting.

Related docs:

• Sync Quota Detection
• Sync users > 2GB mitigation
• Sync Tombstone Learnings

[jrconlin]

See internal doc

(Note: The internal doc is mostly a personal brain dump around tombstones and how they work, since I'm not part of the client dev team. In summary: Tombstones are short records to indicate that something was removed. They generally contain just the date and the collection type, and sometimes an ID. Each collection has it's own tombstone and how it deals with them. There's some discussion about how long a given tombstone should persist. They're used by older clients to note that something was deleted and that it should be deleted locally as well.)

[rfk]

FWIW the server protocol docs already reserve 400 Bad Request with error code 14 for "user has exeeded their quota", and there is even a special failure status for this in the desktop client code. I don't think this special failure status provides an satisfying UX or anything like that, but if nothing else it should help the failure show up in an intelligible way in the sync logs.

[jrconlin]

As noted in the bug: We will definitely use these codes, since we're not doing so currently. I'd like to push for better UX as well and suggest the client be mindful if storage reaches some threshold less than the absolute limit. I suppose the server should probably broadcast what that absolute limit may be since it may vary per user or service.

[tublitzed]

@jrconlin to confirm that the client does indeed behave as expected re: the existing "exceeded quota" error code, would it be possible to get us to a point where we could confirm that against staging next week via some forced response mechanism?

My thinking is that if for some reason the client (let's assume release versions of all) does not behave as expected, we want to know that ASAP given the desktop release train schedule. Even if we hack together something that forces this response on a normal sync for say, a specific account or something like that, I think that might be enough to confirm this. Probably also if we could set it to some low size too (vs 2GB) to make testing easier.

Then we do the actual implementation work in future weeks/etc. What do you think?

[jrconlin]

I'll see what I can come up with. It might be possible to have the server return something based on a passed header.

[tublitzed]

Thanks JR. I'll track testing results here whenever you're ready.

[jrconlin]

Per Rachel's requirements:

No random versions. Latest of all. But when you get back, what about:

    Hardcode a few test FxA account ids into the sync response. (ie, my accounts, or QA/etc)
    When those tests ids come through, hardcode a forced response of what we want.
    Also send this error to sentry.
    Deploy to staging only (then we revert changes doesn’t need to go to prod)
    I test with those fxa accounts against staging.
    We confirm we see errors in Sentry.

[tublitzed]

@jrconlin to be clear, that's one possible approach here, not requirements :)

What we need to do here is:

1. Mimic exact client behaviour that we know will trigger this. Ie, try to sync > some threshold, see a returned quota error (or that it's triggered in some way). I'd suggest we go with less than 2GB of data to make testing easier, so if you want to hardcode it to something small, that'd be great.
2. For desktop - looking at the sync logs should be enough to verify. Latest version of desktop is fine.
3. For mobile (iOS, Fenix, Fennec) - since we don't have sync logs easily accessible there without running locally I would suggest we send a server side error to sentry so we can confirm this that way. If you have an easier way to confirm, that's fine too.

[tublitzed]

Re-opening as what was implemented was the ability to validate client behaviour, not the actual feature. @jrconlin did you want to track that as a separate ticket, or are we good to just keep this one around for it?

[jrconlin]

Let's keep this ticket open for now.

[jrconlin]

Related issues:
#789
#790
#791

[jrconlin]

closed as part of #806