Explicitly track the key-rotation timestamp for the user's sync key

[rfk]

Over in https://github.com/mozilla/fxa-auth-server/issues/2674 we're going to add a new field to the metadata returned in FxA identity assertions telling us the timestamp at which the user's key material last changed. This is similar to, but not the same as, the "generation number" that we currently track.

Let's update tokenserver to track both "fxa-generation" and "fxa-keysChangedAt" for each user, so that it can accurately construct the kid for the user's sync key. This will in turn allow the new durable sync storage server to more accurately purge data that was encrypted with an older key.

Concretely, we currently communicate fxa_uid and fxa_kid fields to the storage nodes like this:

{
    'fxa_uid': user['fxa_uid'],        // e.g. "00001111222233334444555566667788"
    'fxa_kid': user['client-state'],   // e.g. "AABBCCDDEEFF11223344556677889900"
}

Given two fxa_kid values, it's not possible to tell by inspection which is the "current" key-id and which is out-of-date. When we hand out scoped keys for OAuth clients, we identify the keys by a combination of timestamp and hash, which means they sort lexicographically in a useful fashion. I think we should do the same here, so we end up sending:

{
    'fxa_uid': user['fxa_uid'],                                        // e.g. "00001111222233334444555566667788"
    'fxa_kid': user['keys-changed-at'] + "-" + user['client-state'],   // e.g. "1569299946203-AABBCCDDEEFF11223344556677889900"
}

This will allow the storage nodes to easily detect and purge data stored under old keys.

The tricky part will be doing this accurately in the presence of incomplete information, because we haven't been tracking the key-rotation timestamp for existing users. I believe we need to do the following, in order:

1. For OAuth sync clients, stop treating the key rotation timestamp as being the same as the generation number. Otherwise we risk locking OAuth clients out of sync once we start tracking the key-rotation timestamp properly. I'm working on a PR for this.
2. Get the FxA changes to track key-rotation timestamp into production on the FxA side, which will make a new keysChangedAt field available to the tokenserver.
3. Update tokenserver to start recording keysChangedAt alongside the generation number in the db, and reporting it out in the fxa_kid field as above.

[rfk]

@bbangert not sure if this is linked into your "python spanner prototype" tree anywhere, but I wanted to flag that it should be. It won't block you from pushing ahead with implementing the storage backend, but I think it'll be needed before we can put any production users in a spanner node. (Otherwise you're going to have a hard time purging a user's old data after a password reset, because you won't have the timestamps necessary to tell which data is "old").

[rfk]

I worked through the required FxA changes over in mozilla/fxa#2570; before I head out on PTO I will take a fresh look at what corresponding changes will be needed here in tokenserver. Broadly I think it's going to look like:

• Adding a keysChangedAt column in the node-assignment table, alongside generation.
• Extracting the value for keysChangedAt from either the assertion or the OAuth Key-ID timestamp, like we currently do for generation
• Fiddling with the way we handle generation for OAuth clients, because it will no longer be possible to find out a generation number for those clients.
• Ensuring that we use keyChangedAt rather than generation when reconstructing the fxa_kid field to embed in the Hawk token.

I will also work on some docs that explain more clearly why each of those things are necessary; I realized that the README etc in this repo is substantially out of date :-(

[rfk]

I've updated the issue description here to say more about the motivation and next steps involved.

[rfk]

A brief status update here, referencing the steps in the issue description:

• Step (1) is pending QA of the deployment; ref Bug 1589565.
• Step (2) has been coded up, awaiting review, and awaiting the deployment from step (1) before it can safely merge to FxA; ref Explicitly track the key-rotation timestamp for kB mozilla/fxa#490 (comment)
• Step (3) is the draft PR Track key-rotation timestamp if provided by the IdP #143, which has been reviewed but is waiting for the deploy from (1) before landing, mostly to keep things separate for QA purposes.

[rfk]

This is ready to go!