Release v16.22.1
Summary
A read already in flight could put an erased compliance key back into the cache.
Fixed
- The compliance key cache no longer restores a key that was deleted or evicted while a read of it was in flight. The cache releases its lock to call the backing store and takes it again to record the answer; an erasure landing in that window was overwritten by the older answer, so a shredded key came back and was served again
- An erasure no longer leaves the key cacheable for the length of the durable delete. The cache was cleared before that call, so a read starting afterwards could still find the key in the store and cache it, and cached keys have no time-to-live. Erasing one revision no longer evicts the others
- A partially failed erasure still clears the cache rather than keeping a key that may already be destroyed, and still evicts from every silo. A composite key store attempts every store and then reports the failure, so the key can be gone and the call still throw
- Saving a key no longer loses to a read that began before it. After a rotation the cache could serve the superseded key, so values encrypted under the new revision failed to decrypt until something evicted the entry