Bigtable Database integrity error: Expected column: connected_at

[data-sync-user]

Yesterday we began seeing new Database error: Database integrity error: Expected column: connected_at errors reported in sentry:

https://mozilla.sentry.io/issues/4517044823/events/b866d03114874e1199c4ca2d5e0d4525/

This was previously seen and resolved by https://mozilla-hub.atlassian.net/browse/SYNC-4159, so we’re seeing a different case of this here.

It’s not clear what could produce a router row record that would produce this. Possibly a remove_channel call on such a ‘chid-only’ record could produce a record similar to a ‘chid-only’ but also including a version cell. However that seems a bit unlikely to have actually occurred.

Let’s double check what kind of row we’re dealing with by emitting the entire invalid row in the sentry event. This might require at least a partial fix for https://mozilla-hub.atlassian.net/browse/SYNC-4145.

┆Issue is synchronized with this Jira Bug

[data-sync-user]

➤ Philip Jenvey commented:

The first form of this occurring was the “chid-only” record (a router record consisting solely of “chid:” cells).

The second form seen was a router record only including a “version” and “current_timestamp” field: (likely caused by a Desktop user failing to migrate due to https://mozilla-hub.atlassian.net/browse/SYNC-4159 ( https://mozilla-hub.atlassian.net/browse/SYNC-4159|smart-link ) followed by a call to increment_storage).

Both these cases could also occur when we later enabled router column family TTLs. #655 ( https://github.com/mozilla-services/autopush-rs/pull/655|smart-link ) adds logic to identify all of these types of incomplete records for deleting them from the db.