sentry: panic: id does not exist: IdentifierMissing(User(4))

[sentry-io]

Sentry Issue: DATABASE-BACKEND-2S8

panic: id does not exist: IdentifierMissing(User(4))
  File "panicking.rs", line 575, in rust_begin_unwind
  File "read_policy.rs", line 501, in mz_adapter::coord::Coordinator::update_read_hold
  File "timeline.rs", line 903, in mz_adapter::coord::Coordinator::advance_timelines::{closure#0}::{closure#0}
  File "timeline.rs", line 878, in mz_adapter::coord::Coordinator::advance_timelines::{closure#0}
  File "message_handler.rs", line 68, in mz_adapter::coord::Coordinator::handle_message::{closure#0}
...
(26 additional frame(s) were not displayed)

[def-]

There was a similar issue once before: #11568

But this seems to come from new RBAC code: adbf847 @jkosh44 (mz_adapter::coord::Coordinator::update_read_hold(read_policy.rs:501))

Happened on Materialize user on staging.

Could it be that a user is deleted during a transaction?

[jkosh44]

This is likely unrelated to RBAC. The User(4) here is the GlobalId of some storage collection. The User indicates that it's a user created collection, like a table or source.

This is also likely not caused by adbf847. That commit just switched all calls to unwrap() with calls to expect().

This error is fairly concerning though, the Coordinator and the controller seem to disagree about a certain collection existing. The Coordinator might have some stale state somewhere.

This does look extremely similar to this issue: #19398 and might in fact be a duplicate.

[benesch]

I think this might be a duplicate of #19743, actually.

[jkosh44]

Reassigning this to @mjibson since he has a PR that fixes this.