sentry: panic: failed to create a delayed capability, the current set does not have an element less than or equa...

[sentry-io]

Sentry Issue: DATABASE-BACKEND-2T6

panic: failed to create a delayed capability, the current set does not have an element less than or equal to 1687930325000
  File "panicking.rs", line 575, in rust_begin_unwind
  File "capability.rs", line 456, in timely::dataflow::operators::capability::CapabilitySet<T>::delayed::delayed_panic
  File "capability.rs", line 464, in timely::dataflow::operators::capability::CapabilitySet<T>::delayed::{closure#0}
  File "capability.rs", line 463, in timely::dataflow::operators::capability::CapabilitySet<T>::delayed
  File "source_reader_pipeline.rs", line 1224, in mz_storage::source::source_reader_pipeline::handle_message::<T>::{closure#0}
...
(38 additional frame(s) were not displayed)

[def-]

There was a similar issue before: #17484

[benesch]

Linking in a Slack conversation about another instance of this: https://materializeinc.slack.com/archives/C04DJT084KA/p1688300014101509

[benesch]

We haven't seen this in a while. Optimistically closing.