Exception running callback for LaterGauge(synapse_federation_transaction_queue_pending_destinations)

[richvdh]

2022-02-24 16:35:18,136 - synapse.metrics - 90 - ERROR - sentinel - Exception running callback for LaterGauge(synapse_federation_transaction_queue_pending_destinations)
Capture point (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/socketserver.py", line 650, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python3.7/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.7/socketserver.py", line 720, in __init__
    self.handle()
  File "/usr/lib/python3.7/http/server.py", line 426, in handle
    self.handle_one_request()
  File "/usr/lib/python3.7/http/server.py", line 414, in handle_one_request
    method()
  File "/home/synapse/src/synapse/metrics/_exposition.py", line 198, in do_GET
    output = generate_latest(registry, emit_help=emit_help)
  File "/home/synapse/src/synapse/metrics/_exposition.py", line 92, in generate_latest
    for metric in registry.collect():
  File "/home/synapse/src/synapse/metrics/__init__.py", line 66, in collect
    for metric in REGISTRY.collect():
  File "/home/synapse/env/lib/python3.7/site-packages/prometheus_client/registry.py", line 83, in collect
    for metric in collector.collect():
Traceback (most recent call last):
  File "/home/synapse/src/synapse/metrics/__init__.py", line 88, in collect
    calls = self.caller()
  File "/home/synapse/src/synapse/federation/sender/__init__.py", line 252, in <lambda>
    for d in self._per_destination_queues.values()
  File "/home/synapse/src/synapse/federation/sender/__init__.py", line 251, in <genexpr>
    1
RuntimeError: dictionary changed size during iteration