The workers log latency regardless of whether the task finished or not:
|
done = done_event.wait(timeout=MAX_LEASE_DURATION) |
|
logging.info('Returned from task thread') |
|
self._log_task_latency(message) |
|
if not done: |
|
self.handle_timeout(subscriber, subscription, ack_id, message) |
|
logging.warning('Timed out processing task') |
This messes with our import latency SLO monitoring, since jobs repeatedly get processed & timed out until it's processed correctly or dead-lettered.
We should also add a subscription to the dead letter topic to monitor failed imports
The workers log latency regardless of whether the task finished or not:
osv.dev/docker/worker/worker.py
Lines 623 to 628 in 01173bf
This messes with our import latency SLO monitoring, since jobs repeatedly get processed & timed out until it's processed correctly or dead-lettered.
We should also add a subscription to the dead letter topic to monitor failed imports