Skip to content

Don't log import latency every time the task times out #1871

@michaelkedar

Description

@michaelkedar

The workers log latency regardless of whether the task finished or not:

done = done_event.wait(timeout=MAX_LEASE_DURATION)
logging.info('Returned from task thread')
self._log_task_latency(message)
if not done:
self.handle_timeout(subscriber, subscription, ack_id, message)
logging.warning('Timed out processing task')

This messes with our import latency SLO monitoring, since jobs repeatedly get processed & timed out until it's processed correctly or dead-lettered.

We should also add a subscription to the dead letter topic to monitor failed imports

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingneeds-gcp-infra-accessTo be able to adequately work on this issue will require access to GCP infrastructure

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions