Track retry counts

[eric-czech]

Is there a way to know how many tasks fail and are are retried when using .compute(retries=N)?

This would be a useful addition in performance_report.

[jrbourbeau]

Sorry for the delayed reply @eric-czech. The scheduler indeed keeps track of how many time each task has been retried. This information is stored in TaskState.suspicious attribute for the corresponding task:

distributed/distributed/scheduler.py

Lines 1046 to 1059 in 3e2ea59

	.. attribute:: suspicious: int
	The number of times this task has been involved in a worker death.
	Some tasks may cause workers to die (such as calling ``os._exit(0)``).
	When a worker dies, all of the tasks on that worker are reassigned
	to others. This combination of behaviors can cause a bad task to
	catastrophically destroy all workers on the cluster, one after
	another. Whenever a worker dies, we mark each task currently
	processing on that worker (as recorded by
	:attr:`WorkerState.processing`) as suspicious.
	If a task is involved in three deaths (or some other fixed constant)
	then we mark the task as ``erred``.

[eric-czech]

Thanks @jrbourbeau. I think this would be useful to surface as summary information somewhere (like Hadoop), particularly when retires are necessary to avoid job failures due to frequent underlying cloud IO libraries for a job (cf. pydata/xarray#4704). It would be a useful counter to keep an eye on over time in a production environment.

[jrbourbeau]

We expose this information today through distributeds integration with prometheus (https://docs.dask.org/en/latest/setup/prometheus.html)

distributed/distributed/http/scheduler/prometheus/__init__.py

Lines 47 to 55 in 6f95c63

	suspicious_tasks = CounterMetricFamily(
	"dask_scheduler_tasks_suspicious",
	"Total number of times a task has been marked suspicious",
	labels=["task_prefix_name"],
	)
	for tp in self.server.task_prefixes.values():
	suspicious_tasks.add_metric([tp.name], tp.suspicious)
	yield suspicious_tasks

I'm not a prometheus expert but know it's a commonly used monitoring tool for production environments