We maintain structured logs about various fine grained events on each task in Scheduler.transition_log, Worker.log, Stealing.log, etc.. these are invaluable both in tracking down issues with methods like Scheduler.transition_story(*keys) and Worker.story(*keys), and in producing visuals.
As people use clusters more dynamically we may find value in recording events around workers, clients, consistency checks, etc.. Ideally we find a consistent way to do this with many kinds of events that makes creating plots or query functions easy in the future.
So some questions:
- What are good events to track?
- What are good consistency checks to track?
- How do we organize these events for easy querying and visualization?
- What visuals and queries would be of pragmatic use?
We maintain structured logs about various fine grained events on each task in
Scheduler.transition_log,Worker.log,Stealing.log, etc.. these are invaluable both in tracking down issues with methods likeScheduler.transition_story(*keys)andWorker.story(*keys), and in producing visuals.As people use clusters more dynamically we may find value in recording events around workers, clients, consistency checks, etc.. Ideally we find a consistent way to do this with many kinds of events that makes creating plots or query functions easy in the future.
So some questions: