Synchronize instrumentation with the actor

[Horneth]

Instrumentation was scheduled on a timer out of band of the actor's thread, and in some cases accessing non thread safe state inside the actor (in a read only fashion but still it can cause incoherent values: https://stackoverflow.com/questions/37690525/multiple-threads-checking-map-size-and-conccurency)
Instead use messages to self to schedule instrumentation on the actor's thread

[geoffjentry]

@Horneth out of curiosity, how did you discover this?

[Horneth]

I was looking for an explanation as to why the values in Grafana sometimes are really weird (I've seen the number of running jobs/workflows double and then go back and then double again within a few minutes without any real explanation).
So I looked at the code and this seemed to potentially explain this behavior

[geoffjentry]

@Horneth are you talking about the weird metadata stuff you were spotting or something else?

[Horneth]

@geoffjentry something else, like this: