Only add reevaluate_occupancy callback once

[alorenzo175]

Scheduler.reevaluate_occupancy is currently added as a callback every time Scheduler.start() is called. Since Scheduler.start() is called every time Scheduler.restart()is called, long running schedulers that have restarted many times eventually consume most of a CPU trying to reevaluate occupancy. To add to the problem, the cpu percent check in reevaluate_occupancy always returns 0.0, at least on linux with psutil 5.4.3, since a new psutil.Processobject is made for each check (see https://psutil.readthedocs.io/en/latest/#psutil.cpu_percent).

I've moved where the reevaluate_occupancycallback is added to the IOLoop so that it is not re-added on a restart. I've also switched to using psutil.cpu_percent() instead of psutil.Process().cpu_percent().

I'm not sure if this needs testing or how to go about adding this to the test suite.

[mrocklin]

This is excellent. Thank you for tracking this down (I can't imagine it was easy) and then providing such a concise fix.

Regarding psutil.cpu_percent() I'm slightly concerned that this might be confused by other processes that are running on the same machine. Perhaps we should make a global process object in the file and use that? Though I can see arguments for either approach.

Agreed about testing. Not sure what to do here. I suggest skipping it for now.

[alorenzo175]

I think you're right about making a dedicated process object; I assumed proc.cpu_percent() only applied to the current PID. I added a Scheduler.procattribute. Not sure if the PID check is really necessary.

[mrocklin]

You observed this happening? This seems surprising ...

[alorenzo175]

I have not. Just a precaution that is likely unnecessary. I was thinking of running the scheduler on an HPC where it can be interrupted and resumed. Looks like that might break other parts of the code though.

[mrocklin]

I'm investigating the failures. They don't seem related, but they are persisting, which is odd.

[mrocklin]

Hrm, nevermind. Likely a false alarm. Merging.

Thanks @alorenzo175 !