Reducing write overhead of last_attempted_check

[shlomi-noach]

This PR reduces the overhead for last_attempted_check writes by postponing the write for 1sec. If instance if found by that time, write is cancelled.

It is OK that last_attempted_check turns to be slightly higher than last_checked even though this chronologically this can't be correct for a successful probe.

Write overhead is only reduced when things are generally OK. The optimization is gone away when probing instances consistently takes over 1sec.

[sjmudd]

Uploaded an image to show the difference from using an earlier version of orchestrator (3.0.1), moving to the latest code without this PR (3.0.8) and then the effect of applying the PR.

So the PR looks good and makes a huge difference in query count, possibly accentuated by the fact that I'm using some custom settings like:

 "BufferInstanceWrites": true,
  "InstanceWriteBufferSize": 500,
  "InstanceFlushIntervalMilliseconds:": 100,

[shlomi-noach]

@sjmudd we are not using same settings as you (we are using "BufferInstanceWrites": false,, and still we get similar reduction in amount of com_update:

[shlomi-noach]

(although we don't see a huge drop in overall query count)