Make all calls in SchedulerStateManagerAdaptor synchronized

[maosongfu]

1. Make all calls in SchedulerStateManagerAdaptor synchronized, i.e. wait for the result unless timeout
2. Correct corresponding invocation of methods in SchedulerStateManagerAdaptor
3. Fix some misuse of IStateManager or SchedulerStateManagerAdaptor

[kramasamy]

👍

[saileshmittal]

This includes changes in the SPI. Won't this break existing scheduler code?

If yes, we probably should keep the existing SPI, and add new interfaces for serial calls. OR we break the SPI this time and communicate to others.

In either case, may we should keep the async version around as well.

[maosongfu]

@saileshmittal
Hi,

You are right, changing SPI is possible to break existing scheduler code.
But consider these:

1. From the start of design, we decided to make all calls in SchedulerStateManagerAdaptor synchronized and there might be some miscommunications or other unknown reasons, it is in async style, which is hard to reason and use in Scheduler implementation case.
2. I checked current scheduler implementations in the repo and fixed the breaking parts. Async interfaces may not be a good idea in SchedulerStateManagerAdaptor: IStateManager is fully async, and we could expose IStateManager from SchedulerStateManagerAdaptor if in future async methods are really needed.
   Consider we are still in an early stage, it is ok to break things rather than keep async version for backward compatibility concern (which may not even exist at current stage).

[saileshmittal]

ok 👍