Worker should stop when operation is canceled

[victor-carvalho]

When an operation is canceled the worker should know and stop the operation and the api should issue a rollback. Right now this is not happening because OperationManager.WasCanceled checks a cached state and never hits redis and the worker never starts the loop to check for operation updates.

I changed to OperationManager to remove the loop and make it so WasCanceled always check on redis. This works for most use cases inside maestro. To prevent having multiple goroutines hitting redis at the same time I moved the loop to inside the worker and separated from OperationManager.

I also added tests to SegmentAndReplacePods that were missing and added a test case to the watcher for when an operation is canceled.

[coveralls]

Coverage increased (+3.6%) to 78.277% when pulling 037c6fe on fix/worker-cancel into 1e7e89b on master.

[gabrielcorado]

Since we're moving the timeout logic to SegmentAndReplacePods, why not also do all the cancellation stuff there?

I think the sub-functions could only just receive a single context, and when it cancels it just stop doing its work, they don't really need to know the reason. And SegmentAndReplacePods could return the reason why it stopped. What do you think?

[victor-carvalho]

Since we're moving the timeout logic to SegmentAndReplacePods, why not also do all the cancellation stuff there?

I think the sub-functions could only just receive a single context, and when it cancels it just stop doing its work, they don't really need to know the reason. And SegmentAndReplacePods could return the reason why it stopped. What do you think?

I think it makes sense. Will make the change.