Compare generations when available

[KnVerey]

Problem

I just observed a instant false positive success on a web deployment. Relevant lines:

[INFO][2018-08-07 19:27:25 +0000]	Successfully deployed in 1.9s: [...] Deployment/web
[INFO][2018-08-07 19:27:25 +0000]	Deployment/web                                    3 replicas, 3 updatedReplicas, 3 availableReplicas

The rollout definitely did not complete in less than 2s.

Checking the code made me notice that we aren't comparing metadata.generation to status.observedGeneration before using the status to determine success. This means we might be making that determination with state data, which could explain the above behaviour.

Solution

Compare metadata.Generation to status.observedGeneration before _succeeded? and _failed? determinations whenever possible. I used the API documentation to figure out which controllers actually populate the observed generation.

Since this is a controller race, I can't think of a way to make it happen for integration tests. I have added regression tests at the unit test level.

@Shopify/cloudx

[karanthukral]

should this not be !=? Or maybe I just misunderstood it

[dturn]

As I understand it, and @KnVerey will hopefully correct me if I'm wrong, current_generation gets bumped when we apply the config. observed_generation gets set by the controller reporting the status. When they don't match it means that we're looking at a status for the previous config not current. E.g. we don't want to say the deployment has failed or succeeded before the status reflects the current config's state.

[karanthukral]

oh! Thanks for the clarification

[karanthukral]

Out of curiosity, do you have an example of a controller that doesn't populate it?

[KnVerey]

Since all objects have metadata generations, theoretically any controller could be setting observedGeneration, though this is only meaningful if the controller is writing a status. An example of a resource that has a status but no observedGeneration would be Job or CronJob. See also this k8s core issue. As a sidenote, in 1.11 (I think it made beta there?), CRs will have a reliable metadata.generation we can use to do this properly in our own controllers. Our generic CR success feature should make use of it when available.

[karanthukral]

Just the one question for clarification but the rest looks good to me

[dturn]

Out of curiosity: is there a reasonable way to return early from sync when observed_generation != current_generation. Doesn't seem like we get anything useful when that happens.

[KnVerey]

Is there a reasonable way to return early from sync when observed_generation != current_generation. Doesn't seem like we get anything useful when that happens.

What do you have in mind that we could skip? The basic sync is just @instance_data = mediator.get_instance(type, name), and we don't know whether the generations match until we get that request. Maybe this is suggesting an additional state for our tentative state machine implementation though (names not to be taken seriously): undeployed -> (outdated) -> pending -> succeeded/failed/timed out

[dturn]

Having to put this check in every class just makes me think we're missing an abstraction layer. The sync method was an attempt but, I think you're right that the state machine is probably the best way forward.

[KnVerey]

Yeah, it definitely irked me to write that line in so many places. I could technically add that pseudo-state in our current model like this, but we might get unexpected results unless we also make everywhere that partitions resources into groups aware of the possibility:

WDYT?

[dturn]

I think its better as is than adding a new state to unwind later.

[KnVerey]

👍 I'll merge as-is then. I added a comment to your WIP so we don't forget about this extra state.