Eliminate Phase and simplify Conditions

[bgrant0607]

Forked from #6951.

This is also discussed #1899 (comment) and elsewhere.

Users and developers apparently think of phases as states in a state machine, regardless of how much we try to dissuade them:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/api-conventions.md#spec-and-status

This interpretation conflicts with the declarative, level-based, observation-driven design, as well as limiting evolution. In particular, the phase should be derivable by observing other state and shouldn't require durable persistence for correctness.

In my experience, the danger to system extensibility is significant, and results from assumptions baked into clients.

Phases aren't themselves properties or conditions of their objects, but clients inevitably infer properties from them.

For example, a client might try to determine whether a pod is active, or whether it has reached a permanent, terminal state, such as with a switch statement, as follows:

switch pod.Status.Phase {
case api.PodPending:
    // not active yet
    ...
case api.PodRunning:
    // pod active
    ...
case api.PodSucceeded:
    fallthrough
case api.PodFailed:
    // terminated and won't re-activate
    ...
case api.PodUnknown:
    // WTF!
    ...
default:
    glog.Fatalf("unexpected phase: %s", pod.Status.Phase)
}

However, let's say someone wanted to add a new Phase where the pod would also be active (containers running/restarting, etc.), as in #6951. That would mean that every client/component that made decisions based on the phase would need to also then consider the new phase.

Enums aren't extensible. Every addition is a breaking, non-backward-compatible API change. We could create a new major API version for every such addition, but that would be very expensive and disruptive. At some point, it will just become too painful to add new phases, and then we'll be left with an unprincipled distinction between phases and non-phases. Creating all imaginable phases up front would not only create a lot of complexity prematurely, but would also almost certainly not be correct.

Conditions are more extensible since the addition of new conditions doesn't (shouldn't) invalidate decisions based on existing conditions, and are also better suited to reflect conditions/properties that may toggle back and forth and/or that may not be mutually exclusive.

Rather than distributing logic that combines multiple conditions or other properties for common inferences, such as whether the scheduler may schedule new pods to a node, we should expose those properties/conditions explicitly, for similar reasons as why we explicitly return fields containing default values.

Proposal:

1. Add LastTransitionTime, Reason, and Message to PodCondition (similar to NodeCondition)
2. Add new Condition types (e.g., PodCondition and NodeCondition), as needed. At minimum, we need to be able to identify whether the pod has reached a terminal condition or not. We could add a Terminated condition for that. The Reason could distinguish Succeeded from failed -- any reason other than Succeeded would be considered a failure. We have at least a couple dozen such failure reasons internally.
3. Add Conditions to whatever other objects currently have Phase but not Conditions
4. Eliminate *.Phase and PodStatus.Message from the v1 API

cc @smarterclayton @derekwaynecarr @thockin @timothysc @davidopp @markturansky

[dchen1107]

This is part of 1.0 though.

[bgrant0607]

Not part of 1.0, you mean.

[lavalamp]

Add LastTransitionTime, Reason, and Message to PodCondition (similar to NodeCondition)

Just thinking out loud: could we auto-populate this from events filed about the object?

[derekwaynecarr]

@lavalamp I thought from prior discussions we did not want to use events as a reliable messaging system to derive status

[lavalamp]

@derekwaynecarr I think it's a bit different if we're doing it in the master as opposed to joining events & objects externally.

[davidopp]

I've mentioned my opinion this before, but to recap...

I don't have a problem with what is being suggested here, but I think we should recognize the tradeoff. Everything I've ever read about job management, or heard from users, suggests that users are going to reason about the state of their pods (and jobs) as being part of a state machine. So, we have a choice: we can make the state machine explicit, or make the user synthesize one from the Conditions, or do something in between (e.g. provide the user with a library that synthesizes states from the Conditions, or have kubectl do it).

Brian described good arguments for pushing this onto the user. The counter-argument is that we're infrastructure providers, and the way we provide value is by deciding what abstractions will be useful for users, and more generally tackling these kinds of difficult problems instead of pushing them onto the user. If we're completely un-opionated about everything (how to reason about your job state, how to deal with workers of a batch job, etc.), then we're not providing as much value as we could.

I think the sweet spot might be the layered approach, i.e. the server works the way Brian described, and then we build a library that synthesizes states (and use that library inside kubectl to print states, as well as allowing users to use it). But I definitely feel that users will want a state machine abstraction.

[thockin]

I know I am contrarian here, but I don't think we will ever prevent users
from thinking of things in terms of state machines - it's a natural mental
model. And I don't think that's bad.

I agree that enumerated states are not extensible, but I still argue that
there exists an abstract state machine that every such object naturally
goes through that is vague enough to never be extended (or very very
rarely) but still provides value and the mental model that users are
seeking. It can be observed, not persisted, and probably simply derives
from some combination of Conditions. As such it is not strictly required
that we even call it out, but it will emerge! If we're clever we can make
the state machine similar or even identical across many objects.

Anyway, orthogonal Conditions is really what we agreed on months ago.
What's not clear to me is how someone is supposed to know what the ABSENCE
of a Condition means. Pulling from the load-balancer conversation - we now
have async external LB assignment and it was suggested that maybe a Service
should be not-ready until Endpoints is non-empty. So how do I denote that
Service.Status.Phase is not-ready? Do I put a Pending condition on it that
EndpointsController has to clear? But there are two things that both need
to be true to be not-not-ready (LB and endpoints). Or do I put a Ready
condition that EndpointsController has to set? But there are still two
things that need to be true. Or do we put 2 orthogonal Conditions? How
does a user determine from 2 orthogonal Conditions that a Service is
"ready"? Didn't we just define a "state" that users have to know about a
priori?

On Wed, May 6, 2015 at 3:20 PM, Daniel Smith notifications@github.com
wrote:

Add LastTransitionTime, Reason, and Message to PodCondition (similar to
NodeCondition)

Just thinking out loud: could we auto-populate this from events filed
about the object?

—
Reply to this email directly or view it on GitHub
#7856 (comment)
.

[lavalamp]

Since we're enumerating cons, let me add one additional reason to retain some sort of pre-cached state in status: it makes it easy for clients to set a fieldSelector and watch for their desired state.

[thockin]

Quick writeup of something Leo (another Borg person) put on the table.

Instead of something like a single field "Phase" being one-of (Pending,
Running, Succeeded, Failed), turn it 90 degrees. Make 4 fields "Pending",
"Running", "Succeeded", "Failed" each of which is of type "outlook" (his
word). An outlook is one-of (Yes, Eventually, Never).

The nugget of clever here is that mostly people want an answer to a
question - "is this pod running?" It's much easier to comprehend this
three-state answer than to encode a state-diagram into your code. It's
also easier to extend. Suppose, hypothetically, we want to add a state for
"scheduled but not yet acknowledged by kubelet". With Phase/State we have
to update everyone who knows the diagram. With this new idea, you just add
one field. Any code that was waiting for a pod to be running still works
just fine.

I suppose this could be modeled in Conditions. E.g. scheduler patches
Status to include a "scheduled" condition, the apiserver realizes this is a
significant moment and adds a "scheduled-not-acked" condition. Kubelet
eventually patches Status to include an "acked" condition and the apiserver
clears "scheduled-not-acked". The outlook approach is strictly more
informational because it indicates future possibilities - "will this pod
ever be running?" which makes life easier for users...

There's a nugget of clever here...

On Wed, May 6, 2015 at 5:34 PM, Daniel Smith notifications@github.com
wrote:

Since we're enumerating cons, let me add one additional reason to retain
some sort of pre-cached state in status: it makes it easy for clients to
set a fieldSelector and watch for their desired state.

—
Reply to this email directly or view it on GitHub
#7856 (comment)
.