Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Eliminate Phase and Conditions from the API #7856

Open
bgrant0607 opened this Issue May 6, 2015 · 42 comments

Comments

Projects
None yet
Owner

bgrant0607 commented May 6, 2015

Forked from #6951.

This is also discussed GoogleCloudPlatform#1899 (comment) and elsewhere.

Users and developers apparently think of phases as states in a state machine, regardless of how much we try to dissuade them:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/api-conventions.md#spec-and-status

This interpretation conflicts with the declarative, level-based, observation-driven design, as well as limiting evolution. In particular, the phase should be derivable by observing other state and shouldn't require durable persistence for correctness.

In my experience, the danger to system extensibility is significant, and results from assumptions baked into clients.

Phases aren't themselves properties or conditions of their objects, but clients inevitably infer properties from them.

For example, a client might try to determine whether a pod is active, or whether it has reached a permanent, terminal state, such as with a switch statement, as follows:

switch pod.Status.Phase {
case api.PodPending:
    // not active yet
    ...
case api.PodRunning:
    // pod active
    ...
case api.PodSucceeded:
    fallthrough
case api.PodFailed:
    // terminated and won't re-activate
    ...
case api.PodUnknown:
    // WTF!
    ...
default:
    glog.Fatalf("unexpected phase: %s", pod.Status.Phase)
}

However, let's say someone wanted to add a new Phase where the pod would also be active (containers running/restarting, etc.), as in #6951. That would mean that every client/component that made decisions based on the phase would need to also then consider the new phase.

Enums aren't extensible. Every addition is a breaking, non-backward-compatible API change. We could create a new major API version for every such addition, but that would be very expensive and disruptive. At some point, it will just become too painful to add new phases, and then we'll be left with an unprincipled distinction between phases and non-phases. Creating all imaginable phases up front would not only create a lot of complexity prematurely, but would also almost certainly not be correct.

Conditions are more extensible since the addition of new conditions doesn't (shouldn't) invalidate decisions based on existing conditions, and are also better suited to reflect conditions/properties that may toggle back and forth and/or that may not be mutually exclusive.

Rather than distributing logic that combines multiple conditions or other properties for common inferences, such as whether the scheduler may schedule new pods to a node, we should expose those properties/conditions explicitly, for similar reasons as why we explicitly return fields containing default values.

Proposal:

  1. Add LastTransitionTime, Reason, and Message to PodCondition (similar to NodeCondition)
  2. Add new Condition types (e.g., PodCondition and NodeCondition), as needed. At minimum, we need to be able to identify whether the pod has reached a terminal condition or not. We could add a Terminated condition for that. The Reason could distinguish Succeeded from failed -- any reason other than Succeeded would be considered a failure. We have at least a couple dozen such failure reasons internally.
  3. Add Conditions to whatever other objects currently have Phase but not Conditions
  4. Eliminate *.Phase and PodStatus.Message from the v1 API

cc @smarterclayton @derekwaynecarr @thockin @timothysc @davidopp @markturansky

This was referenced May 6, 2015

Owner

dchen1107 commented May 6, 2015

This is part of 1.0 though.

Owner

bgrant0607 commented May 6, 2015

Not part of 1.0, you mean.

Owner

lavalamp commented May 6, 2015

Add LastTransitionTime, Reason, and Message to PodCondition (similar to NodeCondition)

Just thinking out loud: could we auto-populate this from events filed about the object?

Member

derekwaynecarr commented May 6, 2015

@lavalamp I thought from prior discussions we did not want to use events as a reliable messaging system to derive status

Owner

lavalamp commented May 7, 2015

@derekwaynecarr I think it's a bit different if we're doing it in the master as opposed to joining events & objects externally.

Owner

davidopp commented May 7, 2015

I've mentioned my opinion this before, but to recap...

I don't have a problem with what is being suggested here, but I think we should recognize the tradeoff. Everything I've ever read about job management, or heard from users, suggests that users are going to reason about the state of their pods (and jobs) as being part of a state machine. So, we have a choice: we can make the state machine explicit, or make the user synthesize one from the Conditions, or do something in between (e.g. provide the user with a library that synthesizes states from the Conditions, or have kubectl do it).

Brian described good arguments for pushing this onto the user. The counter-argument is that we're infrastructure providers, and the way we provide value is by deciding what abstractions will be useful for users, and more generally tackling these kinds of difficult problems instead of pushing them onto the user. If we're completely un-opionated about everything (how to reason about your job state, how to deal with workers of a batch job, etc.), then we're not providing as much value as we could.

I think the sweet spot might be the layered approach, i.e. the server works the way Brian described, and then we build a library that synthesizes states (and use that library inside kubectl to print states, as well as allowing users to use it). But I definitely feel that users will want a state machine abstraction.

Owner

thockin commented May 7, 2015

I know I am contrarian here, but I don't think we will ever prevent users
from thinking of things in terms of state machines - it's a natural mental
model. And I don't think that's bad.

I agree that enumerated states are not extensible, but I still argue that
there exists an abstract state machine that every such object naturally
goes through that is vague enough to never be extended (or very very
rarely) but still provides value and the mental model that users are
seeking. It can be observed, not persisted, and probably simply derives
from some combination of Conditions. As such it is not strictly required
that we even call it out, but it will emerge! If we're clever we can make
the state machine similar or even identical across many objects.

Anyway, orthogonal Conditions is really what we agreed on months ago.
What's not clear to me is how someone is supposed to know what the ABSENCE
of a Condition means. Pulling from the load-balancer conversation - we now
have async external LB assignment and it was suggested that maybe a Service
should be not-ready until Endpoints is non-empty. So how do I denote that
Service.Status.Phase is not-ready? Do I put a Pending condition on it that
EndpointsController has to clear? But there are two things that both need
to be true to be not-not-ready (LB and endpoints). Or do I put a Ready
condition that EndpointsController has to set? But there are still two
things that need to be true. Or do we put 2 orthogonal Conditions? How
does a user determine from 2 orthogonal Conditions that a Service is
"ready"? Didn't we just define a "state" that users have to know about a
priori?

On Wed, May 6, 2015 at 3:20 PM, Daniel Smith notifications@github.com
wrote:

Add LastTransitionTime, Reason, and Message to PodCondition (similar to
NodeCondition)

Just thinking out loud: could we auto-populate this from events filed
about the object?


Reply to this email directly or view it on GitHub
GoogleCloudPlatform#7856 (comment)
.

Owner

lavalamp commented May 7, 2015

Since we're enumerating cons, let me add one additional reason to retain some sort of pre-cached state in status: it makes it easy for clients to set a fieldSelector and watch for their desired state.

Owner

thockin commented May 7, 2015

Quick writeup of something Leo (another Borg person) put on the table.

Instead of something like a single field "Phase" being one-of (Pending,
Running, Succeeded, Failed), turn it 90 degrees. Make 4 fields "Pending",
"Running", "Succeeded", "Failed" each of which is of type "outlook" (his
word). An outlook is one-of (Yes, Eventually, Never).

The nugget of clever here is that mostly people want an answer to a
question - "is this pod running?" It's much easier to comprehend this
three-state answer than to encode a state-diagram into your code. It's
also easier to extend. Suppose, hypothetically, we want to add a state for
"scheduled but not yet acknowledged by kubelet". With Phase/State we have
to update everyone who knows the diagram. With this new idea, you just add
one field. Any code that was waiting for a pod to be running still works
just fine.

I suppose this could be modeled in Conditions. E.g. scheduler patches
Status to include a "scheduled" condition, the apiserver realizes this is a
significant moment and adds a "scheduled-not-acked" condition. Kubelet
eventually patches Status to include an "acked" condition and the apiserver
clears "scheduled-not-acked". The outlook approach is strictly more
informational because it indicates future possibilities - "will this pod
ever be running?" which makes life easier for users...

There's a nugget of clever here...

On Wed, May 6, 2015 at 5:34 PM, Daniel Smith notifications@github.com
wrote:

Since we're enumerating cons, let me add one additional reason to retain
some sort of pre-cached state in status: it makes it easy for clients to
set a fieldSelector and watch for their desired state.


Reply to this email directly or view it on GitHub
GoogleCloudPlatform#7856 (comment)
.

Owner

bgrant0607 commented May 7, 2015

A "state" abstraction pushes the decision logic onto the user, as shown by my switch statement.

Owner

bgrant0607 commented May 7, 2015

@thockin Yes, the 4 fields is similar to my conditions proposal, just more ad hoc and with less information.

Owner

bgrant0607 commented May 7, 2015

I proposed more possible Conditions, analogous to Leo's proposal, here: GoogleCloudPlatform#6951 (comment)

Owner

bgrant0607 commented May 7, 2015

@lavalamp We need to extend field selectors to be able to select on conditions. That's a subset of #1362.

Owner

thockin commented May 7, 2015

I meant that these would work WITH conditions not instead of, in which case
they offer more information. Conditions could be added by any entity (are
they purely additive or can one entity set and another clear?) and would
carry richer details. The outlooks would be curated and small in number,
for programmatic consumption, derived (primarily?) from Conditions.

On Wed, May 6, 2015 at 8:09 PM, Brian Grant notifications@github.com
wrote:

@thockin https://github.com/thockin Yes, the 4 fields is similar to my
conditions proposal, just more ad hoc and with less information.


Reply to this email directly or view it on GitHub
GoogleCloudPlatform#7856 (comment)
.

Owner

bgrant0607 commented May 7, 2015

Conditions aren't annotations. They can't be added by just any entity. They should be curated.

Owner

bgrant0607 commented May 7, 2015

And conditions are for programmatic consumption. For instance, the Ready condition is used to remove pods from Endpoints.

Owner

thockin commented May 7, 2015

Have you worked out the mapping between Unkown, Pending, Running, Failed,
Success and conditions?

I still don't feel like I understand how to handle the case of a Service
being Pending while LB and Endpoints setup are happening.

On Wed, May 6, 2015 at 8:54 PM, Brian Grant notifications@github.com
wrote:

And conditions are for programmatic consumption. For instance, the Ready
condition is used to remove pods from Endpoints.


Reply to this email directly or view it on GitHub
GoogleCloudPlatform#7856 (comment)
.

Owner

bgrant0607 commented May 7, 2015

Running would be replaced by the Active condition (True/False): GoogleCloudPlatform#7868 (comment).

Above I proposed a Terminated condition. The Reason associated with that condition would indicate Succeeded vs. Failed, as described above.

Pending could be derived from not Active and not Terminated for older API versions. I'm not sure we need a more explicit condition for it, since most scenarios check for Active or Terminated (or their complements), though I suggested Initialized (if we explicitly registered initializers) and Instantiated (essentially !Pending) here: GoogleCloudPlatform#6951 (comment)

I'm also not sure we need Unknown. That would just be the desired condition (whichever that is) is not present. We shouldn't ever be returning Unknown currently, I don't think.

I'll look at the LB case again.

Owner

bgrant0607 commented May 7, 2015

Why can't the service just have a Ready condition? Because no controller is responsible for updating its status currently? Pods have multiple preconditions from multiple components: they must be scheduled, and their images must be pulled.

Owner

lavalamp commented May 7, 2015

Can we get a "deleting" (or deleted) condition? Can we get it for all objects?

Owner

bgrant0607 commented May 7, 2015

Deleting would be analogous to Vacating. I'm not opposed to a condition for that, though as discussed on #6951 it could be inferred uniformly for all objects from metadata.

Whether we want to make at least certain conditions uniform for all objects is another issue. As discussed in GoogleCloudPlatform#1899 (comment), #6487, and elsewhere, orchestrators on top of Kubernetes want certain uniform conditions, such as fully initialized (so the resource can be read), fully active (so creation can be considered successful and/or dependent entities can be created), and terminated (so entities they depend on can be deleted and/or entities dependent on them can be GCed).

Owner

bgrant0607 commented May 7, 2015

An example where the current PodPhase Running was found to be confusing: #7868.

An example where "Pending" was confusing, or at least insufficiently informative on its own: openshift/origin#2048.

Owner

thockin commented May 8, 2015

Running would be replaced by the Active condition (True/False)

I assume that "Active" is amalgamated from some number of other Conditions (scheduled, installed, ...)? As a consumer of this API, some of the things I will want to do are "wait until pod is running" or "wait until pod is terminated". If we publish a small set of very core conditions, these questions could be answered by blocking until the conditions exist. But this is sort of a trap - blocking until a pod is Active is wrong - it might never become active. I need to block until it is Active or Terminated. This is a place where outlooks are a better API. I can tell just by looking at the outlook field for Active whether I should keep waiting.

I've come around to getting rid of Phase, but I want to comprehend the details of the replacement.

Member

timothysc commented May 11, 2015

But I definitely feel that users will want a state machine abstraction.

+1

Owner

bgrant0607 commented Aug 12, 2015

To answer some misc. points above, since I'm writing this into API conventions:

The absence of a condition should be interpreted the same as Unknown for that condition.

Conditions will not be mutually exclusive, in general. That would defeat the purpose, since they would be isomorphic to an enum then.

@bgrant0607 bgrant0607 changed the title from Eliminate Phase from the API to Eliminate Phase and Conditions from the API Aug 17, 2017

Owner

bgrant0607 commented Aug 17, 2017

Updated from kubernetes/community#606

Conditions were created as a way to wean contributors off of the idea of state machines with mutually exclusive states (called "phase").

Each condition should represent a non-orthogonal "state" of a resource -- is the resource in that state or not (or unknown).

Arbitrarily typed/structured status properties specific to a particular resource should just be status fields.

Status, represented as conditions or otherwise, should be complementary to events pertaining to the resource. Events provide additional information for debugging and tracing, akin to log messages, but aren't intended to be used for event-driven automation, since they don't undergo API review, don't have deprecation guarantees, and don't have guaranteed delivery. We need to ensure that watchable status changes encompass all actionable changes. We likely need to add some missing content to status fields about things that did not happen, such as failing to create resources due to being out of quota. One could then write adapters to convert watch "events" to monitoring information (e.g., kube-state-metrics), messages/pubsub, webhooks, etc.

Unfortunately, Conditions have not worked out as originally intended:

  • The question of what should be an ordinary status field vs. a condition has come up a number of times.
  • Conditions are less discoverable and more cumbersome to use than fields.
  • The tri-state nature of conditions (i.e., they may not be present) has been found to be confusing, though this can occur for status fields, as well, across releases, and due to asynchrony, when populated only by a controller.
  • Conditions have proven to be expensive to update. In particular, the LastHeartbeatTime / LastProbeTime / LastUpdateTime fields are usually not used as a result.
  • Time skew across components prevents the time fields from being used for decision-making.
  • LastTransitionTime and Message are primarily for debugging, and overlap enough with Events to reconsider them.
  • As discussed in the API orchestration issue, we will want to be able to interpret common statuses polymorphically across resource types, but conditions are neither necessary nor sufficient for that, and it's likely that subresources aren't necessary, either. I'm currently inclined towards more of a duck-typing approach. Top-level status fields with the same name should have identical semantics across resources.
  • We're converting most Node conditions to taints, which can express more clear and more sophisticated properties, and which enable more robust and generic handling in schedulers.
  • Conditions were intended to be handled uniformly, across multiple conditions within a resource and across resources, and extensibly -- it should be possible for kubectl to extract status.conditions from any resource and display them. However, Conditions are neither consistent across condition types nor across resources, currently. Additionally, in kubectl get output, we found that just a boolean (true/false) wasn't sufficiently informative. Users found it useful to pack more information into a single field, such as CrashLoopBackoff. I do think that retaining subcategorization of exceptional conditions is valuable to users.
Owner

erictune commented Aug 22, 2017

I filed #50798 with data about use of conditions that supports the conclusion to not use conditions.

Owner

bgrant0607 commented Aug 23, 2017

@erictune's comment on #50798:

I've surveyed kubernetes API types to see how Status Conditions are used, and evaluated their usability as well.

Method: Examine 54 kubernetes API types, spanning core types, incubator project, and operator (TPR/CRD) types. Detailed Data. Deeper examination of condition subfield on 14 conditions across 4 api types.

Goal: see whether Status.Conditions is useful and how used.

Survey Findings:

  • Status is widely used across Kubernetes types. Half of the types surveyed have Status.
  • Status.Conditions is not widely used. 18% of types surveyed have Status.Conditions.
  • Ready is the most common condition type for Status.Conditions.
  • Only 5 types have a Condition type with a meaning other than Ready.
  • In many cases, there are non-Condition status items that convey some of the same information as the conditions.
  • Most .status.condition types do not use the Reason or Message fields.
  • Of those that do, the Reason often does not contain additional information beyond what can be inferred from Type and Status.
  • Of those that do, all but one have Reason and Message which contain the same information. One had some additional information in Message (the intended use of the field).
  • Times (Probe/Update/Transition) were not used by 2 of the 4 types with conditions that were closely examined.
  • Some conditions that are about Readiness do not provide Reason or Message

Usability Issues with Conditions:

  • A Condition is verbose: it takes 4 lines, whereas a normal k-v pair in Status can express the main thing (whether the condition is met) in 1 line. Some status stanzas don't fit on a screen (let alone in a UI Card).
  • Because it is in a list rather than a map, and information is spread across two lines, it is a lot harder to extract condition using common tools like grep, awk, jq or kubectl -o jsonpath, compared to single k-v pair Status items. Source code to evaluate truth of a condition is also verbose compare to plain fields (search array vs lookup).

Possible Conclusions:

  • All of those 5 types that do use conditions (other than Ready) were implemented in part by very early project members. These people were involved in the discussions about the Conditions type and so baised towards using it.
  • Many other types have been added since then, and nobody has followed the convention of using a list of several Conditions and most don't use conditions at all. This suggests that most people don't find them useful, and none find it useful outside of Readiness.
  • Readiness condition might be a generally useful concept. (It is discussed in these issues #1899 and #34363). It does not appear to strongly require the other fields of Conditions, though.

Possible next Steps:

  • We could make it official that use of Conditions is not recommended for new types.
  • When we move existing types to new stable versions (such as moving the core controllers to apps/v1, we could drop Status.Conditions.
  • Readiness can be expressed as a non-condition status field.
  • Other status can be dropped or expressed as a non-condition status field.

cc @pmorie

@bgrant0607 bgrant0607 added the sig/apps label Aug 23, 2017

Contributor

smarterclayton commented Aug 29, 2017

Many other types have been added since then, and nobody has followed the convention of using a list of several Conditions and most don't use conditions at all. This suggests that most people don't find them useful, and none find it useful outside of Readiness.

I don't know that this follows from the prior (or at least, needs more supporting evidence gathered). I see lots of use of conditions in monitoring tools, have seen it in alerting and UIs, and it is the primary mechanism to communicate status info beyond events (which suffer from their own problems). If we remove conditions, we have to assert every place that is using them and assess the impact of removing them.

(maybe I missed a deeper discussion in one of the SIG meetings that goes over the findings here?)

Contributor

ash2k commented Aug 30, 2017

To add a data point - we use conditions in Smith and I find them quite convenient.
https://github.com/atlassian/smith/blob/master/pkg/controller/controller_worker.go#L392-L429
There are 3 conditions: Ready, InProgress and Error.

I thought it is a cool pattern and some generic machinery can be provided if objects expose their status in a more uniform, programmable way. E.g. a generic "wait for object(s) to reach a set of conditions". The more uniform api objects are the more generic tooling can be implemented for them. Think about TypeMeta and ObjectMeta - it is completely uniform across all objects and because of that it is possible to work with them generically. Why it is not the same/similar for status? Maybe things should change in the opposite direction (towards more broad use of conditions)? Right now readiness may be the only widely used condition but who knows what may be needed one day.

I think we should keep in mind external api types. How can we make things look and work consistently across built-in and external api types?

If people were not following the recommendation maybe it was not promoted enough? Or maybe there should be an API review process (I don't know, maybe there is)?

Owner

bgrant0607 commented Aug 30, 2017

@ash2k In fact, the current thinking is to use an approach more similar to metadata: simply uniform status fields with consistent paths and meaning.

Contributor

seh commented Aug 30, 2017

I too was surprised by the change of heart here, having missed the start of the discussion 2.5 years ago. Better late than never.

What I appreciate about the conditions approach is that it's multidimensional. The burden is on the status type author to come up with dimensions that are orthogonal; that required careful thought when writing my own condition types. Unfortunately, though, the conditions approach is really just a convention; there's no relationship between any of the condition types, as written in Go, at least, other than a few of their serialized field names being the same.

Another challenge I've faced is writing a controller that updates the conditions in the status of objects that it also monitors. It's possible to stir up an infinite loop if the conditions don't remain stable. I'll give a clarifying example.

I have a TPR data object (not yet a CRD) that includes a condition indicating whether it's valid. (We can't reject an invalid object at time of creation or update.) We observe a change to the object, inspect it, and find a few invalid details, and want to include them in the condition's detail. We publish an update to the object's status including this condition with the validity problems, but then we observe our own update, validate the object again, find a few invalid details, and make sure that the conditions include those complaints. If the complaints don't match the complaints we published last time exactly, we'll conclude that we need to publish another update, which we'll then observe, and around it goes again. At first we were using a Go map value to collect these validity failures, and were concatenating them for the condition detail—in random order. Hence we discovered that our condition values need to be fixed points, lest we keep trying to publish slightly different variants of what are really the same logical assertions.

Granted, those same problems can arise in a controller using any other status fields too. The difficulty of working with the condition types—finding which conditions are present in the status, and which is which—obfuscated this problem.

Owner

bgrant0607 commented Aug 30, 2017

@smarterclayton Spurred by questions for @pmorie and decisions that need to be made for the workload APIs before they advance to v1, @erictune and I independently analyzed current usage of conditions and came to the same conclusion.

My analysis:
#7856 (comment)
Eric's:
#7856 (comment)

It's not that we want to drop the information provided by conditions, but to simplify the representation.

Owner

thockin commented Aug 30, 2017

My concern is that we have arbitrary things that controllers may want to say about an object. Many are carried in events today, which is imperfect for many reasons. Annotations feel like a hacky way of reporting status. Conditions are a cleaner status-annotation, IMO, but are perhaps under-specified.

Example that came up in the events thread is a pod that request a configmap be parsed into env vars, but some configmap keys are invalid syntactically for env vars. Today that is an event. "I started this pod, but you should know, not everything from the configmap made it through". I expect more such cases exist and need an outlet.

Contributor

smarterclayton commented Aug 31, 2017 edited

I do agree there are simplifications to be made. Here's how OpenShift uses conditions:

For these types we use message, reason, transition time, and type extensively. I don't think we use probe time outside of image signatures.

I was fairly certain the cli used conditions in its "overview status" graph traversal, but can't find any examples. The UI uses conditions mostly as the CLI does for describe - routes (ingresses), for instance, are specified with:

status:
  ingress:
    - conditions:
        - lastTransitionTime: '2017-08-30T15:48:28Z'
          status: 'True'
          type: Admitted
      host: origin-sq-status-ci.svc.ci.openshift.org
      routerName: router
      wildcardPolicy: None

where the router provides the host, status, and transition time. If Admitted was false, there would be an error which can be any of a number of reasons:

  • Route has the same host as another route from another tenant - router rejected because domain squatting is a security risk
  • Route has a TLS certificate that didn't pass validation in how it would be used - router rejects to protect one tenant from abusing other tenants with certs
  • Route specified a custom host, but the admin of the router rejects that host because it doesn't fit guidelines allowed for external host names (effectively arbitrary reasons)
Contributor

smarterclayton commented Aug 31, 2017

Etcd operator - https://github.com/coreos/etcd-operator/blob/master/pkg/spec/cluster.go#L222

Recovering is an interesting one.

Member

hongchaodeng commented Sep 9, 2017

Well.. The current status in EtcdCluster type in etcd operator was hacked for UI.. There still needs some improvement..

I can share our lessons in building operators. I couldn't speak highly enough of Tim's comment. Over time, conditions are very useful for human debugging/troubleshooting rather than composing automation:

  • It would be better to split them. Not in annotations, not in other status fields.
  • It would be better for user facing conditions to be highly extensible.

Of course, there are many ways to do troubleshooting. It's just one opinion :)

Owner

lavalamp commented Sep 14, 2017

Man, this issue brings back memories.

From @bgrant0607's analysis a few comments up I think a possible interpretation is: it's not that conditions have been tried and found too hard, but rather they have been found too hard and mostly left untried.

I still think the "outlook" model deserves a trial.

I think promoting every condition to be a loose field under status is likely to be a disaster. I think we should still group them so they aren't confused with other state. I'm also not sure how to give an explanation for a condition if they are bool fields. I guess they could instead be string fields, with the empty string meaning "everything is good" and any other string meaning "things are not good" with the string describing why they are not good. But that would be confusing. Is there a concrete replacement recommendation that I missed?

Owner

bgrant0607 commented Sep 15, 2017

@lavalamp

Using strings to represent the reasons for exceptional conditions could look as follows:

status:
  unquiescentReason: "N/A"  # I didn't use "False" due to confusion with JSON false
  unreadyReason: "N/A"
  unavailableReason: "RecentlyRestarted"

or:

status:
  reasonNotQuiescent: "ScalingUp"
  reasonNotReady: "Pending"
  reasonNotAvailable: "Pending"

I thought about empty string rather than "N/A", but that seemed less clear.

This does have the issue that the conditions are expressed in the negative sense, but I think that's inevitable, since that's the case where additional information is desired.

Member

kargakis commented Sep 15, 2017

@bgrant0607 what about timestamps? There are conditions where we want to know when they were last updated (Progressing in Deployments) or flipped from False to True (Ready in Pods).

Owner

bgrant0607 commented Sep 15, 2017

@kargakis Timestamps that may be collected across multiple nodes, such as controller-manager/kubelet or even multiple instances of a HA cluster or even after moving the master from a dead node, risk time skew. Even if NTP is set up correctly, there may still be non-negligible skew. So, the approach we've been moving towards is conservative in-memory tracking within a single instance, based on when it notices the transitions. Moving minReadySeconds to Kubelet will enable Kubelet to do that for Ready. The deployment controller can do that for Progressing. Due to the cost, most controllers didn't record the last updated (heartbeat or probe) time, anyway.

Owner

bgrant0607 commented Sep 15, 2017

If there are a few cases where timestamps are useful for debugging, those could also just be added as fields.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment