kubectl wait on arbitrary jsonpath

[dims]

kubectl currently can wait on condition or delete
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kubectl/pkg/cmd/wait/wait.go#L110-L111

In addition to the above, can we please add the ability to wait on high level summary that we have phase?
https://cs.k8s.io/?q=Phase%20.*json%3A%22phase&i=nope&files=&repos=kubernetes/kubernetes

why? Looks like we are not adding conditions anymore, so CRD(s) seem to have picked up the phase at least. Will help avoid having to write loops like this ( from https://github.com/kubernetes-sigs/cluster-api-provider-gcp/pull/175/files )

  # Wait till all machines are running (bail out at 30 mins)
  attempt=0
  while true; do
    kubectl get machines --kubeconfig=$(kind get kubeconfig-path --name="clusterapi")
    read running total <<< $(kubectl get machines --kubeconfig=$(kind get kubeconfig-path --name="clusterapi") \
      -o json | jq -r '.items[].status.phase' | awk '/running/ {count++} END{print count " " NR}') ;
    if [[ $total == "5" && $running == "5" ]]; then
      return
    fi
    timestamp=$(date +"[%H:%M:%S]")
    if [ $attempt -gt 180 ]; then
      echo "cluster did not start in 30 mins ... bailing out!"
      exit 1
    fi
    echo "$timestamp Total machines : $total / Running : $running .. waiting for 10 seconds"
    sleep 10
    attempt=$((attempt+1))
  done

Another example is : https://github.com/kubernetes-incubator/kube-aws/blob/master/contrib/cluster-backup/restore.sh#L80-L86

[dims]

/sig cli

[liggitt]

Looks like we are not adding conditions anymore

The reports of conditions' death have been greatly exaggerated :)

Rather than a specific phase field, a jsonpath evaluation and expected value might be interesting

[dims]

something like the following?

wait --for='jsonpath={.status.phase == "Running"}'
wait --for='jsonpath={.items..status.phase == "Ready"}'

[dims]

guess what i want to see is ... wait until all the items reached the phase (or) wait until the single item reached the phase i specified. right?

[dims]

cc @hpandeycodeit

[dims]

@deads2k Can you please suggest some command lines that will be useful for a "generic wait"?

[hpandeycodeit]

/assign

[deads2k]

Conditions are generally better than phase. Phases were a mistake in our original pod implementation because they implied a single unified state machine, but we found that instead we had multiple orthogonal states that are better represented as separate conditions.

You could try kubectl wait --for=jsonpath=.status.phase=wish-i-was-a-condition

[dims]

A bit more on chat:

dims 10:18 AM if the resource.group/resource.name | resource.group has multiple items, then we ensure that the jsonpath is true for each? what happens when there is nothing selected?
deads2k 10:19 AM I think the jsonpath must result in a single node. If it results in multiple nodes, it should fail. If it results in zero nodes, it waits but we shouldn't codify phase

[aparedero]

@deads2k But how can you capture when a pod is up and running, but not ready?
Just a small example with Hashicorp Vault Helm and this being managed with Ansible.

Putting Ansible aside this helm manifest deploys a pod with Vault but it won't be ready until the application has been configured; so is 0/1 but in running state (due to a readiness probe).

NAME      READY   STATUS    RESTARTS   AGE
vault-0   0/1     Running   0          3m55s

Currently tt's required to execute some kubectl exec commands into this pod to become ready.

As I'm automating the process with Ansible I've tried creating a task with the bare command
kubectl wait --for=condition=Initialized pod/vault-0 --timeout=90s but its not working because the initalized state does not reflect the pod is still being created, so the "phase" is the most accured state to wait.

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-11-04T10:54:55Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2019-11-04T10:54:55Z"
    message: 'containers with unready status: [vault]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2019-11-04T10:54:55Z"
    message: 'containers with unready status: [vault]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2019-11-04T10:54:55Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: vault:1.2.2
    imageID: ""
    lastState: {}
    name: vault
    ready: false
    restartCount: 0
    state:
      waiting:
        reason: ContainerCreating
  hostIP: 172.16.16.97
  phase: Pending
  qosClass: BestEffort
  startTime: "2019-11-04T10:54:55Z"

As a workaround, I'm handling the error of not ready adding retries in Ansible, but I think it could be interesting having more control from K8s

[nicorikken]

I have a similar use-case, that can benefit from this feature. I'm waiting for a Kubernetes Job that runs an integration test. I want to wait for the output in my Jenkins CI/CD tool.

Now we wait for completion:

kubectl wait --for=condition=complete --timeout=${timeoutMinutes}m job/${jobName} --namespace ${namespace}

But if the job fails, the waiting continues until the timeout, as the failed condition is not equal to the complete condition. I prefer to have an early return on waiting, so I can retrieve the logs and mark the CI/CD job as failed and notify the developers.

Design

Logical conditions

I was expecting the CLI to take multiple arguments as conditions:

kubectl wait --for=condition=failed,complete

But that doesn't work (yet 🤞 ). As a job can only have one condition, providing multiple options as OR logic seems to be sufficient for all use-cases.

Generic

Going by this thread, I could use something like:

kubectl wait --for='jsonpath={ (.status.conditions.type == "Complete") or (.status.conditions.type == "Failed" }'

If I understand correctly, the logic operators are currently lacking in the Kubectl JSONpath syntax.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nicorikken]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[qalinn]

any update on this thread ?

[AjithPanneerselvam]

/assign

[MadhavJivrajani]

/unassign @hpandeycodeit
Un-assigning due to inactivity.
Please feel free to provide inputs/feedback. And also re-assign it to yourself at a later point if @AjithPanneerselvam can't work on this, thanks! :)

[joca-bt]

We also need this. It's flexible enough to cover lots of use cases.

[dims]

it's been more than an a month
/unassign AjithPanneerselvam

[lauchokyip]

Will work on this incorporate with PR #92277 and #83959.
I am new to kubectl wait and JSON handling so definitely need some time to come out with a PR. Please be patient :)

/assign

Update (10/4): A bit busy, but making little progress. Have tested Jsonpath function (https://github.com/kubernetes/client-go/blob/master/util/jsonpath/jsonpath.go) , will start implementation soon

Update (10/19): Finished implementation, will clean up and add more tests soon.

Update (10/25): Adding unit test cases

Update (10/27): Finished testing. Ready to review and merge

[ldemailly]

which version of kubectl has this? any example btw?

[lauchokyip]

which version of kubectl has this? any example btw?

@ldemailly it's available from kubectl version 1.23, you can try kubectl wait --help to look for examples

[minherz]

It seems that the current --for=jsonpath syntax does not support waiting for any ("non-empty") value.
In my use case, I would like to wait until an instance of the external load balancer is provisioned. I cannot do it on the instance of the load balancer itself because I cannot identify it using K8s. So, it leaves me with the readiness indicator that is the status.loadBalancer.ingress field of the service. If I call wait for the value of status.loadBalancer.ingress[0].ip, I need to provide a concrete "value". But I cannot know the IP of the load balancer.

Is it possible to modify the existing syntax of --for=jsonpath='{}'=value to support also return on non-empty value when querying for jsonpath only: --for=jsonpath='{}'?

[jonashackt]

Would be highly appreciated to have kubectl wait be usable with status.loadBalancer.ingress[0].ip or status.loadBalancer.ingress[0].hostname (as also discussed here and here). Allowing an empty value like @minherz suggested would be a great option. Maybe this needs another issue, since that one here is closed?

[minherz]

@lauchokyip can you please advise if the implementation that you merged can be used to reach the goal mentioned in these two comments (1 and 2)?

[lauchokyip]

Hi @minherz , thanks for bringing it up. I used to contribute during my free time but currently I am looking for new opportunities so I don't have the resources to do that. Would you be able to bring it up to sig-cli meeting to get their attention? Much appreciated

[minherz]

@lauchokyip thank you. I've submitted a discussion topic for the next sig-cli bi-weekly meeting. I will post the results later.

[minherz]

Following today's SIG CLI meeting, I have created a new feature request #117761.

[minherz]

@jonashackt I hope that the change in the version 1.28 should allow to wait for a K8s service to get its IPs. Syntax is supposed to be like:

kubectl wait --for=jsonpath='{.status.loadBalancer.ingress}' service/my_lb_service

or the old style:

kubectl wait --for=jsonpath='{.status.loadBalancer.ingress[0].ip}' service/my_lb_service

[volatilemolotov]

@jonashackt I hope that the change in the version 1.28 should allow to wait for a K8s service to get its IPs. Syntax is supposed to be like:

kubectl wait --for=jsonpath='{.status.loadBalancer.ingress}' service/my_lb_service

or the old style:

kubectl wait --for=jsonpath='{.status.loadBalancer.ingress[0].ip}' service/my_lb_service

This works in 1.28

[minherz]

@volatilemolotov thank you for confirming that!