Support liveness and readiness probe with HTTP check that checks response text

[pjfanning]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind feature
/sig node

Problem:
This is equivalent to the HAProxy http-check expect string READY (https://www.haproxy.com/documentation/aloha/7-0/haproxy/healthchecks/)
Based http://kubernetes.io/docs/user-guide/liveness/ it appears that the healthz checks only check the HTTP response code.
I'm working with a health check that is a company standard and that returns status strings.

Proposed Solution:
Some capability to check the text of the response to see if it contains a particular string.
This check would probably need to only check the first n characters to prevent issues with memory.
HAProxy also supports an rstring that is a regex but my use case does not require it.

Page to Update:
http://kubernetes.io/docs/user-guide/liveness/

[bgrant0607]

This could also be done by an exec probe.

[pjfanning]

Thanks @bgrant0607 - that is the workaround we've been using - but it would be nice not to have a custom shell script that the exec probe calls.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[danielqsj]

Hi @bgrant0607 @thockin @liggitt , does this feature sound reasonable to you ?
I'd like to submit a PR to support this feature if you agree to do it.

[thockin]

I don't hate it, but I don't love it. I can't help but think there has to be some opaque way to do this internal to your app (or in a sidecar that hits the real URL and then converts string-valiation into 2xx or 4xx).

Is there really no better way?

[timotheobborges]

Hi,
I have the same company standard here (request a default url path [http://app_uri/status] and check if there is a string "OK"). This feature is common in load balancers to check the healthy of the web application. It would be great if it would be implemented in kubernetes Rediness/liveless httpget probe
best regards!

[dheerajdwivedi]

Did we conclude on this ?
I would like to pick this up.

Here is the proposed config format:

kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-http
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/liveness
    args:
    - /server
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
        httpHeaders:
        - name: Custom-Header
          value: Awesome
        response
          headers:
            status: []
          body:
            - content-type: application/json
              matchExpression: $.response.code
              value: 2004
      initialDelaySeconds: 3
      periodSeconds: 3

[jproig]

The proposed solution will be useful for us aswell.
We implemented Atlassian Jira DataCenter version, and we are trying to use the /status healtcheck, that URL always returns a 200 code, but the content in the body reflects the real status.

[HariSekhon]

+1 for HTTP content checks.

Many HTTP applications have JSON HTTP endpoints containing their status so checking for 200 OK is not sufficient.

Also, when shutting down services it's common for services behind load balancers to change a status content semaphore to allow for graceful removal from a serving pool.

Kubernetes should support at least:

• HTTP string content checks
• HTTP regex content checks

and preferably also:

• HTTP JSON field string content checks
• HTTP JSON field regex content checks

[mscholze]

+1 for HTTP content checks.

Found that issue when I've googled for http content checks for liveness and readiness probes. Also found some workarounds with shell + curl, but then we have to modify the image when curl is not installed.

[adisky]

/triage accepted
/priority low
We will evaluate it later!!

[k8s-ci-robot]

@adisky: The label(s) priority/low cannot be applied, because the repository doesn't have them.

In response to this:

/triage accepted
/priority low
We will evaluate it later!!

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[adisky]

/priority backlog

[khenidak]

I don't think we should be implementing this. Parsing the entire response in order to identify a certain health signal just adds complexities we don't need. Take the following examples:

• container x returns some large response. That is an added load on kubelet health check go routines.
• What happens to failures? e.g. we couldn't find string xyz or header abc these needs to be logged somewhere for users to debug.

The use-case is very specific to content-type and content and does seem like something users should warp in custom script that walks through the custom logic needed to match to a custom health signal beyond HTTP response status.

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[thockin]

/triage accepted

This probably warrants a small KEP to discuss.

Proposals are welcome

[SergeyKanzhelev]

substring is implemented in may systems like https://cloud.google.com/load-balancing/docs/health-check-concepts#criteria-protocol-http. So it makes sense to implement it in kubelet as well. It is a great quality of life improvement for kubernetes probes.

Some notes on the potential implementation:

1. We will need to introduce a limit - how many bytes we check for the substring. Not parse the entire response
2. Maybe we can limit this capability to the probes configured without custom host. Using it with custom host may expose some information from the node that we may not want to expose otherwise (

   kubernetes/pkg/probe/http/http.go

   Line 120 in 35f3fc5

   // Note: Until https://issue.k8s.io/99425 is addressed, this user-facing failure message must not contain the response body.

   ).

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[thockin]

/triage accepted

This remains a plausible feature request. Needs an owner and a KEP.

Question: substring on regexp?
Question: max body-len to parse?

One issue is that pod probes are sometimes used to derive external LB health-checks. Need to define how this change would affect that.