[Bug]: Kubernetes Readiness & Liveness Probe Require x-litellm-key Header.

[elisasimoni]

What happened?

I am deploying LiteLLM Proxy on Kubernetes and using ArgoCD for synchronization. The application is running, and both readiness and liveness probes return 200 OK, but the logs are continuously filled with the following error:

INFO: 10.116.161.254:36682 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO: 10.116.161.254:36690 - "GET /health/readiness HTTP/1.1" 200 OK
09:40:06 - LiteLLM Proxy:ERROR: user_api_key_auth.py:1490 - No LiteLLM Virtual Key pass. Please set header=x-litellm-key: Bearer <api_key>

The application itself is functioning correctly, but this error message keeps repeating, cluttering the logs.

Expected Behavior
• The /health/liveliness and /health/readiness endpoints should be accessible without requiring authentication.
• The logs should not be flooded with authentication errors when probes are executed.

Actual Behavior
• The logs keep showing the error:
"No LiteLLM Virtual Key pass. Please set header=x-litellm-key: Bearer <api_key>"
• This happens even though health probes return 200 OK and everything works fine.

already test this in my kubernetes with and without httpHeader, tried also calling the endpoint with a curl e it gives me 200:

      readinessProbe:
        httpGet:
          path: /health/readiness
          scheme: HTTP
          port: 8080
          httpHeaders:
            - name: x-litellm-key
              value: "Bearer readiness"
      livenessProbe:
        httpGet:
          path: /health/liveliness
          scheme: HTTP
          port: 8080
          httpHeaders:
            - name: x-litellm-key
              value: "Bearer liveliness"

Would appreciate any guidance on whether this is intended behavior or a misconfiguration on my side. Let me know if further details are needed!

Relevant log output

INFO:     10.116.161.254:36682 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.116.161.254:36690 - "GET /health/readiness HTTP/1.1" 200 OK
09:40:06 - LiteLLM Proxy:ERROR: user_api_key_auth.py:1490 - No LiteLLM Virtual Key pass. Please set header=x-litellm-key: Bearer <api_key>

Are you a ML Ops Team?

Yes

What LiteLLM version are you on ?

v1.60.0

Twitter / LinkedIn details

No response

[elisasimoni]

HI any news?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[ludens11]

@elisasimoni did you managed to solve the log error? Mine starting to produce such log when I defined the custom header in proxy_config:

general_settings:
  litellm_key_header_name: custom-header-key

When I removed it, the log disappeared. As for my use case, I need to set custom header name, but it seems the LiteLLM proxy error will always log such error.

[elisasimoni]

Hi! Sorry for the delay. I updated LiteLLM to the latest version and the issue resolved itself.