Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable access logs for health endpoints by default #448

Closed
3 of 6 tasks
TwiN opened this issue Dec 29, 2021 · 5 comments
Closed
3 of 6 tasks

Disable access logs for health endpoints by default #448

TwiN opened this issue Dec 29, 2021 · 5 comments
Assignees
Labels
bug Something is not working.

Comments

@TwiN
Copy link

TwiN commented Dec 29, 2021

Preflight checklist

Describe the bug

Access logs from Kubernetes' readiness and liveness probes are polluting the logs, which makes it harder to debug issues.

Reproducing the bug

  1. Deploy hydra using https://github.com/ory/k8s/tree/master/helm/charts/hydra
  2. Wait for a few minutes
  3. View the logs and note how there are several logs like the ones in the Relevant log output section below.

Relevant log output

time=2021-12-29T19:39:28Z level=info msg=started handling request http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.21] host:10.42.1.105:4445 method:GET path:/health/ready query:<nil> remote:10.42.x.x:38882 scheme:http]
time=2021-12-29T19:39:28Z level=info msg=started handling request http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.21] host:10.42.1.105:4445 method:GET path:/health/alive query:<nil> remote:10.42.x.x:38884 scheme:http]
time=2021-12-29T19:39:28Z level=info msg=completed handling request http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.21] host:10.42.1.105:4445 method:GET path:/health/alive query:<nil> remote:10.42.x.x:38884 scheme:http] http_response=map[headers:map[content-type:application/json; charset=utf-8] size:16 status:200 text_status:OK took:405.428µs]
time=2021-12-29T19:39:28Z level=info msg=completed handling request http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.21] host:10.42.1.105:4445 method:GET path:/health/ready query:<nil> remote:10.42.x.x:38882 scheme:http] http_response=map[headers:map[content-type:application/json; charset=utf-8] size:16 status:200 text_status:OK took:5.90746ms]

Relevant configuration

No response

Version

v1.10.6-sqlite

On which operating system are you observing this issue?

No response

In which environment are you deploying?

Kubernetes with Helm

Additional Context

I believe that the endpoints used for health checks (/health/ready and /health/alive) should not be logged by the middleware taking care of access logs.

There is already a feature in place to enable this, and it is used to ignore the /ping endpoint:

func (m *Middleware) ExcludePaths(paths ...string) *Middleware {


Update

See comments below; it turns out there is a way to remove access logs for health endpoints (serve.public.access_logs.disable_for_health and serve.admin.access_logs.disable_for_health), so instead, I'll like to know if there's really any reason to not enabling these by default? Just fixing it for my project is trivial -- it's just a configuration change -- but I feel that disabling this by default might make the experience a bit simpler for devs.

@TwiN TwiN added the bug Something is not working. label Dec 29, 2021
@TwiN TwiN changed the title access logs Logs are poluted by middleware when hydra is deployed in Kubernetes Dec 29, 2021
@TwiN TwiN changed the title Logs are poluted by middleware when hydra is deployed in Kubernetes Logs are poluted by middleware when hydra is deployed on k8s due to probes Dec 29, 2021
@TwiN
Copy link
Author

TwiN commented Dec 29, 2021

Upon closer investigation, it looks like the health endpoints should be excluded already: https://github.com/ory/hydra/blob/f588ec69d4fa03f602d3cbb20abd4188195a7375/cmd/server/handler.go#L204

I'll investigate a bit deeper.


Looks like you have to explicitly disable it: https://github.com/ory/hydra/blob/f588ec69d4fa03f602d3cbb20abd4188195a7375/driver/config/serve.go#L71

@TwiN
Copy link
Author

TwiN commented Dec 29, 2021

So it looks like this was my fault and I didn't investigate deeply enough before creating this; it's actually documented in https://www.ory.sh/hydra/docs/reference/configuration/

    access_log:
      ## disable_for_health ##
      #
      # Disable access log for health endpoints.
      #
      # Default value: false
      #
      # Set this value using environment variables on
      # - Linux/macOS:
      #    $ export SERVE_ADMIN_ACCESS_LOG_DISABLE_FOR_HEALTH=<value>
      # - Windows Command Line (CMD):
      #    > set SERVE_ADMIN_ACCESS_LOG_DISABLE_FOR_HEALTH=<value>
      #
      disable_for_health: false

With that in mind, I wonder how feasible it would be to disable this by default. Does anybody truly benefit from having access logs enabled for health endpoints?

@TwiN TwiN changed the title Logs are poluted by middleware when hydra is deployed on k8s due to probes Disable access logs for health endpoints by default Dec 29, 2021
@aeneasr
Copy link
Member

aeneasr commented Dec 30, 2021

Thank you for the report! Indeed, the default is that all logs are available. That way you can investigate easily wether e.g. health checks pass. This can help when services start up and are not responsive. If you have high traffic websites or production deployments, it usually makes sense to disable them. Given that this option is available and documented (although we could probably document it better), do you feel that this issue can be closed?

@aeneasr
Copy link
Member

aeneasr commented Dec 30, 2021

For clarification, changing this to default to true might improve the experience for new users who expect this to be the case, but it will be a breaking change for people who rely on this feature. So we won't be changing the default, but it would make sense to better document it.

@aeneasr aeneasr closed this as completed Dec 30, 2021
@aeneasr aeneasr reopened this Dec 30, 2021
@aeneasr aeneasr self-assigned this Mar 1, 2022
@aeneasr
Copy link
Member

aeneasr commented Mar 1, 2022

I'm closing this as there are currently no plans to change the default

@aeneasr aeneasr closed this as completed Mar 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is not working.
Projects
None yet
Development

No branches or pull requests

2 participants