Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add FILTER_METRICS_ACCESS_LOGS flag that allows to silence metrics ac… #2337

Merged

Conversation

RafalSkolasinski
Copy link
Contributor

…cess logs

What this PR does / why we need it:

This allow to silence access logs to the metrics endpoint. Due to periodic and constant probing from Prometheus this obscures the log.

Which issue(s) this PR fixes:

Closes #1907

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Add `FILTER_METRICS_ACCESS_LOGS` flag (environmental variable) that allows to silence access logs related to metrics endpoint.

@seldondev
Copy link
Collaborator

Thu Aug 27 16:46:13 UTC 2020
The logs for [lint] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2337/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2337 --build=2

@seldondev
Copy link
Collaborator

Thu Aug 27 16:47:15 UTC 2020
The logs for [pr-build] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2337/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2337 --build=1

@RafalSkolasinski
Copy link
Contributor Author

/test integration

@seldondev
Copy link
Collaborator

Thu Aug 27 17:25:15 UTC 2020
The logs for [integration] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2337/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2337 --build=3

python/seldon_core/microservice.py Show resolved Hide resolved
@@ -188,6 +195,11 @@ def setup_logger(log_level: str) -> logging.Logger:
flask_logger = logging.getLogger("werkzeug")
flask_logger.setLevel(log_level_num)

if getenv_as_bool(FILTER_METRICS_ACCESS_LOGS_ENV_NAME, default=False):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though it would be a "breaking change", would it be too crazy to filter out the metrics logs by default? What are your thoughts on this @RafalSkolasinski ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely something we could discuss but you're right on point - I wanted to avoid breaking changes

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make sense to set it as the default on 1.3. We can always make it very clear on the release notes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would also make sense to tie it to debug mode

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As in: if debug mode run verbose

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, will make it default and will ping you when ready for re-review!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adriangonz made filtering default in production mode but disabled if SELDON_DEBUG flag is set, tell me what you think :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, tested all combinations

@RafalSkolasinski RafalSkolasinski marked this pull request as draft September 1, 2020 13:02
@RafalSkolasinski RafalSkolasinski marked this pull request as ready for review September 7, 2020 16:15
@seldondev
Copy link
Collaborator

Mon Sep 7 16:16:30 UTC 2020
The logs for [pr-build] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2337/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2337 --build=4

@seldondev
Copy link
Collaborator

Mon Sep 7 16:17:54 UTC 2020
The logs for [lint] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2337/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2337 --build=5

@adriangonz
Copy link
Contributor

/approve

Change looks good!

@seldondev
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adriangonz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@seldondev
Copy link
Collaborator

Wed Sep 9 09:01:55 UTC 2020
The logs for [pr-build] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2337/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2337 --build=6

@seldondev
Copy link
Collaborator

Wed Sep 9 09:01:59 UTC 2020
The logs for [lint] [7] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-2337/7.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-2337 --build=7

@seldondev seldondev merged commit 13762c0 into SeldonIO:master Sep 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

silence flask logs from prometheus probing python wrapper
3 participants