Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hubble: Remove spammy debug log message on lost events #25321

Merged

Conversation

pchaigno
Copy link
Member

@pchaigno pchaigno commented May 8, 2023

When debug log is enabled, and in particular when monitor aggregation is disabled [1], this debug message tends to spam the agent logs. Instead of rate-limiting it, given we already have a metric for the same (hubble_lost_events_total), let's just remove it.

1 - This combination of debug enabled and monitor aggregation disabled is very frequent on datapath debugging sessions.

When debug log is enabled, and in particular when monitor aggregation is
disabled [1], this debug message tends to spam the agent logs. Instead
of rate-limiting it, given we already have a metric for the same [2],
let's just remove it.

1 - This combination of debug enabled and monitor aggregation disabled
   is very frequent on datapath debugging sessions.
2 - The hubble_lost_events_total metric.
Signed-off-by: Paul Chaignon <paul.chaignon@gmail.com>
@pchaigno pchaigno added release-note/misc This PR makes changes that have no direct user impact. sig/hubble Impacts hubble server or relay labels May 8, 2023
@pchaigno pchaigno requested a review from kaworu May 8, 2023 18:26
@pchaigno pchaigno requested a review from a team as a code owner May 8, 2023 18:26
@pchaigno
Copy link
Member Author

pchaigno commented May 9, 2023

I don't think we need to run the full end-to-end tests for this simple removal of a debug log, so marking ready to merge.

@pchaigno pchaigno added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label May 9, 2023
@youngnick youngnick merged commit ffa06b1 into cilium:main May 9, 2023
45 checks passed
@pchaigno pchaigno deleted the remove-spammy-debug-log-on-lost-events branch May 9, 2023 16:46
@learnitall learnitall added the needs-backport/1.13 This PR / issue needs backporting to the v1.13 branch label Sep 5, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot added this to Needs backport from main in 1.13.3 Sep 5, 2023
@learnitall
Copy link
Contributor

Hey! I'd like to have this backported to 1.13.6, please. This debug log can have a negative impact on CPU utilization, see #27661

@gandro gandro mentioned this pull request Sep 12, 2023
6 tasks
@gandro gandro added backport-pending/1.13 The backport for Cilium 1.13.x for this PR is in progress. and removed needs-backport/1.13 This PR / issue needs backporting to the v1.13 branch labels Sep 12, 2023
@julianwiedmann julianwiedmann added backport-done/1.13 The backport for Cilium 1.13.x for this PR is done. and removed backport-pending/1.13 The backport for Cilium 1.13.x for this PR is in progress. labels Sep 14, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Needs backport from main to Backport done to v1.13 in 1.13.3 Sep 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-done/1.13 The backport for Cilium 1.13.x for this PR is done. ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/misc This PR makes changes that have no direct user impact. sig/hubble Impacts hubble server or relay
Projects
No open projects
1.13.3
Backport done to v1.13
Development

Successfully merging this pull request may close these issues.

None yet

8 participants