Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Many SidecarWriter write failed warnings #1025

Closed
jzhuge opened this issue Jan 10, 2023 · 2 comments · Fixed by #1026
Closed

Many SidecarWriter write failed warnings #1025

jzhuge opened this issue Jan 10, 2023 · 2 comments · Fixed by #1026
Milestone
1.5.2

Comments

@jzhuge
Copy link
Contributor

jzhuge commented Jan 10, 2023

Still many warnings when the node does not have sidecar running despite #1020

23/01/10 08:36:38 WARN SidecarWriter: write to udp://127.0.0.1:1234 failed: g:jvm.gc.maxDataSize:5.36870912E9
java.net.PortUnreachableException
...
23/01/10 08:37:18 WARN SidecarWriter: write to udp://127.0.0.1:1234 failed: g:jvm.buffer.count,id=mapped:0.0
...
23/01/10 08:37:18 WARN SidecarWriter: write to udp://127.0.0.1:1234 failed: g:jvm.gc.maxDataSize:5.36870912E9
@brharrington brharrington added this to the 1.5.2 milestone Jan 10, 2023
@brharrington
Copy link
Contributor

It appears that it doesn't always reliably detect that the write failed and so some of the calls do not throw which is being treated as a success. For now we'll limit it to just the first failure.

brharrington added a commit to brharrington/spectator that referenced this issue Jan 10, 2023
@brharrington
sidecar: only log the first failure
55d3a93 
Before we would reenable after successful calls so we could
detect if the sidecar was having periodic problems. However,
for UDP some calls can appear to be successful even if they
have failed so it still leads to spammy logs. For now limit
to the first failure.

Fixes Netflix#1025.
@brharrington brharrington mentioned this issue Jan 10, 2023
Merged
@jzhuge
Copy link
Contributor Author

jzhuge commented Jan 10, 2023

Thanks for the quick fix!

brharrington added a commit to brharrington/spectator that referenced this issue Apr 6, 2023
@brharrington
sidecar: only log the first failure (Netflix#1026)
dc4f698 
Before we would reenable after successful calls so we could
detect if the sidecar was having periodic problems. However,
for UDP some calls can appear to be successful even if they
have failed so it still leads to spammy logs. For now limit
to the first failure.

Fixes Netflix#1025.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.5.2
Development

Successfully merging a pull request may close this issue.

sidecar: only log the first failure brharrington/spectator
2 participants
@brharrington @jzhuge