-
Notifications
You must be signed in to change notification settings - Fork 18.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Syslog entry: "superfluous response.WriteHeader call" in Docker 25.x #47448
Comments
docker compose stats will also create syslog messages |
This line currently accounts for 8% of my total server syslog/journal logs (out of about 120k lines). |
Wonder if this is a bug in OTEL here; I see it keeps track whether it already wrote headers, but then proceeds calling moby/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/wrap.go Lines 93 to 99 in 330d777
It looks like moby/vendor/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/wrap.go Lines 76 to 80 in 330d777
That said, the log itself seems to be coming from |
It would be nice to have a way to disable telemetry either. |
Hello, we have the same problem with this message since Docker 25. I tried to update to 26 but no changes:
There are about 50000 Lines a day spamming the Syslog. docker info
docker version
Additional Info |
Telemetry is disabled by default. |
Not in Go 😢 : open-telemetry/opentelemetry-go#3559 |
our tracing code supports this explicitly. |
Ah, I see. Thank you! But as you suspected, it doesn't seem to have any effect. |
I've done a bit of research into this issue with the help of @laurazard, here's the situation so far: What seems to be happening is that the OTEL wrapper is unaware of any calls that are made to the underlying ResponseWriter.WriteHeader() func it is wrapping. In scenarios where the underlying
As a general question i've been having: should the otel wrapper be this opinionated? In the case of Lines 47 to 53 in f9dfd13
I'm not sure if we can (or should) change that implementation as a fix for this issue (the commit where this is included on our side is from quite a while back, see here). |
Related to the above (at least from a discussion earlier today); c0391bf was (I think) where the "write headers early" was added) edit: this was related 1cbf5a5 We also discussed briefly; what if we would make the writer we pass on a wrapped writer that ignores writing headers (as we know we already sent headers, so anything after us was not expected to send any) |
To add a little more info, the "early write" was originally added in 1cbf5a5. Ahh, beat me to it @thaJeztah . |
Opened a quick draft pr that should fix the |
@krissetto I think the simplest fix here may be best, which is to just call Line 51 in a73e63c
Same for logs. |
Opened an issue over in the otelhttp repo: open-telemetry/opentelemetry-go-contrib#5438 In the meantime we can change from using direct flush calls to writing an empty value so it triggers otelhttp. |
Description
When retrieving the logs for a container (via
docker logs
for example), a log entry is made in/var/log/syslog
:This appears to have been introduced alongside the OpenTelemetry addition in Docker 25.0.0 as this log message is not created on previous versions of Docker (for example 24.0.6).
Reproduce
/var/log/syslog
(viatail -f
for example)docker logs containername
Expected behavior
No "superfluous response" message.
docker version
Client: Docker Engine - Community Version: 25.0.3 API version: 1.44 Go version: go1.21.6 Git commit: 4debf41 Built: Tue Feb 6 21:14:17 2024 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 25.0.3 API version: 1.44 (minimum version 1.24) Go version: go1.21.6 Git commit: f417435 Built: Tue Feb 6 21:14:17 2024 OS/Arch: linux/amd64 Experimental: true containerd: Version: 1.6.28 GitCommit: ae07eda36dd25f8a1b98dfbf587313b99c0190bb runc: Version: 1.1.12 GitCommit: v1.1.12-0-g51d5e94 docker-init: Version: 0.19.0 GitCommit: de40ad0
docker info
Additional Info
While a single log line entry doesn't seem like much, when using a separate tool (such as Portainer) to view container logs that auto-refreshes, this can result in repeated syslog entries.
The text was updated successfully, but these errors were encountered: