Plugins panic #8937
Comments
|
Hello @jliebert, Thanks for your interest in Traefik! Can you share the configuration of the router for which the problem is happening? |
|
Hello @rtribotte, I have 144 routers, 14 use blockpath plugins like this example: labels:
traefik.enable: 'true'
traefik.http.middlewares.dummy-site--web--headers.headers.customresponseheaders.server: server
traefik.http.middlewares.dummy-site--web--redirect-host.redirectregex.permanent: 'true'
traefik.http.middlewares.dummy-site--web--redirect-host.redirectregex.regex: '^https://([^/]+)'
traefik.http.middlewares.dummy-site--web--redirect-host.redirectregex.replacement: 'https://www.dummy.com'
traefik.http.routers.dummy-site--web--secure.entrypoints: https
traefik.http.routers.dummy-site--web--secure.middlewares: 'block-badpath@http,block-badpath-drupal@http,dummy-site--web--redirect-host,dummy-site--web--headers'
traefik.http.routers.dummy-site--web--secure.priority: '100'
traefik.http.routers.dummy-site--web--secure.rule: 'Host(`www.dummy.com`,`dummy.com`)'
traefik.http.routers.dummy-site--web--secure.tls: 'true'
traefik.http.services.dummy-site--web--secure.loadbalancer.server.port: '80'I found no correlation between client http request and panic or http request not being logged at all. |
|
Hello @jliebert, Thanks for your feedback, We have everything to try to reproduce the problem, we'll let you know if we can reproduce it, or get back to you if we need more information. |
rtribotte
added
kind/bug/possible
|
I still have "panic" with version 2.6.6. Our conclusion that was may be caused by a bug in Yaegi, reference from this discussion: traefik/plugin-blockpath#12 (comment) |
|
the same problems with version 2.9.4 {"level":"error","module":"github.com/traefik/plugin-blockpath","msg":"plugins-storage/sources/gop-3290823723/src/github.com/traefik/plugin-blockpath/blockpath.go:48:17: panic","plugin":"plugin-blockpath","time":"2022-11-20T17:13:37Z"}
{"level":"error","module":"github.com/traefik/plugin-blockpath","msg":"plugins-storage/sources/gop-3290823723/src/github.com/traefik/plugin-blockpath/blockpath.go:48:17: panic","plugin":"plugin-blockpath","time":"2022-11-20T17:13:38Z"}
{"level":"error","module":"github.com/traefik/plugin-blockpath","msg":"plugins-storage/sources/gop-3290823723/src/github.com/traefik/plugin-blockpath/blockpath.go:48:17: panic","plugin":"plugin-blockpath","time":"2022-11-20T17:13:50Z"} |
|
FYI, there is nothing in the core of the plugin that is not supported by yaegi, i.e. there is no problem if you manually run with yaegi something like that: package main
import (
"log"
"net/http"
"net/http/httputil"
"net/url"
"regexp"
)
type Middleware struct {
regexps []*regexp.Regexp
next http.Handler
}
func (h *Middleware) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
currentPath := r.URL.EscapedPath()
for _, re := range h.regexps {
println(currentPath, "VS", re.String())
if re.MatchString(currentPath) {
rw.WriteHeader(http.StatusForbidden)
return
}
}
h.next.ServeHTTP(rw, r)
}
func main() {
reg := regexp.MustCompile("/foo.*")
dest, err := url.Parse("http://localhost:6060")
if err != nil {
log.Fatal(err)
}
next := httputil.NewSingleHostReverseProxy(dest)
http.Handle("/", &Middleware{
next: next,
regexps: []*regexp.Regexp{reg},
})
log.Fatal(http.ListenAndServe(":9090", nil))
}Moreover, if there was something fundamental unsupported, then you'd get panics/errors for each request. |
|
I get the panics randomly only on production and I don't have service with websocket. |
|
@jliebert just a detail, but why do I see the name "traefik-ha_V2" in your logs? where does it come from? is it the name of a docker image you built yourself? |
|
"traefik-ha_V2" is the name of the stack on the Swarm cluster, I use the standard image for traefik service. |
|
We're seeing the same problem, only in our production environment as well which makes this really hard to debug as there's quite a lot of traffic so we can't just enable debug mode and I don't see anything strange in the access log (if it would even be logged in that case). For us this occurs with a custom middleware in this line: so I assume the request is null in that case as described above. |
|
Hello, I've observed the same issue with Traefik 2.9.6 and a basic plugin which only serves the requests. func (bouncer *Bouncer) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
if !bouncer.enabled {
bouncer.next.ServeHTTP(rw, req)
return
}
...
}An issue #60 is open in a plugin repository, but I believe the error could come from Traefik. I've found a way to reproduce the "panic" at each request. The dozzle container triggers this error each time you change container logs. Here is a following docker-compose that can be used: version: "3.8"
services:
traefik:
image: "traefik:v2.9.6"
container_name: "traefik"
restart: unless-stopped
command:
- "--log.level=DEBUG"
- "--accesslog"
- "--accesslog.filepath=/var/log/traefik/access.log"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--experimental.plugins.bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
- "--experimental.plugins.bouncer.version=v1.1.5"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "logs:/var/log/traefik"
ports:
- 8000:80
- 8080:8080
depends_on:
- 'crowdsec'
crowdsec:
image: crowdsecurity/crowdsec:v1.4.1
container_name: "crowdsec"
restart: unless-stopped
environment:
COLLECTIONS: crowdsecurity/traefik
CUSTOM_HOSTNAME: crowdsec
# We need to register one api key per service we will use
BOUNCER_KEY_TRAEFIK_1: FIXME-LAPI-KEY-1
volumes:
- ./acquis.yaml:/etc/crowdsec/acquis.yaml:ro
- logs:/var/log/traefik:ro
- crowdsec-db:/var/lib/crowdsec/data/
- crowdsec-config:/etc/crowdsec/
labels:
- "traefik.enable=false"
dozzle:
container_name: dozzle
image: amir20/dozzle:latest
volumes:
- /var/run/docker.sock:/var/run/docker.sock
labels:
- "traefik.enable=true"
- "traefik.http.routers.router-doz.rule=Host(`localhost`)"
- "traefik.http.routers.router-doz.entrypoints=web"
- "traefik.http.routers.router-doz.middlewares=crowdsec-doz@docker"
- "traefik.http.services.service-doz.loadbalancer.server.port=8080"
- "traefik.http.middlewares.crowdsec-doz.plugin.bouncer.enabled=false"
- "traefik.http.middlewares.crowdsec-doz.plugin.bouncer.crowdseclapikey=FIXME-LAPI-KEY-1"
- "traefik.http.middlewares.crowdsec-doz.plugin.bouncer.loglevel=DEBUG"
volumes:
logs:
crowdsec-db:
crowdsec-config:Navigate to http://localhost:8000/ This will trigger the following logs: It happens every time a new request is closed and open. |
|
Thank you, I'll see if I can reproduce as well with the new information. |
|
I found an easier way to reproduce this using httpbin: https://github.com/Thor77/traefik-plugin-panic |
|
@Thor77 Hey. I wanted to try your repro, but I haven't found github.com/thor77/forward . Is it a copy of another existing plugin that I can find elsewhere? |
|
@mpl it's included in the repo - https://github.com/Thor77/traefik-plugin-panic/tree/master/forward and mounted to the correct location in the docker-compose file. If there's any difference between the local plugin mode and plugins fetched remotely I can also create a separate repo for it. |
|
Hi all, TLDR: I believe there's nothing to worry about. Longer story: I believe what is happening (and I am almost sure of it in the case of @mathieuHa , because he provided debug logs), is simply that you see the response copy error that is happening when a request is cancelled (for a variety of reasons). When such a thing happens, the stdlib reverse-proxy's handler sends a panic with the http.ErrAbortHandler sentinel error (which is the "net/http: abort Handler" message we see in @mathieuHa logs). This panic is then caught by the caller, and in that particular sentinel case, it is suppressed, which is why (among other things) you do not see any stack trace. So I believe what you see is the intended behaviour, but if you want to make sure of it, you should enable the debug logs, and check whether you see the message with the abort Handler error, whenever you see the panic message from the plugin. If you see them together, then chances are there is nothing wrong going on. Now, the last bit that I have to investigate is whether there's something wrong with yaegi regarding the fact that it does print the panic message. I believe that it should not have had to deal with the panic in the first place, as the panic should have been suppressed before the recover behaviour of yaegi takes control, but I need to make sure of that. In any case, aside from making the log disappear, I don't think anything would change in the behaviour, so I'm not really worried either way. I'll keep the issue open for now, until I can figure out that last part. |
|
Hi, I don't panic about this panic, but how-to now when they are a real panic? "If all the boats send sos the rescue is lost". |
|
@jliebert that's what I already explained above. Enable debug level for logging, and if you see the "net/http: abort Handler" messages as well, then you're probably in the scenario I described. |
|
Hey @mpl, Thanks for the detailed description of what is happening. Clear on my side. Mathieu |
Welcome!
What did you do?
I use the plugins traefik/plugin-blockpath v0.2.1, the plugins is working, path are correctly blocked as I want.
But I get many "panic" error in the log files of Traefik.
What did you see instead?
Panic messages from Traefik:
What version of Traefik are you using?
Version: 2.6.1
Codename: rocamadour
Go version: go1.17.7
Built: 2022-02-14T16:50:25Z
OS/Arch: linux/amd64
What is your environment & configuration?
dynamic_conf.yaml is:
If applicable, please paste the log output in DEBUG level
The text was updated successfully, but these errors were encountered: