Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

replacer: use RWMutex to protect static provider #6184

Merged
merged 2 commits into from
Mar 21, 2024
Merged

replacer: use RWMutex to protect static provider #6184

merged 2 commits into from
Mar 21, 2024

Conversation

mohammed90
Copy link
Member

@mohammed90 mohammed90 commented Mar 21, 2024 •
edited

I received a report of concurrent map-write crash. When the the executable was built with the race detector, the following stacktrace is produced.

==================
WARNING: DATA RACE
Write at 0x00c00470cc00 by goroutine 3795:
  runtime.mapassign_faststr()
      /usr/local/go/src/runtime/map_faststr.go:203 +0x0
  github.com/caddyserver/caddy/v2.(*Replacer).Set()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/replacer.go:68 +0x9b
  github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP.func1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/reverseproxy/reverseproxy.go:432 +0x45
  runtime.deferreturn()
      /usr/local/go/src/runtime/panic.go:477 +0x30
  github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:331 +0x81
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/<private-org>/<private-repo>.PerHostRateLimiterMiddleware.ServeHTTP()
      /app/per_host_rate_limiter.go:101 +0x883
  github.com/<private-org>/<private-repo>.(*PerHostRateLimiterMiddleware).ServeHTTP()
      <autogenerated>:1 +0x112
  github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:331 +0x81
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/headers/headers.go:110 +0x4c9
  github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.(*Handler).ServeHTTP()
      <autogenerated>:1 +0x94
  github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:331 +0x81
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/cache-handler.(*SouinCaddyMiddleware).ServeHTTP.func1()
      /go/pkg/mod/github.com/caddyserver/cache-handler@v0.12.0/httpcache.go:85 +0x5e
  github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream.func2()
      /go/pkg/mod/github.com/mohammed90/souin@v0.0.0-20240317145109-54af8511c28e/pkg/middleware/middleware.go:345 +0xb3
  golang.org/x/sync/singleflight.(*Group).doCall.func2()
      /go/pkg/mod/golang.org/x/sync@v0.5.0/singleflight/singleflight.go:198 +0xc3
  golang.org/x/sync/singleflight.(*Group).doCall()
      /go/pkg/mod/golang.org/x/sync@v0.5.0/singleflight/singleflight.go:200 +0x153
  golang.org/x/sync/singleflight.(*Group).Do()
      /go/pkg/mod/golang.org/x/sync@v0.5.0/singleflight/singleflight.go:113 +0x204
  github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream()
      /go/pkg/mod/github.com/mohammed90/souin@v0.0.0-20240317145109-54af8511c28e/pkg/middleware/middleware.go:344 +0x3f9
  github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func4()
      /go/pkg/mod/github.com/mohammed90/souin@v0.0.0-20240317145109-54af8511c28e/pkg/middleware/middleware.go:709 +0x8e
  github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func10()
      /go/pkg/mod/github.com/mohammed90/souin@v0.0.0-20240317145109-54af8511c28e/pkg/middleware/middleware.go:710 +0x4f

Previous write at 0x00c00470cc00 by goroutine 3792:
  runtime.mapassign_faststr()
      /usr/local/go/src/runtime/map_faststr.go:203 +0x0
  github.com/caddyserver/caddy/v2.(*Replacer).Set()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/replacer.go:68 +0x1f6
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).logRequest()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/server.go:702 +0x192
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP.func2()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/server.go:328 +0xc4
  runtime.deferreturn()
      /usr/local/go/src/runtime/panic.go:477 +0x30
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2938 +0x2a1
  net/http.(*conn).serve()
      /usr/local/go/src/net/http/server.go:2009 +0xc24
  net/http.(*Server).Serve.func3()
      /usr/local/go/src/net/http/server.go:3086 +0x4f

Goroutine 3795 (running) created at:
  github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP()
      /go/pkg/mod/github.com/mohammed90/souin@v0.0.0-20240317145109-54af8511c28e/pkg/middleware/middleware.go:708 +0x1e75
  github.com/caddyserver/cache-handler.(*SouinCaddyMiddleware).ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/cache-handler@v0.12.0/httpcache.go:84 +0x112
  github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:331 +0x81
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:300 +0x471
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/subroute.go:74 +0x96
  github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:331 +0x81
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:300 +0x471
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/subroute.go:74 +0x96
  github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:331 +0x81
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:300 +0x471
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:268 +0x3a1
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:268 +0x3a1
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).enforcementHandler()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/server.go:429 +0x208
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Provision.(*Server).wrapPrimaryRoute.func1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/server.go:405 +0x68
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/server.go:341 +0x1598
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2938 +0x2a1
  net/http.(*conn).serve()
      /usr/local/go/src/net/http/server.go:2009 +0xc24
  net/http.(*Server).Serve.func3()
      /usr/local/go/src/net/http/server.go:3086 +0x4f

Goroutine 3792 (running) created at:
  net/http.(*Server).Serve()
      /usr/local/go/src/net/http/server.go:3086 +0x86c
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Start.func3()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/app.go:509 +0x4f
==================
==================
WARNING: DATA RACE
Write at 0x00c00470cc00 by goroutine 3795:
  runtime.mapassign_faststr()
      /usr/local/go/src/runtime/map_faststr.go:203 +0x0
  github.com/caddyserver/caddy/v2.(*Replacer).Set()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/replacer.go:68 +0x196
  github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP.func1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/reverseproxy/reverseproxy.go:433 +0xe2
  runtime.deferreturn()
      /usr/local/go/src/runtime/panic.go:477 +0x30
  github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:331 +0x81
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/<private-org>/<private-repo>.PerHostRateLimiterMiddleware.ServeHTTP()
      /app/per_host_rate_limiter.go:101 +0x883
  github.com/<private-org>/<private-repo>.(*PerHostRateLimiterMiddleware).ServeHTTP()
      <autogenerated>:1 +0x112
  github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:331 +0x81
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.Handler.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/headers/headers.go:110 +0x4c9
  github.com/caddyserver/caddy/v2/modules/caddyhttp/headers.(*Handler).ServeHTTP()
      <autogenerated>:1 +0x94
  github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:331 +0x81
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/cache-handler.(*SouinCaddyMiddleware).ServeHTTP.func1()
      /go/pkg/mod/github.com/caddyserver/cache-handler@v0.12.0/httpcache.go:85 +0x5e
  github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream.func2()
      /go/pkg/mod/github.com/mohammed90/souin@v0.0.0-20240317145109-54af8511c28e/pkg/middleware/middleware.go:345 +0xb3
  golang.org/x/sync/singleflight.(*Group).doCall.func2()
      /go/pkg/mod/golang.org/x/sync@v0.5.0/singleflight/singleflight.go:198 +0xc3
  golang.org/x/sync/singleflight.(*Group).doCall()
      /go/pkg/mod/golang.org/x/sync@v0.5.0/singleflight/singleflight.go:200 +0x153
  golang.org/x/sync/singleflight.(*Group).Do()
      /go/pkg/mod/golang.org/x/sync@v0.5.0/singleflight/singleflight.go:113 +0x204
  github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).Upstream()
      /go/pkg/mod/github.com/mohammed90/souin@v0.0.0-20240317145109-54af8511c28e/pkg/middleware/middleware.go:344 +0x3f9
  github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func4()
      /go/pkg/mod/github.com/mohammed90/souin@v0.0.0-20240317145109-54af8511c28e/pkg/middleware/middleware.go:709 +0x8e
  github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP.func10()
      /go/pkg/mod/github.com/mohammed90/souin@v0.0.0-20240317145109-54af8511c28e/pkg/middleware/middleware.go:710 +0x4f

Previous read at 0x00c00470cc00 by goroutine 3792:
  runtime.mapaccess2_faststr()
      /usr/local/go/src/runtime/map_faststr.go:108 +0x0
  github.com/caddyserver/caddy/v2.(*Replacer).fromStatic()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/replacer.go:97 +0x6d
  github.com/caddyserver/caddy/v2.(*Replacer).fromStatic-fm()
      <autogenerated>:1 +0x1f
  github.com/caddyserver/caddy/v2.(*Replacer).Get()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/replacer.go:75 +0x9b
  github.com/caddyserver/caddy/v2.(*Replacer).GetString()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/replacer.go:85 +0x3a
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).logRequest()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/server.go:717 +0x690
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP.func2()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/server.go:328 +0xc4
  runtime.deferreturn()
      /usr/local/go/src/runtime/panic.go:477 +0x30
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2938 +0x2a1
  net/http.(*conn).serve()
      /usr/local/go/src/net/http/server.go:2009 +0xc24
  net/http.(*Server).Serve.func3()
      /usr/local/go/src/net/http/server.go:3086 +0x4f

Goroutine 3795 (running) created at:
  github.com/darkweak/souin/pkg/middleware.(*SouinBaseHandler).ServeHTTP()
      /go/pkg/mod/github.com/mohammed90/souin@v0.0.0-20240317145109-54af8511c28e/pkg/middleware/middleware.go:708 +0x1e75
  github.com/caddyserver/cache-handler.(*SouinCaddyMiddleware).ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/cache-handler@v0.12.0/httpcache.go:84 +0x112
  github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:331 +0x81
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:300 +0x471
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/subroute.go:74 +0x96
  github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:331 +0x81
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:300 +0x471
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/subroute.go:74 +0x96
  github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:331 +0x81
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:300 +0x471
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:268 +0x3a1
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.Compile.wrapRoute.func1.1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/routes.go:268 +0x3a1
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).enforcementHandler()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/server.go:429 +0x208
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Provision.(*Server).wrapPrimaryRoute.func1()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/server.go:405 +0x68
  github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/caddyhttp.go:58 +0x47
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/server.go:341 +0x1598
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2938 +0x2a1
  net/http.(*conn).serve()
      /usr/local/go/src/net/http/server.go:2009 +0xc24
  net/http.(*Server).Serve.func3()
      /usr/local/go/src/net/http/server.go:3086 +0x4f

Goroutine 3792 (running) created at:
  net/http.(*Server).Serve()
      /usr/local/go/src/net/http/server.go:3086 +0x86c
  github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Start.func3()
      /go/pkg/mod/github.com/caddyserver/caddy/v2@v2.7.6/modules/caddyhttp/app.go:509 +0x4f
==================

It points at the Replacer. It might be the case on high traffic, the Set on replacer in the reverse proxy handling here is accessed concurrently and upsets the runtime:

caddy/modules/caddyhttp/reverseproxy/reverseproxy.go

Lines 430 to 434 in c37e16f

defer func() {
// total proxying duration, including time spent on LB and retries
repl.Set("http.reverse_proxy.duration", time.Since(start))
repl.Set("http.reverse_proxy.duration_ms", time.Since(start).Seconds()*1e3) // multiply seconds to preserve decimal (see #4666)
}()

This might have risen due to reverse-proxy, cache, and probably other handler that spinoff goroutines may indeed cause concurrent map access. This might've slipped through the cracks because most other handlers don't spinoff goroutines and the concurrent request just didn't happen to cross this threshold.

@mohammed90 mohammed90 requested a review from mholt March 21, 2024 12:50
@mohammed90 mohammed90 added bug 🐞 Something isn't working under review 🧐 Review is pending before merging labels Mar 21, 2024
@mohammed90 mohammed90 added this to the v2.8.0 milestone Mar 21, 2024
@mohammed90 mohammed90 marked this pull request as ready for review March 21, 2024 12:51
mholt
mholt approved these changes Mar 21, 2024
View reviewed changes
Copy link
Member

@mholt mholt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! Huh, I find it a little surprising the replacers are being written to concurrently like that, but since these are per-request I imagine this should be pretty non-invasive and lightweight.

If it works/fixes the race, let's go ahead and merge it 😃

@mohammed90 mohammed90 enabled auto-merge (squash) March 21, 2024 18:12
@mohammed90 mohammed90 merged commit e7336cc into master Mar 21, 2024
25 checks passed
@mohammed90 mohammed90 deleted the replacer-mutex branch March 21, 2024 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mholt mholt mholt approved these changes

Assignees
No one assigned
Labels
bug 🐞 Something isn't working under review 🧐 Review is pending before merging
Projects
None yet
Milestone
v2.8.0
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@mohammed90 @mholt