Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic slice out of bands when querying namespace metric #635

Open
pmalek opened this issue Jan 17, 2024 · 1 comment
Open

Panic slice out of bands when querying namespace metric #635

pmalek opened this issue Jan 17, 2024 · 1 comment
Assignees
@dgrisonnet
Labels
kind/bug Categorizes issue or PR as related to a bug. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@pmalek
Copy link

pmalek commented Jan 17, 2024

What happened?:

When playing around with config and querying a namespace metric I got a panic:

E0117 21:06:41.565151       1 wrap.go:58] "apiserver panic'd" method="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/kong/kong_upstream_latency_ms" auditID="8de84528-caec-4193-b41a-e146d2d871a1"
http2: panic serving 10.244.0.1:38584: runtime error: slice bounds out of range [2:1]
goroutine 1465 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1()
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/timeout.go:110 +0x9c
panic({0x1bdd200, 0x4002fd7560})
        /usr/local/go/src/runtime/panic.go:884 +0x1f4
sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/endpoints/handlers.getRequestOptions(_, {{0x210cbd0, 0x4005a3bab8}, {0x210c660, 0x4005a58380}, {0x20f5f80, 0x4005a58300}, {0x0, 0x0, 0x0}, ...}, ...)
        /go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.27.0/pkg/apiserver/endpoints/handlers/get.go:146 +0x3ac
sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/endpoints/handlers.ListResourceWithOptions.func1({0x210c8d0, 0x4005a6b100}, 0x4003566100)
        /go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.27.0/pkg/apiserver/endpoints/handlers/get.go:109 +0x4a8
sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer.restfulListResourceWithOptions.func1(0x4005a6b080, 0x40001ee930)
        /go/pkg/mod/sigs.k8s.io/custom-metrics-apiserver@v1.27.0/pkg/apiserver/installer/installer.go:280 +0x94
k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0x4005a6b080, 0x40001ee930)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/metrics/metrics.go:571 +0x1c8
github.com/emicklei/go-restful/v3.(*Container).dispatch(0x4005a24ab0, {0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/github.com/emicklei/go-restful/v3@v3.10.2/container.go:299 +0x470
github.com/emicklei/go-restful/v3.(*Container).Dispatch(...)
        /go/pkg/mod/github.com/emicklei/go-restful/v3@v3.10.2/container.go:204
k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x1d671da?, 0x116b334?}, 0x4005a24ab0?, 0x4000784b60?}, {0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/handler.go:146 +0x46c
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:110 +0x174
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210c8d0?, 0x4005a6af40?}, 0x4?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/authorization.go:64 +0x39c
net/http.HandlerFunc.ServeHTTP(0x433a3342b2?, {0x210c8d0?, 0x4005a6af40?}, 0x912280?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:84 +0x150
net/http.HandlerFunc.ServeHTTP(0x327c3e0?, {0x210c8d0?, 0x4005a6af40?}, 0x4?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/maxinflight.go:197 +0x22c
net/http.HandlerFunc.ServeHTTP(0x32dde98?, {0x210c8d0?, 0x4005a6af40?}, 0x12c50d4?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:110 +0x174
net/http.HandlerFunc.ServeHTTP(0x4007439208?, {0x210c8d0?, 0x4005a6af40?}, 0x163?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/impersonation.go:50 +0x1a4
net/http.HandlerFunc.ServeHTTP(0x20f2498?, {0x210c8d0?, 0x4005a6af40?}, 0xa?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:84 +0x150
net/http.HandlerFunc.ServeHTTP(0x32dde98?, {0x210c8d0?, 0x4005a6af40?}, 0x12c50d4?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:110 +0x174
net/http.HandlerFunc.ServeHTTP(0x32a7660?, {0x210c8d0?, 0x4005a6af40?}, 0xa?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:84 +0x150
net/http.HandlerFunc.ServeHTTP(0x32dde98?, {0x210c8d0?, 0x4005a6af40?}, 0x12c50d4?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:110 +0x174
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210c8d0?, 0x4005a6af40?}, 0x20dd120?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1({0x210c8d0, 0x4005a6af40}, 0x4003566100)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/authentication.go:105 +0x504
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210c8d0?, 0x4005a6af40?}, 0x20e1cc0?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x210c8d0, 0x4005a6af40}, 0x400354df00)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filterlatency/filterlatency.go:94 +0x2f4
net/http.HandlerFunc.ServeHTTP(0x4001378780?, {0x210c8d0?, 0x4005a6af40?}, 0x7cf54?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1()
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/timeout.go:115 +0x68
created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/timeout.go:101 +0x188

goroutine 1464 [running]:
golang.org/x/net/http2.(*serverConn).runHandler.func1()
        /go/pkg/mod/golang.org/x/net@v0.8.0/http2/server.go:2304 +0x138
panic({0x1903ca0, 0x4003535810})
        /usr/local/go/src/runtime/panic.go:884 +0x1f4
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x40038c8cd0, 0x1, 0x400736cc60?})
        /go/pkg/mod/k8s.io/apimachinery@v0.27.2/pkg/util/runtime/runtime.go:56 +0xe0
panic({0x1903ca0, 0x4003535810})
        /usr/local/go/src/runtime/panic.go:884 +0x1f4
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0x40046c8db0, {0x210c8d0, 0x4005a6ae80}, 0xdf8475800?)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/timeout.go:121 +0x2d4
k8s.io/apiserver/pkg/endpoints/filters.withRequestDeadline.func1({0x210c8d0, 0x4005a6ae80}, 0x400354dd00)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/request_deadline.go:100 +0x298
net/http.HandlerFunc.ServeHTTP(0xffff7db645d8?, {0x210c8d0?, 0x4005a6ae80?}, 0x1cc4c?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/server/filters.withWaitGroup.func1({0x210c8d0, 0x4005a6ae80}, 0x400354dd00)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/waitgroup.go:86 +0x18c
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210c8d0?, 0x4005a6ae80?}, 0x20e1cc0?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1({0x210c8d0?, 0x4005a6ae80}, 0x400354dc00)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/warning.go:35 +0x180
net/http.HandlerFunc.ServeHTTP(0x1b833a0?, {0x210c8d0?, 0x4005a6ae80?}, 0xd?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1({0x210c8d0, 0x4005a6ae80}, 0xffff7df188d0?)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/cachecontrol.go:31 +0x118
net/http.HandlerFunc.ServeHTTP(0x210e268?, {0x210c8d0?, 0x4005a6ae80?}, 0x20e1cc0?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/server/httplog.withLogging.func1({0x210cff0, 0x4005d16f40}, 0x400354db00)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/httplog/httplog.go:131 +0x28c
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x210cff0?, 0x4005d16f40?}, 0x20e1cc0?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP(0x40000e48f0, {0x210c8d0?, 0x4005a6ad40}, 0x400354d700)
        /go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.35.1/handler.go:204 +0xe6c
k8s.io/apiserver/pkg/endpoints/filters.WithLatencyTrackers.func1({0x20fed10?, 0x4003a7e028}, 0x400354d600)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/webhook_duration.go:57 +0x23c
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x20fed10?, 0x4003a7e028?}, 0x20e1cc0?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1({0x20fed10, 0x4003a7e028}, 0x400354d500)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/requestinfo.go:39 +0x1b8
net/http.HandlerFunc.ServeHTTP(0x210d5b8?, {0x20fed10?, 0x4003a7e028?}, 0x433a2e934c?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.withRequestReceivedTimestampWithClock.func1({0x20fed10, 0x4003a7e028}, 0x400354d400)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/request_received_time.go:38 +0x13c
net/http.HandlerFunc.ServeHTTP(0x4002c36ab8?, {0x20fed10?, 0x4003a7e028?}, 0xffffa54c0108?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.WithMuxAndDiscoveryComplete.func1({0x20fed10, 0x4003a7e028}, 0x400354d400)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/mux_discovery_complete.go:52 +0x180
net/http.HandlerFunc.ServeHTTP(0xffff7dbcece8?, {0x20fed10?, 0x4003a7e028?}, 0x4000844400?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1({0x20fed10?, 0x4003a7e028?}, 0x4002f9b101?)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/filters/wrap.go:74 +0x9c
net/http.HandlerFunc.ServeHTTP(0x1b833a0?, {0x20fed10?, 0x4003a7e028?}, 0x8?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/endpoints/filters.withAuditInit.func1({0x20fed10, 0x4003a7e028}, 0x400354d300)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/endpoints/filters/audit_init.go:63 +0x240
net/http.HandlerFunc.ServeHTTP(0x1d4a967?, {0x20fed10?, 0x4003a7e028?}, 0x3?)
        /usr/local/go/src/net/http/server.go:2122 +0x38
k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0x0?, {0x20fed10?, 0x4003a7e028?}, 0x1c7fc?)
        /go/pkg/mod/k8s.io/apiserver@v0.27.2/pkg/server/handler.go:189 +0x30
net/http.serverHandler.ServeHTTP({0x4000058058?}, {0x20fed10, 0x4003a7e028}, 0x400354d300)
        /usr/local/go/src/net/http/server.go:2936 +0x2c0
net/http.initALPNRequest.ServeHTTP({{0x210d5b8?, 0x4007340840?}, 0x4000a14a80?, {0x400618e0f0?}}, {0x20fed10, 0x4003a7e028}, 0x400354d300)
        /usr/local/go/src/net/http/server.go:3545 +0x1d0
golang.org/x/net/http2.(*serverConn).runHandler(0x0?, 0x7cf54?, 0x4007258c00?, 0x0?)
        /go/pkg/mod/golang.org/x/net@v0.8.0/http2/server.go:2311 +0x78
created by golang.org/x/net/http2.(*serverConn).processHeaders
        /go/pkg/mod/golang.org/x/net@v0.8.0/http2/server.go:2025 +0x58c
I0117 21:06:42.704196       1 httplog.go:132] "HTTP" verb="GET" URI="/healthz" latency="143.919µs" userAgent="kube-probe/1.28" audit-ID="7f1ca763-2e93-4173-a77e-9abbb5ff9af2" srcIP="10.244.0.1:41000" resp=200
I0117 21:06:42.704485       1 httplog.go:132] "HTTP" verb="GET" URI="/healthz" latency="67.584µs" userAgent="kube-probe/1.28" audit-ID="84404544-fe98-4b6e-bd6b-49b747cb8af2" srcIP="10.244.0.1:40990" resp=200
I0117 21:06:45.829907       1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/kong/services/%2A/kong_upstream_latency_ms" latency="4.124525ms" userAgent="kubectl/v1.29.0 (darwin/arm64) kubernetes/3f7a50f" audit-ID="2bb83135-de14-4236-9b5c-e512d11625d1" srcIP="10.244.0.1:38584" resp=404
I0117 21:06:48.453310       1 httplog.go:132] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1/namespaces/kong/pods/%2A/kong_upstream_latency_ms" latency="7.282783ms" userAgent="kubectl/v1.29.0 (darwin/arm64) kubernetes/3f7a50f" audit-ID="68f2e92d-dac0-4a71-b7bf-84be159f4c12" srcIP="10.244.0.1:38584" resp=404

What did you expect to happen?:

No panic.

Please provide the prometheus-adapter config:

prometheus-adapter config 
prometheus:
  url: http://prometheus-kube-prometheus-prometheus.kong.svc

rules:
  custom:
  - seriesQuery: '{__name__=~"^kong_upstream_latency_ms_"}'
    resources:
      template: "<<.Resource>>."
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      as: "kong_upstream_latency_ms"
    metricsQuery: |-
      sum(rate(kong_upstream_latency_ms_sum{route=~".*httproute.*"}[5m])) by (exported_service)
      /
      sum(rate(kong_upstream_latency_ms_count{route=~".*httproute.*"}[5m])) by (exported_service)

Anything else we need to know?:

Environment:

  • prometheus-adapter version: registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.11.2
  • prometheus version: 2.48.1
  • Kubernetes version (use kubectl version): 1.28.0
@pmalek pmalek added the kind/bug Categorizes issue or PR as related to a bug. label Jan 17, 2024
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jan 17, 2024
@dgrisonnet
Copy link
Member

/triage accepted
/assign

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dgrisonnet dgrisonnet

Labels
kind/bug Categorizes issue or PR as related to a bug. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pmalek @k8s-ci-robot @dgrisonnet