[Bug]: "HTTP handler, Internal Server Error","error":"search services failed.."

[mk-raven]

What happened?

When want to get traces, there was 504 error code.
All worked fine before upgrade jaeger with the same version of Elastic Search.

Steps to reproduce

Just install the new version of Jaeger stack.

Expected behavior

Jaeger UI should work as expected.

jaeger collector config:

es:
  version: 7
  server-urls: "https://localhost:9200"
  username: ""
  password: ""
  index-date-separator: "."
  create-index-templates: false
  bulk:
    size: 8000
    workers: 6
  tags-as-fields:
    all: true
  tls:
    enabled: true
    skip-host-verify: true

jaeger query config:

log-level: debug

admin:
 http:
   host-port: 127.0.0.1:16687
query:
 grpc-server:
   host-port: 127.0.0.1:16685
 http-server:
   host-port: 127.0.0.1:16686


es:
 server-urls: "https://localhost:9200"
 username: ""
 password: ""
 index-date-separator: "."
 tls:
   enabled: true
   skip-host-verify: true
 bulk:
   workers: 4

Relevant log output

jaeger-elk_deprecation.log
[CRITICAL][o.e.d.i.m.T.TypeFieldType] [jaeger-elk] [types removal] Using the _type field in queries and aggregations is deprecated, prefer to use a field instead.

 jaeger-query[1952612]: {"level":"error","ts":1713781124.9002626,"caller":"app/http_handler.go:504","msg":"HTTP handler, Internal Server Error","error":"search services failed: Post \"https://localhost:9200/jaeger-span-2024.04.22/_search?ignore_unavailable=true&rest_total_hits_as_int=true\": context canceled","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:504\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).search\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:259\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.traceResponseHandler.func2\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:548\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.WithRouteTag.func3\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.49.0/handler.go:281\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.49.0/handler.go:225\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.49.0/handler.go:83\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/mux@v1.8.1/mux.go:212\ngithub.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.additionalHeadersHandler.func4\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\ngithub.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.CompressHandler.CompressHandlerLevel.func6\n\tgithub.com/gorilla/handlers@v1.5.1/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2166\ngithub.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\tgithub.com/gorilla/handlers@v1.5.1/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:3137\nnet/http.(*conn).serve\n\tnet/http/server.go:2039"}

Screenshot

Additional context

No response

Jaeger backend version

v1.56.0

SDK

No response

Pipeline

OTEL sdk -> OTEL collector -> Jaeger collector -> Elastic Search <- Jaeger Query -> Jaeger UI

Stogage backend

Elastic Search 7.17.14

Operating system

Linux

Deployment model

binaries on bare metal

Deployment configs

Ansible roles

[jkowall]

Did you upgrade all components, it seems like you are using an older version of jaeger-query still. Did you try to point a new instance of query and UI to the backend?