[Bug]: Traces are not found when searching from the query UI

[TheOxFromOutOfTheBox]

What happened?

I am able to see the services listed in the UI but on searching it returns "No Traces Found". Elasticsearch records storing of spans, and it is also visible from Jaeger metrics endpoint. On using the query mentioned in #2816 by @albertteoh , a valid trace ID is returned.

Steps to reproduce

1. Start ES container
2. Start Jaeger container
3. Send traces having multiple services from a sample application
4. Search for recent traces.

Expected behavior

Relevant Traces should appear.

Relevant log output

ES- Response
{"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":null,"hits":[]},"aggregations":{"traceIDs":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"46957fde391023b1a7a22eb181ad3cbe","doc_count":1,"startTime":{"value":1.70608619842243E15}}]}}}

Response to index query:

health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   jaeger-span-2024-01-24    DemlfKN-RNOmUswmFL2lpQ   5   1       4396            0    429.3kb        429.3kb
yellow open   jaeger-service-2024-01-24 Dgu_VCkNT0qvNCj9oUx1YA   5   1        312            0     69.2kb         69.2kb

Jaeger metrics
# TYPE jaeger_spans_saved_by_svc_total counter
jaeger_spans_saved_by_svc_total{debug="false",result="err",svc="other-services"} 0
jaeger_spans_saved_by_svc_total{debug="false",result="ok",svc="algo-service"} 100
jaeger_spans_saved_by_svc_total{debug="false",result="ok",svc="jaeger-query"} 141
jaeger_spans_saved_by_svc_total{debug="false",result="ok",svc="job-service"} 1
jaeger_spans_saved_by_svc_total{debug="false",result="ok",svc="other-services"} 0
jaeger_spans_saved_by_svc_total{debug="false",result="ok",svc="pc-service"} 100
jaeger_spans_saved_by_svc_total{debug="false",result="ok",svc="site-service"} 100
jaeger_spans_saved_by_svc_total{debug="true",result="err",svc="other-services"} 0
jaeger_spans_saved_by_svc_total{debug="true",result="ok",svc="other-services"} 0

Screenshot

Additional context

No response

Jaeger backend version

v1.47

SDK

OpenTelemetry C++ SDK

Pipeline

OTEL SDK -> OTel Collector -> Jaeger -> Elasticsearch

Stogage backend

Elasticsearch v7.17.10

Operating system

Linux

Deployment model

No response

Deployment configs

No response

[TheOxFromOutOfTheBox]

querying elastic with both queries in #2816 returns responses, but using the traceID in the response via Jaeger UI yields no results.

[jkowall]

The service pulldown is populated so there is data there to populate some of the UI. Are the trace timestamps correct? Do you have Kibana or OpenSearch Dashboards connected to the backend so you can query?

Can you share the output of the query?

[jkowall]

Please re-open this when you reply to my questions.