Bug: Mimir 3.0.2 with MQE query‑frontend/querier warnings/errors about missing query IDs

[sirishkumar]

What is the bug?

I have updated from mimir 2.17.1  to 3.0.2 with classic architecture using grafana helm chart . I do see following warnings and errors in frontend and querier logs appearing at a very high frequency almost for every query from Grafana. But i do not see any errors in Grafana and can see that dashboards/explorer are updated , which i assume indicates that queries to mimir have succeeded.

Frontend logs:

mimir-query-frontend-75d7dcf96b-xqm5k query-frontend ts=2026-01-09T08:38:05.590621613Z caller=grpc_logging.go:76 level=warn method=/frontendv2pb.FrontendForQuerier/QueryResult duration=13.399µs msg=gRPC err="query 7466090012873038560 not found or response already received"
mimir-query-frontend-75d7dcf96b-xqm5k query-frontend ts=2026-01-09T08:38:05.611355213Z caller=grpc_logging.go:76 level=warn method=/frontendv2pb.FrontendForQuerier/QueryResult duration=21.612µs msg=gRPC err="query 7466090012873038560 not found or response already received"
mimir-query-frontend-75d7dcf96b-78467 query-frontend ts=2026-01-09T08:39:05.605208732Z caller=grpc_logging.go:76 level=warn method=/frontendv2pb.FrontendForQuerier/QueryResult duration=13.341µs msg=gRPC err="query 6565548320877864354 not found or response already received"
mimir-query-frontend-75d7dcf96b-xqm5k query-frontend ts=2026-01-09T08:38:05.633023212Z caller=grpc_logging.go:76 level=warn method=/frontendv2pb.FrontendForQuerier/QueryResult duration=39.803µs msg=gRPC err="query 7466090012873038560 not found or response already received"
mimir-query-frontend-75d7dcf96b-xqm5k query-frontend ts=2026-01-09T08:38:05.667477225Z caller=grpc_logging.go:76 level=warn method=/frontendv2pb.FrontendForQuerier/QueryResult duration=22.982µs msg=gRPC err="query 7466090012873038560 not found or response already received"
mimir-query-frontend-75d7dcf96b-78467 query-frontend ts=2026-01-09T08:39:05.605349104Z caller=grpc_logging.go:76 level=warn method=/frontendv2pb.FrontendForQuerier/QueryResult duration=11.216µs msg=gRPC err="query 6565548320877864352 not found or response already received"
mimir-query-frontend-75d7dcf96b-xqm5k query-frontend ts=2026-01-09T08:38:05.732831469Z caller=grpc_logging.go:76 level=warn method=/frontendv2pb.FrontendForQuerier/QueryResult duration=21.866µs msg=gRPC err="query 7466090012873038560 not found or response already received"

Querier Logs:

mimir-querier-584987ff55-4zqpl querier ts=2026-01-09T08:41:05.7464108Z caller=scheduler_processor.go:298 level=error user=bbsupport-test msg="error notifying scheduler about finished query after the scheduler stream previously failed and returned error" err=EOF addr=[2a03:1e84:1902:14:1fc1::a]:9095 previousErr="rpc error: code = Canceled desc = context canceled: frontend cancelled query"
mimir-querier-584987ff55-4zqpl querier ts=2026-01-09T08:41:05.754991393Z caller=scheduler_processor.go:387 level=warn user=bbsupport-test msg="retrying to notify frontend about finished query" err="rpc error: code = Unknown desc = query 6565548320877864830 not found or response already received" frontend=[2a03:1e84:1902:14:ee2b::]:9095 retries=4 query_id=6565548320877864830
mimir-querier-584987ff55-4zqpl querier ts=2026-01-09T08:41:05.75503053Z caller=scheduler_processor.go:393 level=error user=bbsupport-test msg="error notifying frontend about finished query" err="rpc error: code = Unknown desc = query 6565548320877864830 not found or response already received" frontend=[2a03:1e84:1902:14:ee2b::]:9095 query_id=6565548320877864830
mimir-querier-584987ff55-4zqpl querier ts=2026-01-09T08:41:05.755050246Z caller=scheduler_processor.go:298 level=error user=bbsupport-test msg="error notifying scheduler about finished query after the scheduler stream previously failed and returned error" err=EOF addr=[2a03:1e84:1902:14:1fc1::a]:9095 previousErr="rpc error: code = Canceled desc = context canceled: frontend cancelled query"
mimir-querier-584987ff55-4zqpl querier ts=2026-01-09T08:41:05.755059739Z caller=scheduler_processor.go:387 level=warn user=bbsupport-test msg="retrying to notify frontend about finished query" err="rpc error: code = Unknown desc = query 6565548320877864828 not found or response already received" frontend=[2a03:1e84:1902:14:ee2b::]:9095 retries=4 query_id=6565548320877864828
mimir-querier-584987ff55-4zqpl querier ts=2026-01-09T08:41:05.755087611Z caller=scheduler_processor.go:393 level=error user=bbsupport-test msg="error notifying frontend about finished query" err="rpc error: code = Unknown desc = query 6565548320877864828 not found or response already received" frontend=[2a03:1e84:1902:14:ee2b::]:9095 query_id=6565548320877864828
mimir-querier-584987ff55-4zqpl querier ts=2026-01-09T08:41:05.75510357Z caller=scheduler_processor.go:298 level=error user=bbsupport-test msg="error notifying scheduler about finished query after the scheduler stream previously failed and returned error" err=EOF addr=[2a03:1e84:1902:14:1fc1::a]:9095 previousErr="rpc error: code = Canceled desc = context canceled: frontend cancelled query"
mimir-querier-584987ff55-cv8rn querier ts=2026-01-09T08:41:05.594535645Z caller=scheduler_processor.go:387 level=warn user=bbsupport-test msg="retrying to notify frontend about finished query" err="rpc error: code = Unknown desc = query 6565548320877864831 not found or response already received" frontend=[2a03:1e84:1902:14:ee2b::]:9095 retries=0 query_id=6565548320877864831
mimir-querier-584987ff55-cv8rn querier ts=2026-01-09T08:41:05.629319678Z caller=scheduler_processor.go:387 level=warn user=bbsupport-test msg="retrying to notify frontend about finished query" err="rpc error: code = Unknown desc = query 6565548320877864831 not found or response already received" frontend=[2a03:1e84:1902:14:ee2b::]:9095 retries=1 query_id=6565548320877864831
mimir-querier-584987ff55-cv8rn querier ts=2026-01-09T08:41:05.65281876Z caller=scheduler_processor.go:387 level=warn user=bbsupport-test msg="retrying to notify frontend about finished query" err="rpc error: code = Unknown desc = query 6565548320877864831 not found or response already received" frontend=[2a03:1e84:1902:14:ee2b::]:9095 retries=2 query_id=6565548320877864831
mimir-querier-584987ff55-cv8rn querier ts=2026-01-09T08:41:05.696734549Z caller=scheduler_processor.go:387 level=warn user=bbsupport-test msg="retrying to notify frontend about finished query" err="rpc error: code = Unknown desc = query 6565548320877864831 not found or response already received" frontend=[2a03:1e84:1902:14:ee2b::]:9095 retries=3 query_id=6565548320877864831
mimir-querier-584987ff55-cv8rn querier ts=2026-01-09T08:41:05.765597203Z caller=scheduler_processor.go:387 level=warn user=bbsupport-test msg="retrying to notify frontend about finished query" err="rpc error: code = Unknown desc = query 6565548320877864831 not found or response already received" frontend=[2a03:1e84:1902:14:ee2b::]:9095 retries=4 query_id=6565548320877864831
mimir-querier-584987ff55-cv8rn querier ts=2026-01-09T08:41:05.765642797Z caller=scheduler_processor.go:393 level=error user=bbsupport-test msg="error notifying frontend about finished query" err="rpc error: code = Unknown desc = query 6565548320877864831 not found or response already received" frontend=[2a03:1e84:1902:14:ee2b::]:9095 query_id=6565548320877864831
mimir-querier-584987ff55-cv8rn querier ts=2026-01-09T08:41:05.765665246Z caller=scheduler_processor.go:298 level=error user=bbsupport-test msg="error notifying scheduler about finished query after the scheduler stream previously failed and returned error" err=EOF addr=[2a03:1e84:1902:14:1fc1::a]:9095 previousErr="rpc error: code = Canceled desc = context canceled: frontend cancelled query"

What I tried

• Change query_engine to prometheus in frontend, these errors no longer appear.
• Setting timeout explicitly on querier to 2 minutes, errors still persist.
• Enabling use_mimir_query_engine_for_sharding and also its dependent config enable_remote_execution,  query-frontend keeps crashing with OOM and i see almost even with 5x times memory before enabling this setting.

How to reproduce it?

Following usecases reproduce this issue

• Go to drilldown metrics window from Grafana.
• Run a query in explorer window, switch time range few times
• Go to any dashboard

What did you think would happen?

Timeout in frontend is set to 2m explicitly, from the logs atleast i do not see the queries lasting more than few milli seconds. These warnings and errors ideally should not appear.

What was your environment?

• Kubernetes 1.33.5 EKS
• Helm chart version 6.0.5 and mimir 3.0.2 running in IPv6 mode in classic architecture mode without ingest storage

Any additional context to share?

No response