all-in-one is extremely slow #135
Description
Requirement - what kind of business use case are you trying to solve?
work with jaeger locally and in Kubernetes
Problem - what in Jaeger blocks you from solving the requirement?
using the all-in-one deployment locally in docker-compose and also in k8s cluster is extremely slow.
i have 5 NodeJS micro-services instrumented with opentracing and jaeger reporting spans to jaeger all-in-one. a simple http request which results in ~100 spans takes almost 90 sec. when tracing is turned off (by using no-op tracer), the request ends in 1-2 secs.
i assume the instrumentation code is not to blame but the actual spans processing and sending to jaeger backend.
looking on traces, i see that outgoing requests from any micro services seems to take quite long, ~around 10-15 sec with no explanation. using a UDP port (6832) AFAIK should not block or cause such hangs.
i don't find any instructions how to debug the issue.
I've tried to add grafana and Prometheus to my compose to metrics but i an;'t find a proper dashboard, nor i don't see any metrics scraping happening. (I've added a task for all-in-one:14269 as described here: https://www.jaegertracing.io/docs/1.17/monitoring
it would be great to have a complete example that includes all-in-one Jaeger + grafana + (with pre-configured dashboard) Prometheus to allow easy Jaeger monitoring/debugging.
AFAIK, Jaeger is built for huge spans processing, and i can't see why all-in-one can't process ~100 spans for a single request (no other requests happens, besides a few /health traces, but i assume they are negligible)
local env:
OS: Windows_NT x64 10.0.18363
Node.js: 12.16.1
"jaeger-client": "^3.18.0",
"opentracing": "^0.14.4",
jaeger all-in-one (latest, i assume 1.17)
docker-compose version 1.25.4, build 8d51620a
Docker version 19.03.8, build afacb8b
slowness happens also in our k8s cluster where i've deployed the all-in-one image
can't use tracing as it extremely slow