[processor/routing] Collector crash #26462
Assignees
Labels
Comments
jpkrohling
added
bug
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
jpkrohling
added
processor/routing
jpkrohling
changed the title
jpkrohling
added a commit
to jpkrohling/opentelemetry-collector-contrib
that referenced
this issue
Sep 5, 2023
This PR changes the routing processor so that it properly creates new instances of pdata.Span and InstrumentationScope, avoiding crashes later on due to nil pointers. Fixes open-telemetry#26462 Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
This was referenced Sep 5, 2023
jpkrohling
added a commit
that referenced
this issue
Sep 6, 2023
This PR changes the routing processor so that it properly creates new instances of pdata.Span and InstrumentationScope, avoiding crashes later on due to nil pointers. Fixes #26462 Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de> --------- Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
This was referenced Sep 12, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Component(s)
processor/routing
What happened?
Description
Collector crash, caused by misuse of the ptrace.Span construct by the routing processor.
Steps to Reproduce
telemetrygen traces --traces 10000 --otlp-insecure --otlp-attributes='attr1="demo",attr2="xxx"'Expected Result
No crashes.
Actual Result
Collector version
0.84.0
Environment information
Environment
OS: Fedora release 38 (Thirty Eight)
OpenTelemetry Collector configuration
Log output
2023-09-05T07:02:35.045-0300 info service/telemetry.go:84 Setting up own telemetry... 2023-09-05T07:02:35.045-0300 info service/telemetry.go:201 Serving Prometheus metrics {"address": ":8888", "level": "Basic"} 2023-09-05T07:02:35.045-0300 info exporter@v0.84.0/exporter.go:275 Development component. May change in the future. {"kind": "exporter", "data_type": "traces", "name": "logging/route-2"} 2023-09-05T07:02:35.045-0300 info exporter@v0.84.0/exporter.go:275 Development component. May change in the future. {"kind": "exporter", "data_type": "traces", "name": "logging"} 2023-09-05T07:02:35.045-0300 info exporter@v0.84.0/exporter.go:275 Development component. May change in the future. {"kind": "exporter", "data_type": "traces", "name": "logging/route-1"} 2023-09-05T07:02:35.047-0300 info service/service.go:138 Starting otelcol-contrib... {"Version": "0.84.0", "NumCPU": 16} 2023-09-05T07:02:35.047-0300 info extensions/extensions.go:31 Starting extensions... 2023-09-05T07:02:35.048-0300 warn internal@v0.84.0/warning.go:40 Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks {"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"} 2023-09-05T07:02:35.048-0300 info otlpreceiver@v0.84.0/otlp.go:83 Starting GRPC server {"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4317"} 2023-09-05T07:02:35.048-0300 info service/service.go:161 Everything is ready. Begin running and processing data. panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb1d720] goroutine 196 [running]: go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.func1() go.opentelemetry.io/otel/sdk@v1.16.0/trace/span.go:383 +0x25 go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc002e7a180, {0x0, 0x0, 0xd5c01e0?}) go.opentelemetry.io/otel/sdk@v1.16.0/trace/span.go:415 +0x975 panic({0x698d520?, 0xd52b350?}) runtime/panic.go:914 +0x21f go.opentelemetry.io/collector/pdata/ptrace.Span.Attributes({0x0?}) go.opentelemetry.io/collector/pdata@v1.0.0-rcv0014/ptrace/generated_span.go:124 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/internal.SpanPathGetSetter[...].func31({{0x0?}, {0x0?}, {0xc002991278?}, {0xc002b00d20?}}) github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl@v0.84.0/contexts/internal/span.go:408 +0x5b github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl.StandardGetSetter[...].Get(...) github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl@v0.84.0/expression.go:49 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl.(*Parser[...]).newComparisonEvaluator.func2({{0x0}, {0x0}, {0xc002991278?}, {0xc002b00d20?}}) github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl@v0.84.0/boolean_value.go:87 +0xcd github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl.BoolExpr[...].Eval(0xc001114fa0?, {0x88e5f18?, 0xc001d90ea0?}, {{0x0?}, {0x0?}, {0xc002991278?}, {0xc002b00d20?}}) github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl@v0.84.0/boolean_value.go:19 +0x47 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl.(*Parser[...].func2({{0x0}, {0x0}, {0xc002991278?}, {0xc002b00d20?}}) github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl@v0.84.0/boolean_value.go:43 +0xec github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl.BoolExpr[...].Eval(0x1?, {0x88e5f18?, 0xc001d90ea0?}, {{0x0?}, {0x0?}, {0xc002991278?}, {0xc002b00d20?}}) github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl@v0.84.0/boolean_value.go:19 +0x47 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl.(*Parser[...].func2({{0x0}, {0x0}, {0xc002991278?}, {0xc002b00d20?}}) github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl@v0.84.0/boolean_value.go:60 +0xec github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl.BoolExpr[...].Eval(...) github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl@v0.84.0/boolean_value.go:19 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl.(*Statement[...]).Execute(0xc002e87120, {0x88e5f18?, 0xc001d90ea0}, {{0x0}, {0x0?}, {0xc002991278?}, {0xc002b00d20?}}) github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl@v0.84.0/parser.go:55 +0x76 github.com/open-telemetry/opentelemetry-collector-contrib/processor/routingprocessor.(*tracesProcessor).route(0xc002932460, {0x88e5f18, 0xc001d90ea0}, {0x7a7910f?}) github.com/open-telemetry/opentelemetry-collector-contrib/processor/routingprocessor@v0.84.0/traces.go:116 +0x3db github.com/open-telemetry/opentelemetry-collector-contrib/processor/routingprocessor.(*tracesProcessor).ConsumeTraces(0x0?, {0x88e5f18?, 0xc001d90ea0?}, {0x6992e00?}) github.com/open-telemetry/opentelemetry-collector-contrib/processor/routingprocessor@v0.84.0/traces.go:81 +0x28 go.opentelemetry.io/collector/consumer.ConsumeTracesFunc.ConsumeTraces(...) go.opentelemetry.io/collector/consumer@v0.84.0/traces.go:25 go.opentelemetry.io/collector/receiver/otlpreceiver/internal/trace.(*Receiver).Export(0xc002925470, {0x88e5f18, 0xc001d90e10}, {0x0?}) go.opentelemetry.io/collector/receiver/otlpreceiver@v0.84.0/internal/trace/otlp.go:41 +0xc5 go.opentelemetry.io/collector/pdata/ptrace/ptraceotlp.rawTracesServer.Export({{0x889a548?, 0xc002925470?}}, {0x88e5f18?, 0xc001d90e10?}, 0xc002e7a100?) go.opentelemetry.io/collector/pdata@v1.0.0-rcv0014/ptrace/ptraceotlp/grpc.go:83 +0xb0 go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler.func1({0x88e5f18, 0xc001d90e10}, {0x769a460?, 0xc002b00a08}) go.opentelemetry.io/collector/pdata@v1.0.0-rcv0014/internal/data/protogen/collector/trace/v1/trace_service.pb.go:310 +0x72 go.opentelemetry.io/collector/config/configgrpc.(*GRPCServerSettings).toServerOption.enhanceWithClientInformation.func9({0x88e5f18?, 0xc001d90db0?}, {0x769a460, 0xc002b00a08}, 0xa?, 0xc002b00cd8) go.opentelemetry.io/collector/config/configgrpc@v0.84.0/configgrpc.go:400 +0x46 google.golang.org/grpc.getChainUnaryHandler.func1({0x88e5f18, 0xc001d90db0}, {0x769a460, 0xc002b00a08}) google.golang.org/grpc@v1.57.0/server.go:1179 +0xb2 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x88e5f18, 0xc001d907e0}, {0x769a460, 0xc002b00a08}, 0xc002e692a0, 0xc002e73100) go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.42.1-0.20230612162650-64be7e574a17/interceptor.go:344 +0x52b google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x88e5f18, 0xc001d907e0}, {0x769a460, 0xc002b00a08}, 0xc002088a50?, 0x698d9a0?) google.golang.org/grpc@v1.57.0/server.go:1170 +0x85 go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler({0x683fd60?, 0xc00293d060}, {0x88e5f18, 0xc001d907e0}, 0xc001faf180, 0xc00292d4a0) go.opentelemetry.io/collector/pdata@v1.0.0-rcv0014/internal/data/protogen/collector/trace/v1/trace_service.pb.go:312 +0x135 google.golang.org/grpc.(*Server).processUnaryRPC(0xc00293a1e0, {0x8916aa0, 0xc0027009c0}, 0xc002923320, 0xc002966d50, 0xd539870, 0x0) google.golang.org/grpc@v1.57.0/server.go:1360 +0xe15 google.golang.org/grpc.(*Server).handleStream(0xc00293a1e0, {0x8916aa0, 0xc0027009c0}, 0xc002923320, 0x0) google.golang.org/grpc@v1.57.0/server.go:1737 +0x9e7 google.golang.org/grpc.(*Server).serveStreams.func1.1() google.golang.org/grpc@v1.57.0/server.go:982 +0x8d created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 195 google.golang.org/grpc@v1.57.0/server.go:980 +0x165Additional context
Logs from telemetrygen:
The text was updated successfully, but these errors were encountered: