Trace (or span) ID collisions occurring in multiprocessing

[rkdls]

Description

In the latest version of the SDK (v1.36.0), there appears to be a potential issue with trace ID collisions. In a multiprocessing environment, trace IDs are being generated simultaneously and sometimes end up duplicated.

I'm using Apache httpd bridged with Golang, which utilizes multiple worker processes. It seems like these workers might be sharing the same random seed, leading to trace ID collisions.

When using v1.35.0, this issue does not occur and trace ID generation is stable.

related PR: #6732

Thanks in advance

Environment

• OS: Linux
• Architecture: x86
• Go Version: 1.22.5
• opentelemetry-go version: v1.36.0 (https://github.com/open-telemetry/opentelemetry-go/blob/v1.36.0/sdk/trace/id_generator.go)

Steps To Reproduce

Expected behavior

A clear and concise description of what you expected to happen.

[dmathieu]

cc @seankhliao

[seankhliao]

do you have a reproducer? and examples of duplicate trace IDs?

[pellared]

there appears to be a potential issue with trace ID collisions.

It seems like these workers might be sharing the same random seed, leading to trace ID collisions.

Where?

From https://pkg.go.dev/math/rand/v2:

Top-level functions, such as Float64 and Int, are safe for concurrent use by multiple goroutines.

[rkdls]

It seems like these workers might be sharing the same random seed, leading to trace ID collisions.

Come to think of it, It's not clear whether random seeds are shared. But It definitely happens.

Let me explain the reproduction steps in detail:

We are building a Go module using cgo with -buildmode=c-shared to be used as a dynamic library in an Apache HTTPD module. Apache HTTPD uses a multi-process worker model (via fork()), and each worker process loads the Go shared library.

When running under Apache with multiple worker processes, we observe that trace IDs generated by the SDK are sometimes duplicated. This suggests that the random number generator used to create trace IDs may be initialized with the same seed in each forked process — although we haven't confirmed the exact cause yet.

Interestingly, this issue does not occur in SDK version v1.35.0, but it does occur in v1.36.0, which implies that something may have changed internally regarding random number generation or trace ID creation in the newer version.

(I previously mentioned that using Go version 1.23.0, but after double-checking, I realized we are actually using Go 1.22.5.)

[seankhliao]

I think we really need to see an example of how this is built.

How can you be on v1.36.0 using go1.22.5 ?
The module specifies a minimum of go1.23.0 https://github.com/open-telemetry/opentelemetry-go/blob/v1.36.0/sdk/go.mod#L3

And does this reproduce using go1.24?

[rkdls]

I test after updated go version to 1.24.4, it is still happens.

In a same trace, duplicated "DeleteSearcher" is exists

In trace.

Duplicated span is exists.

Spans inform

I guess the last span is included because of the overlapping trace IDs.

[pellared]

If you are unable to provide a Minimal, Reproducible Example can you maybe please share with us:

1. how many trace were produced before a collision happened (what is the ratio of "good" traces vs traces with collision)?
2. has the collision occurred on the same container or was it a separate one (is the container.id different)?

[seankhliao]

you've managed to omit any useful information that would assist in debugging.
what generates the original trace ID? apache? or Go? or something else?
If it was a duplicate trace ID being generated, I'd expect to see 2 unrelated traces merged together. Instead you're showing just a single span.

go runtime seeding uses the AT_RANDOM aux vector https://cs.opensource.google/go/go/+/master:src/runtime/rand.go;l=45?q=startupRand&ss=go%2Fgo