[Bug]: Trace ids are no longer cascaded on next spans after Tracer::extract(Format::TEXT_MAP) & using context on startxSpan

[mrtus]

Bug report

👋

We use some custom tooling to facilitate better insights through AMQP processes where the resource types are the type of messages, rather than e.g. basic_deliver

After the upgrade the given trace ids are no longer cascaded onto next spans after $spanContext = Tracer::extract(Format::TEXT_MAP, [...]) and use them on Trager::startActiveSpan('operation', ["references" => Reference::create(Reference::CHILD_OF, $spanContext); We already tried startRootSpan as an alternative too.
The given span context is created with ::inject(TEXT_MAP)

However the resources are still being registered in the service catalog, but there are no traces at all anymore.
We don't seem to understand what changed, and which changes we should make to resolve those issues again.

If we were to publish a message without tracing context to be used, it does create the traces correctly again.

And when debugging a bit more it seems that there is a new trace id created and the given one is being ignored

Implementation details

use DDTrace\Contracts\Scope;
use DDTrace\Contracts\Tracer;
use DDTrace\Exceptions\UnsupportedFormat;
use DDTrace\Format;
use DDTrace\Reference;
use DDTrace\Tag;
use Throwable;

final class TracingConsumerMiddleware implements ConsumerMiddleware
{
    private const SERVICE_TYPE = 'AMQP';
    private const OPERATION_NAME = 'Consumer.listen';
    private const PROPERTY_DATADOG_SPAN = 'dd_span';

    public function __construct(
        private readonly Tracer $tracer,
        private readonly string $serviceName,
    ) {
    }

    public function handle(Message $message, ConsumerHandler $handler): Status
    {
        $scope = $this->startActiveSpan($message);

        $span = $scope->getSpan();

        $span->setTag(Tag::SPAN_TYPE, self::SERVICE_TYPE);
        $span->setTag(Tag::SERVICE_NAME, $this->serviceName);

        $type = $this->determineTypeFromMessage($message);

        if ($type !== null) {
            $span->setResource($type);
        }

        try {
            return $handler->handle($message);
        } catch (Throwable $exception) {
            $span->setError($exception);

            throw $exception;
        } finally {
            $scope->close();

            $this->tracer->flush();
        }
    }

    private function determineTypeFromMessage(Message $message): ?string
    {
        if ($message->getProperty('type')) {
            return $message->getProperty('type');
        }

        $payload = json_decode($message->getBody(), true);

        if (isset($payload['type'])) {
            return $payload['type'];
        }

        return null;
    }

    private function startActiveSpan(Message $message): Scope
    {
        $propertyDataDogSpan = $message->getProperty(self::PROPERTY_DATADOG_SPAN);

        if ($propertyDataDogSpan === null) {
            return $this->tracer->startActiveSpan(self::OPERATION_NAME);
        }

        $context = json_decode($propertyDataDogSpan, true, 512, JSON_THROW_ON_ERROR);

        try {
            $spanContext = $this->tracer->extract(Format::TEXT_MAP, $context);
        } catch (UnsupportedFormat) {
            return $this->tracer->startActiveSpan(self::OPERATION_NAME);
        }

        if ($spanContext === null) {
            return $this->tracer->startActiveSpan(self::OPERATION_NAME);
        }

        return $this->tracer->startActiveSpan(
            self::OPERATION_NAME,
            [
                'references' => Reference::create(Reference::CHILD_OF, $spanContext),
            ],
        );
    }
}

Trace Debug

Would you be able to give us any pointers on what can adjust?

This same code structure did work for us pre-1.x.

Thank you!

PHP version

8.1.x

Tracer or profiler version

1.1.0

Installed extensions

ddtrace is on version 1.1.0

[PHP Modules]
apcu
bcmath
bz2
calendar
Core
ctype
curl
datadog-profiling
date
ddappsec
ddtrace
dom
exif
fileinfo
filter
ftp
gd
gettext
gmp
hash
iconv
igbinary
imagick
imap
intl
json
libxml
mbstring
mcrypt
memcached
mysqli
mysqlnd
OAuth
openssl
pcntl
pcre
PDO
pdo_mysql
PDO_ODBC
pdo_pgsql
pdo_sqlite
pgsql
Phar
posix
readline
redis
Reflection
session
shmop
SimpleXML
soap
sockets
sodium
SPL
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tidy
tokenizer
uuid
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
datadog-profiling
ddappsec
ddtrace

Output of phpinfo()

Our configuration

DD_TRACE_ENABLED=true
DD_TRACE_DEBUG=true
DD_TRACE_AMQP_ENABLED=false
DD_TRACE_CLI_ENABLED=true
DD_AUTOFINISH_SPANS=true
DD_TRACE_GENERATE_ROOT_SPAN=true
DD_TRACE_AUTO_FLUSH_ENABLED=true

Upgrading from

0.99

[mrtus]

Turned out I simply had to use startRootSpan instead of startActiveSpan.