[Bug]: Memory leak in creating Tracer

[iNviNho]

Bug report

Hi,

After an ugprade of the library, we are experiencing memory leaks.

We are running a long running process that starts tracing every time a message is consumed from Kafka and we close the global trace every time message is successfully processed.

This is the code we run every time we consume a message to start tracing

protected function startDatadogTracing(
        string $topicName,
        string $traceId = null,
        string $parentId = null
    ): void {
        if (!empty($traceId) && !empty($parentId)) {
            set_distributed_tracing_context($traceId, $parentId);
        }

        $tracer = GlobalTracer::get();
        GlobalTracer::set($tracer);

        $resourceName = $topicName . '::process';

        $options = ['start_time' => ((int)(microtime(true) * 1000 * 1000))];
        $rootSpan = $tracer->startRootSpan(
            'Jobs',
            $options
        )
            ->getSpan();

        $rootSpan->setTag(Tag::SPAN_TYPE, Type::CLI);
        $rootSpan->setTag(Tag::SERVICE_NAME, 'broker.xxx');
        $rootSpan->setTag(Tag::RESOURCE_NAME, $resourceName, true);
        $rootSpan->setTag('command_origin', $this->signature);

        if (!$this->isTracingStarted()) {
            $this->setupTracingOfClasses();
            $this->setTracingStarted(true);
        }
    }

and this is the whole code together with an example how we used to close root spans and send them to dd-agent./

$this->startDatadogTracing(
    $message->topic_name . '-persist-offer-created',
    $message->headers['x-datadog-trace-id'] ?? null,
    $message->headers['x-datadog-parent-id'] ?? null
);
$this->offerService->processOfferCreatedToPersistenceCommand(
    $message
);
GlobalTracer::get()->getRootScope()?->getSpan()->setTag('result', 'OK');
GlobalTracer::get()->getRootScope()?->close();
GlobalTracer::get()->flush()

however when running the setup locally, after every processed message memory increases by a bit. I am tracking the used memory using this simple line of code

dump(memory_get_usage());

Do you guys have any idea what we might be doing wrong?

Thank you.

PHP version

8.1.27

Tracer or profiler version

0.98.0

Installed extensions

/var/www/app # php -m
[PHP Modules]
bcmath
calendar
Core
ctype
curl
datadog-profiling
date
ddappsec
ddtrace
dom
fileinfo
filter
ftp
hash
iconv
intl
json
libxml
mbstring
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
rdkafka
readline
redis
Reflection
session
SimpleXML
soap
sockets
sodium
SPL
sqlite3
standard
tokenizer
xdebug
xml
xmlreader
xmlwriter
Zend OPcache
zip
zlib

[Zend Modules]
Xdebug
Zend OPcache
datadog-profiling
ddappsec
ddtrace

Output of phpinfo()

/var/www/app # php --ri=ddtrace; php --ri=datadog-profiling

ddtrace

Datadog PHP tracer extension
For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/
(c) Datadog 2020

Datadog tracing support => enabled
Version => 0.98.0
DATADOG TRACER CONFIGURATION => {
"date": "2024-03-25T12:58:26Z",
"os_name": "Linux 1bb3197e090f 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64",
"os_version": "5.15.49-linuxkit-pr",
"version": "0.98.0",
"lang": "php",
"lang_version": "8.1.26",
"env": null,
"enabled": true,
"service": "broker.xxx",
"enabled_cli": true,
"agent_url": "http://datadog:8126",
"debug": false,
"analytics_enabled": false,
"sample_rate": -1,
"sampling_rules": [],
"tags": [],
"service_mapping": {
"pdo": "broker.xxx.db",
"phpredis": "broker.xxx.redis"
},
"distributed_tracing_enabled": true,
"priority_sampling_enabled": true,
"dd_version": null,
"architecture": "aarch64",
"sapi": "cli",
"datadog.trace.request_init_hook": "/opt/datadog/dd-library/0.98.0/dd-trace-sources/bridge/dd_wrap_autoloader.php",
"open_basedir_configured": false,
"uri_fragment_regex": null,
"uri_mapping_incoming": null,
"uri_mapping_outgoing": null,
"auto_flush_enabled": true,
"generate_root_span": false,
"http_client_split_by_domain": false,
"measure_compile_time": true,
"report_hostname_on_root_span": false,
"traced_internal_functions": null,
"auto_prepend_file_configured": false,
"integrations_disabled": "default",
"enabled_from_env": true,
"opcache.file_cache": null,
"agent_error": "Could not resolve host: datadog"
}

                           Diagnostics                               

agent_error => Could not resolve host: datadog
Diagnostic checks => failed

Directive => Local Value => Master Value
ddtrace.disable => Off => Off
ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup
datadog.trace.sidecar_trace_sender => Off => Off
datadog.trace.request_init_hook => /opt/datadog/dd-library/0.98.0/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.98.0/dd-trace-sources/bridge/dd_wrap_autoloader.php
ddtrace.request_init_hook => /opt/datadog/dd-library/0.98.0/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.98.0/dd-trace-sources/bridge/dd_wrap_autoloader.php
datadog.trace.agent_url => no value => no value
datadog.agent_host => datadog => datadog
datadog.dogstatsd_url => no value => no value
datadog.api_key => no value => no value
datadog.distributed_tracing => On => On
datadog.dogstatsd_port => 8125 => 8125
datadog.env => no value => no value
datadog.autofinish_spans => Off => Off
datadog.trace.url_as_resource_names_enabled => On => On
datadog.http_server_route_based_naming => On => On
datadog.integrations_disabled => default => default
datadog.priority_sampling => On => On
datadog.service => broker.offer-service => broker.offer-service
datadog.service_name => broker.offer-service => broker.offer-service
datadog.service_mapping => pdo:broker.xxx.db,phpredis:broker.xxx.redis => pdo:broker.xxx.db,phpredis:broker.xxx.redis
datadog.tags => no value => no value
datadog.trace.global_tags => no value => no value
datadog.trace.agent_port => 8126 => 8126
datadog.trace.analytics_enabled => Off => Off
datadog.trace.append_trace_ids_to_logs => Off => Off
datadog.trace.auto_flush_enabled => On => On
datadog.trace.cli_enabled => On => On
datadog.trace.measure_compile_time => On => On
datadog.trace.debug => Off => Off
datadog.trace.enabled => On => On
datadog.instrumentation_telemetry_enabled => On => On
datadog.trace.health_metrics_enabled => Off => Off
datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001
datadog.trace.db_client_split_by_instance => Off => Off
datadog.trace.http_client_split_by_domain => Off => Off
datadog.trace.redis_client_split_by_host => On => On
datadog.trace.memory_limit => no value => no value
datadog.trace.report_hostname => Off => Off
datadog.trace.flush_collect_cycles => Off => Off
datadog.trace.laravel_queue_distributed_tracing => On => On
datadog.trace.remove_root_span_laravel_queue => On => On
datadog.trace.remove_autoinstrumentation_orphans => Off => Off
datadog.trace.resource_uri_fragment_regex => no value => no value
datadog.trace.resource_uri_mapping_incoming => no value => no value
datadog.trace.resource_uri_mapping_outgoing => no value => no value
datadog.trace.resource_uri_query_param_allowed => no value => no value
datadog.trace.http_url_query_param_allowed => * => *
datadog.trace.http_post_data_param_allowed => no value => no value
datadog.trace.rate_limit => 0 => 0
datadog.trace.sample_rate => -1 => -1
datadog.sampling_rate => -1 => -1
datadog.trace.sampling_rules => [] => []
datadog.trace.sampling_rules_format => regex => regex
datadog.span_sampling_rules => [] => []
datadog.span_sampling_rules_file => no value => no value
datadog.trace.header_tags => no value => no value
datadog.trace.x_datadog_tags_max_length => 512 => 512
datadog.trace.peer_service_mapping => no value => no value
datadog.trace.peer_service_defaults_enabled => Off => Off
datadog.trace.remove_integration_service_names_enabled => Off => Off
datadog.trace.propagate_service => Off => Off
datadog.trace.propagation_style_extract => datadog,tracecontext,B3,B3 single header => datadog,tracecontext,B3,B3 single header
datadog.propagation_style_extract => datadog,tracecontext,B3,B3 single header => datadog,tracecontext,B3,B3 single header
datadog.trace.propagation_style_inject => datadog,tracecontext => datadog,tracecontext
datadog.propagation_style_inject => datadog,tracecontext => datadog,tracecontext
datadog.trace.propagation_style => datadog,tracecontext => datadog,tracecontext
datadog.trace.traced_internal_functions => no value => no value
datadog.trace.agent_timeout => 500 => 500
datadog.trace.agent_connect_timeout => 100 => 100
datadog.trace.debug_prng_seed => -1 => -1
datadog.log_backtrace => Off => Off
datadog.trace.generate_root_span => Off => Off
datadog.trace.spans_limit => 5000 => 5000
datadog.trace.128_bit_traceid_generation_enabled => On => On
datadog.trace.128_bit_traceid_logging_enabled => Off => Off
datadog.trace.agent_max_consecutive_failures => 3 => 3
datadog.trace.agent_attempt_retry_time_msec => 5000 => 5000
datadog.trace.bgs_connect_timeout => 2000 => 2000
datadog.trace.bgs_timeout => 5000 => 5000
datadog.trace.agent_flush_interval => 5000 => 5000
datadog.trace.agent_flush_after_n_requests => 10 => 10
datadog.trace.shutdown_timeout => 5000 => 5000
datadog.trace.startup_logs => On => On
datadog.trace.once_logs => On => On
datadog.trace.agent_retries => 0 => 0
datadog.trace.agent_debug_verbose_curl => Off => Off
datadog.trace.debug_curl_output => Off => Off
datadog.trace.beta_high_memory_pressure_percent => 80 => 80
datadog.trace.agentless => Off => Off
datadog.trace.warn_legacy_dd_trace => On => On
datadog.trace.retain_thread_capabilities => Off => Off
datadog.version => no value => no value
datadog.trace.obfuscation_query_string_regexp => (?i)(?:(?:"|%22)?)(?:(?:old[-]?|new[-]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-]?phrase)?|secret|(?:api[-]?|private[-]?|public[-]?|access[-]?|secret[-]?|app(?:lication)?[-]?)key(?:[-]?id)?|token|consumer[-]?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|(?:bearer(?:\s|%20)+[a-z0-9.-]+|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|-{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY-{5}[^\-]+-{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY(?:-{5})?(?:\n|%0A)?|(?:ssh-(?:rsa|dss)|ecdsa-[a-z0-9]+-[a-z0-9]+)(?:\s|%20|%09)+(?:[a-z0-9/.+]|%2F|%5C|%2B){100,}(?:=|%3D)*(?:(?:\s|%20|%09)+[a-z0-9.-]+)?) => (?i)(?:(?:"|%22)?)(?:(?:old[-]?|new[-]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-]?phrase)?|secret|(?:api[-]?|private[-]?|public[-]?|access[-]?|secret[-]?|app(?:lication)?[-]?)key(?:[-]?id)?|token|consumer[-]?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|(?:bearer(?:\s|%20)+[a-z0-9.-]+|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|-{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY-{5}[^\-]+-{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY(?:-{5})?(?:\n|%0A)?|(?:ssh-(?:rsa|dss)|ecdsa-[a-z0-9]+-[a-z0-9]+)(?:\s|%20|%09)+(?:[a-z0-9/.+]|%2F|%5C|%2B){100,}(?:=|%3D)*(?:(?:\s|%20|%09)+[a-z0-9.-]+)?)
datadog.trace.client_ip_enabled => Off => Off
datadog.trace.client_ip_header => no value => no value
datadog.trace.forked_process => On => On
datadog.trace.hook_limit => 100 => 100
datadog.trace.agent_max_payload_size => 52428800 => 52428800
datadog.trace.agent_stack_initial_size => 131072 => 131072
datadog.trace.agent_stack_backlog => 12 => 12
datadog.trace.propagate_user_id_default => Off => Off
datadog.dbm_propagation_mode => full => full
datadog.trace.wordpress_additional_actions => no value => no value
datadog.trace.wordpress_callbacks => Off => Off
datadog.trace.wordpress_enhanced_integration => Off => Off
datadog.trace.otel_enabled => Off => Off
datadog.trace.log_file => no value => no value
datadog.trace.log_level => error => error
datadog.trace.amqp_enabled => On => On
datadog.trace.amqp_analytics_enabled => Off => Off
datadog.amqp_analytics_enabled => Off => Off
datadog.trace.amqp_analytics_sample_rate => 1 => 1
datadog.amqp_analytics_sample_rate => 1 => 1
datadog.trace.cakephp_enabled => On => On
datadog.trace.cakephp_analytics_enabled => Off => Off
datadog.cakephp_analytics_enabled => Off => Off
datadog.trace.cakephp_analytics_sample_rate => 1 => 1
datadog.cakephp_analytics_sample_rate => 1 => 1
datadog.trace.codeigniter_enabled => On => On
datadog.trace.codeigniter_analytics_enabled => Off => Off
datadog.codeigniter_analytics_enabled => Off => Off
datadog.trace.codeigniter_analytics_sample_rate => 1 => 1
datadog.codeigniter_analytics_sample_rate => 1 => 1
datadog.trace.exec_enabled => On => On
datadog.trace.exec_analytics_enabled => Off => Off
datadog.exec_analytics_enabled => Off => Off
datadog.trace.exec_analytics_sample_rate => 1 => 1
datadog.exec_analytics_sample_rate => 1 => 1
datadog.trace.curl_enabled => On => On
datadog.trace.curl_analytics_enabled => Off => Off
datadog.curl_analytics_enabled => Off => Off
datadog.trace.curl_analytics_sample_rate => 1 => 1
datadog.curl_analytics_sample_rate => 1 => 1
datadog.trace.drupal_enabled => On => On
datadog.trace.drupal_analytics_enabled => Off => Off
datadog.drupal_analytics_enabled => Off => Off
datadog.trace.drupal_analytics_sample_rate => 1 => 1
datadog.drupal_analytics_sample_rate => 1 => 1
datadog.trace.elasticsearch_enabled => On => On
datadog.trace.elasticsearch_analytics_enabled => Off => Off
datadog.elasticsearch_analytics_enabled => Off => Off
datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1
datadog.elasticsearch_analytics_sample_rate => 1 => 1
datadog.trace.eloquent_enabled => On => On
datadog.trace.eloquent_analytics_enabled => Off => Off
datadog.eloquent_analytics_enabled => Off => Off
datadog.trace.eloquent_analytics_sample_rate => 1 => 1
datadog.eloquent_analytics_sample_rate => 1 => 1
datadog.trace.guzzle_enabled => On => On
datadog.trace.guzzle_analytics_enabled => Off => Off
datadog.guzzle_analytics_enabled => Off => Off
datadog.trace.guzzle_analytics_sample_rate => 1 => 1
datadog.guzzle_analytics_sample_rate => 1 => 1
datadog.trace.laminas_enabled => On => On
datadog.trace.laminas_analytics_enabled => Off => Off
datadog.laminas_analytics_enabled => Off => Off
datadog.trace.laminas_analytics_sample_rate => 1 => 1
datadog.laminas_analytics_sample_rate => 1 => 1
datadog.trace.laravel_enabled => On => On
datadog.trace.laravel_analytics_enabled => Off => Off
datadog.laravel_analytics_enabled => Off => Off
datadog.trace.laravel_analytics_sample_rate => 1 => 1
datadog.laravel_analytics_sample_rate => 1 => 1
datadog.trace.laravelqueue_enabled => On => On
datadog.trace.laravelqueue_analytics_enabled => Off => Off
datadog.laravelqueue_analytics_enabled => Off => Off
datadog.trace.laravelqueue_analytics_sample_rate => 1 => 1
datadog.laravelqueue_analytics_sample_rate => 1 => 1
datadog.trace.logs_enabled => Off => Off
datadog.logs_injection => Off => Off
datadog.trace.logs_analytics_enabled => Off => Off
datadog.logs_analytics_enabled => Off => Off
datadog.trace.logs_analytics_sample_rate => 1 => 1
datadog.logs_analytics_sample_rate => 1 => 1
datadog.trace.lumen_enabled => On => On
datadog.trace.lumen_analytics_enabled => Off => Off
datadog.lumen_analytics_enabled => Off => Off
datadog.trace.lumen_analytics_sample_rate => 1 => 1
datadog.lumen_analytics_sample_rate => 1 => 1
datadog.trace.magento_enabled => On => On
datadog.trace.magento_analytics_enabled => Off => Off
datadog.magento_analytics_enabled => Off => Off
datadog.trace.magento_analytics_sample_rate => 1 => 1
datadog.magento_analytics_sample_rate => 1 => 1
datadog.trace.memcache_enabled => On => On
datadog.trace.memcache_analytics_enabled => Off => Off
datadog.memcache_analytics_enabled => Off => Off
datadog.trace.memcache_analytics_sample_rate => 1 => 1
datadog.memcache_analytics_sample_rate => 1 => 1
datadog.trace.memcached_enabled => On => On
datadog.trace.memcached_analytics_enabled => Off => Off
datadog.memcached_analytics_enabled => Off => Off
datadog.trace.memcached_analytics_sample_rate => 1 => 1
datadog.memcached_analytics_sample_rate => 1 => 1
datadog.trace.mongo_enabled => On => On
datadog.trace.mongo_analytics_enabled => Off => Off
datadog.mongo_analytics_enabled => Off => Off
datadog.trace.mongo_analytics_sample_rate => 1 => 1
datadog.mongo_analytics_sample_rate => 1 => 1
datadog.trace.mongodb_enabled => On => On
datadog.trace.mongodb_analytics_enabled => Off => Off
datadog.mongodb_analytics_enabled => Off => Off
datadog.trace.mongodb_analytics_sample_rate => 1 => 1
datadog.mongodb_analytics_sample_rate => 1 => 1
datadog.trace.mysqli_enabled => On => On
datadog.trace.mysqli_analytics_enabled => Off => Off
datadog.mysqli_analytics_enabled => Off => Off
datadog.trace.mysqli_analytics_sample_rate => 1 => 1
datadog.mysqli_analytics_sample_rate => 1 => 1
datadog.trace.nette_enabled => On => On
datadog.trace.nette_analytics_enabled => Off => Off
datadog.nette_analytics_enabled => Off => Off
datadog.trace.nette_analytics_sample_rate => 1 => 1
datadog.nette_analytics_sample_rate => 1 => 1
datadog.trace.pcntl_enabled => On => On
datadog.trace.pcntl_analytics_enabled => Off => Off
datadog.pcntl_analytics_enabled => Off => Off
datadog.trace.pcntl_analytics_sample_rate => 1 => 1
datadog.pcntl_analytics_sample_rate => 1 => 1
datadog.trace.pdo_enabled => On => On
datadog.trace.pdo_analytics_enabled => Off => Off
datadog.pdo_analytics_enabled => Off => Off
datadog.trace.pdo_analytics_sample_rate => 1 => 1
datadog.pdo_analytics_sample_rate => 1 => 1
datadog.trace.phpredis_enabled => On => On
datadog.trace.phpredis_analytics_enabled => Off => Off
datadog.phpredis_analytics_enabled => Off => Off
datadog.trace.phpredis_analytics_sample_rate => 1 => 1
datadog.phpredis_analytics_sample_rate => 1 => 1
datadog.trace.predis_enabled => On => On
datadog.trace.predis_analytics_enabled => Off => Off
datadog.predis_analytics_enabled => Off => Off
datadog.trace.predis_analytics_sample_rate => 1 => 1
datadog.predis_analytics_sample_rate => 1 => 1
datadog.trace.psr18_enabled => On => On
datadog.trace.psr18_analytics_enabled => Off => Off
datadog.psr18_analytics_enabled => Off => Off
datadog.trace.psr18_analytics_sample_rate => 1 => 1
datadog.psr18_analytics_sample_rate => 1 => 1
datadog.trace.roadrunner_enabled => On => On
datadog.trace.roadrunner_analytics_enabled => Off => Off
datadog.roadrunner_analytics_enabled => Off => Off
datadog.trace.roadrunner_analytics_sample_rate => 1 => 1
datadog.roadrunner_analytics_sample_rate => 1 => 1
datadog.trace.sqlsrv_enabled => On => On
datadog.trace.sqlsrv_analytics_enabled => Off => Off
datadog.sqlsrv_analytics_enabled => Off => Off
datadog.trace.sqlsrv_analytics_sample_rate => 1 => 1
datadog.sqlsrv_analytics_sample_rate => 1 => 1
datadog.trace.slim_enabled => On => On
datadog.trace.slim_analytics_enabled => Off => Off
datadog.slim_analytics_enabled => Off => Off
datadog.trace.slim_analytics_sample_rate => 1 => 1
datadog.slim_analytics_sample_rate => 1 => 1
datadog.trace.symfony_enabled => On => On
datadog.trace.symfony_analytics_enabled => Off => Off
datadog.symfony_analytics_enabled => Off => Off
datadog.trace.symfony_analytics_sample_rate => 1 => 1
datadog.symfony_analytics_sample_rate => 1 => 1
datadog.trace.web_enabled => On => On
datadog.trace.web_analytics_enabled => Off => Off
datadog.web_analytics_enabled => Off => Off
datadog.trace.web_analytics_sample_rate => 1 => 1
datadog.web_analytics_sample_rate => 1 => 1
datadog.trace.wordpress_enabled => On => On
datadog.trace.wordpress_analytics_enabled => Off => Off
datadog.wordpress_analytics_enabled => Off => Off
datadog.trace.wordpress_analytics_sample_rate => 1 => 1
datadog.wordpress_analytics_sample_rate => 1 => 1
datadog.trace.yii_enabled => On => On
datadog.trace.yii_analytics_enabled => Off => Off
datadog.yii_analytics_enabled => Off => Off
datadog.trace.yii_analytics_sample_rate => 1 => 1
datadog.yii_analytics_sample_rate => 1 => 1
datadog.trace.zendframework_enabled => On => On
datadog.trace.zendframework_analytics_enabled => Off => Off
datadog.zendframework_analytics_enabled => Off => Off
datadog.trace.zendframework_analytics_sample_rate => 1 => 1
datadog.zendframework_analytics_sample_rate => 1 => 1

datadog-profiling

Version => 0.98.0
Profiling Enabled => false
Profiling Experimental Features Enabled => false (profiling disabled)
Experimental CPU Time Profiling Enabled => false (profiling disabled)
Allocation Profiling Enabled => false (profiling disabled)
Timeline Enabled => false (profiling disabled)
Exception Profiling Enabled => false (profiling disabled)
Endpoint Collection Enabled => false (profiling disabled)
Platform's CPU Time API Works => true
Profiling Log Level => off (profiling disabled)
Profiling Agent Endpoint => http://datadog:8126/
Application's Environment (DD_ENV) =>
Application's Service (DD_SERVICE) => broker.xxx
Application's Version (DD_VERSION) =>

Directive => Local Value => Master Value
datadog.profiling.enabled => Off => Off
datadog.profiling.experimental_features_enabled => Off => Off
datadog.profiling.endpoint_collection_enabled => On => On
datadog.profiling.experimental_cpu_time_enabled => On => On
datadog.profiling.experimental_cpu_enabled => On => On
datadog.profiling.allocation_enabled => Off => Off
datadog.profiling.experimental_allocation_enabled => Off => Off
datadog.profiling.timeline_enabled => On => On
datadog.profiling.experimental_timeline_enabled => On => On
datadog.profiling.exception_enabled => Off => Off
datadog.profiling.experimental_exception_enabled => Off => Off
datadog.profiling.exception_message_enabled => Off => Off
datadog.profiling.exception_sampling_distance => 100 => 100
datadog.profiling.experimental_exception_sampling_distance => 100 => 100
datadog.profiling.log_level => off => off
datadog.profiling.output_pprof => no value => no value

Upgrading from

Upgrading from 0.84.0 -> 0.98.0

[iNviNho]

Oh,

after exchanging

        $tracer = GlobalTracer::get();

to

        $tracer = new Tracer();

memory doesn't increase anymore. I am trying to think now whether it makes sense or not 🤔

[bwoebi]

Hey @iNviNho,

Is there something in the internal state of GlobalTracer::get() which is ever-growing? I.e. observing var_dump(GlobalTracer::get()), do you observe some arrays growing or data structures getting more and more nested?

[iNviNho]

After we call

GlobalTracer::get()->getRootScope()?->getSpan()->setTag('result', 'OK');
GlobalTracer::get()->getRootScope()?->close();
dump(GlobalTracer::get());

we always get DDTrace\Tracer object with a different object id as with every new message we consume, we create a new Tracer.

DDTrace\Tracer^ {#4063 // app/Console/Commands/LinkMultiSwapTradesCommand.php:91
  -traces: []
  -transport: DDTrace\Transport\Internal^ {#4087}
  -propagators: array:2 [
    "text_map" => DDTrace\Propagators\TextMap^ {#4086
      -tracer: DDTrace\Tracer^ {#4063}
    }
    "http_headers" => DDTrace\Propagators\TextMap^ {#4086}
  ]
  -config: array:3 [
    "service_name" => "cli"
    "enabled" => true
    "global_tags" => []
  ]
  -scopeManager: DDTrace\ScopeManager^ {#4037
    -scopes: []
    -hostRootScopes: array:1 [
      0 => DDTrace\Scope^ {#4047
        -span: DDTrace\Span^ {#4049
          +context: DDTrace\SpanContext^ {#4052
            +traceId: "89806696424059378"
            +spanId: "89806696424059378"
            +parentId: null
            +isDistributedTracingActivationContext: false
            +propagatedPrioritySampling: null
            +origin: null
            +parentContext: null
            +baggageItems: []
          }
          +hasError: false
          #internalSpan: ?
          +internalSpan: DDTrace\RootSpanData {#4048
            +name: "Jobs"
            +resource: "..."
            +service: "..."
            +type: "cli"
            +meta: array:5 [
              "runtime-id" => "52936d31-37bd-414e-a474-8a0842804c58"
              "version" => "0.0.1"
              "command_origin" => "..."
              "result" => "OK"
              "_dd.p.dm" => "-0"
            ]
            +metrics: array:2 [
              "process_id" => 753.0
              "_dd.agent_psr" => 1.0
            ]
            +exception: null
            +id: "89806696424059378"
            +links: []
            +peerServiceSources: []
            +parent: null
            +stack: DDTrace\SpanStack {#4006
              +parent: DDTrace\SpanStack {#1
                +parent: null
                +active: null
              }
              +active: null
            }
            +origin: ? string
            +propagatedTags: []
            +samplingPriority: 1
            +propagatedSamplingPriority: ? int
            +tracestate: ? string
            +tracestateTags: []
            +parentId: ? string
            +traceId: "6602ef6300000000013f0eb6eeceb1f2"
          }
        }
        -scopeManager: DDTrace\ScopeManager^ {#4037}
        -finishSpanOnClose: true
      }
    ]
    -rootContext: null
  }
  -serviceVersion: "0.0.1"
  -environment: ""
  -rootContext: null
}

and the memory keeps increasing until we do

gc_collect_cycles();
gc_mem_caches();

[TheLevti]

We see that all our daemon deployments slowly climb in memory usage. This screenshot shows a datadog profile of total allocated memory over an hour in one of our service.

[realFlowControl]

Hey @TheLevti 👋

your problem looks like a different problem then what the author wrote.
Could you please signup to our public Slack or open a Support Case with us?
If you choose Slack, please ping "Florian Engelhardt" and/or "Bob Weinand"

[TheLevti]

Actually it is related as we investigate the same issue with @iNviNho.

What we have found out is that calling $tracer->reset() after flushing is fixing the memory leak.

Can we maybe have more details about when reset() should be called and whether thats the correct approach?

Why is the tracer accumulating already closed/flushed traces if reset is not called?

This call also allows us to use the global tracer instance as a singleton and not reinstantiate it after each trace.

[rmikalkenas]

@TheLevti unrelated question, but what tool do you use to measure memory usage and plot such graphs?

[TheLevti]

@TheLevti unrelated question, but what tool do you use to measure memory usage and plot such graphs?

Its datadog's profiler component. Shows you CPU time, wall time, memory allocations etc.