High CPU usage

[joeldodge79]

We're using Magento 1 and I decided to just throw this code into our index.php file as a first stab at seeing some data:

require_once('lib/vendor/autoload.php');
$tracer = new \DDTrace\Tracer(new \DDTrace\Transport\Http(new \DDTrace\Encoders\Json()));
\OpenTracing\GlobalTracer::set($tracer);
$scope = $tracer->startActiveSpan('ourorg.www.request');
$span = $scope->getSpan();
$span->setTag(\DDTrace\Tags\SERVICE_NAME, 'www');
$span->setTag(\DDTrace\Tags\SPAN_TYPE, \DDTrace\Types\WEB_SERVLET);
\DDTrace\Integrations\PDO::load();
register_shutdown_function(function () use ($scope) {
    $scope->close();
    \OpenTracing\GlobalTracer::get()->flush();
});

we got data in the datadog APM UI ... but the webserver slammed to 100% cpu and stopped responding. any suggestions?

[joeldodge79]

using this extension: https://github.com/DataDog/dd-trace-php/releases/download/0.2.7/datadog-php-tracer-0.2.7_beta-1.x86_64.rpm

[labbati]

Hi! @joeldodge79, sorry to hear that! Can you provide a little bit more detail about your environment, e.g. OS, php version, bits (I assume 64)? I will try to reproduce the issue in a docker + centos (I guessed from .rpm) environment with your code and will post the outcome here. Also bringing @pawelchcki in the loop (our C-extension master)

[labbati]

Ok I did a very quick test and things looks good as I do not see any CPU anomaly, so must be something else.
I would ask you to share more info about your env so we can see if we can replicate.
It may also be an issue specifically related to magento, and in that case we may need to postpone the investigation a little bit.

==============================================================================
Dumping here a few resources to speed up debug process:

I used the following Dockerfile

FROM naqoda/centos-apache-php:php71
RUN rpm -i https://github.com/DataDog/dd-trace-php/releases/download/0.2.7/datadog-php-tracer-0.2.7_beta-1.x86_64.rpm

Build:

docker build -t debug_centos .

Run through docker-compose:

  sample:
    image: "debug_centos"
    volumes:
      - .:/var/www/app/public_html
    environment:
      DD_AGENT_HOST: <YOUR_HOST_HERE>
    ports:
      - "8111:80"

index.php

<?php

require_once('./vendor/autoload.php');

echo "setup tracer\n";

$tracer = new \DDTrace\Tracer(new \DDTrace\Transport\Http(new \DDTrace\Encoders\Json()));
\OpenTracing\GlobalTracer::set($tracer);
$scope = $tracer->startActiveSpan('ourorg.www.request');
$span = $scope->getSpan();
$span->setTag(\DDTrace\Tags\SERVICE_NAME, 'www');
$span->setTag(\DDTrace\Tags\SPAN_TYPE, \DDTrace\Types\WEB_SERVLET);

\DDTrace\Integrations\PDO::load();

register_shutdown_function(function () use ($scope) {
    $scope->close();
    \OpenTracing\GlobalTracer::get()->flush();
});

echo "done setup tracer\n";

echo "Hey!\n";

[labbati]

Sorry, I closed it by mistake

[joeldodge79]

thanks for the quick reply! Here's some initial env info, I'll work on debugging this some more today:

php-fpm 7.1.23 running via nginx 1.12.1 on amazon linux 2018.3 :

> php-fpm -v
PHP 7.1.23 (fpm-fcgi) (built: Oct 16 2018 23:41:50)
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.1.23, Copyright (c) 1999-2018, by Zend Technologies
> nginx -v
nginx version: nginx/1.12.1
> uname -a
Linux ip-10-0-3-222 4.14.26-46.32.amzn1.x86_64 #1 SMP Fri Mar 30 22:29:54 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[SammyK]

FYI I've discovered some memory leaks in the extension which might be at least tangentially related to this issue. Working on a patch. :)

[joeldodge79]

I'm still trying to reproduce this issue in a way that's easy to share. I've narrowed it down to enabling the PDO integration: commenting out \DDTrace\Integrations\PDO::load(); avoids the cpu spike and pages render fine.

[labbati]

Hey @joeldodge79, just out of curiosity, were you able to re-enable the PDO integration and to have it work properly?

[labbati]

Hey @joeldodge79, I am closing this issue as it refers to a relatively old version and it had no interactions from a while. Feel free to re-open it if you still experience the issue. We are very interested in addressing any memory issue that may exist.

[danelowe]

Hey,

We're using Magento 1 and the site crashes when enabling the PDOIntegration, and also other integrations such as ElasticSearchIntegration and CurlIntegration

Simply adding the following code causes our site to crash using dd-trace-php 0.9.1

        dd_trace('PDOStatement', 'execute', function () {
            $params = func_get_args();
            return call_user_func_array([$this, 'execute'], $params);
        });

If I create a very simple Magento 1 shell script, and enable the PDOIntegration, that shell script will work, e.g:

<?php
use DDTrace\Integrations\PDO\PDOIntegration;

require_once 'abstract.php';
class TestShell extends Mage_Shell_Abstract
{
    public function run()
    {
      var_dump(PDOIntegration::load());
$products = Mage::getModel('catalog/product')->getCollection()->load();
    }

    public function usageHelp()
    {
        return <<<USAGE
USAGE;
    }
}

$shell = new TestShell();
$shell->run();

To me, the behaviour seems like memory leaks, because it crashes as soon as we use it on a full Magento page load.

[labbati]

Hi @danelowe thanks for the detailed feedback. We may need a few more info in order to track down the bug.

Which PHP and OS versions are using?
I don't think that this would change, but can you try with 0.10.0 latest release? Please be aware that there are a couple of breaking changes documented here.

Simply adding the following code causes our site to crash using dd-trace-php 0.9.1

So if my understanding of the problem is correct, when you add that code you do not use our other integrations at all. So, for example, you don't call IntegrationsLoader::load().

Magento is in our road map, but if you can point us to a simple test app that replicates the issue with instructions about how to run it we would will be happy to try a quick debug session!

Thanks!

[danelowe]

Hi @labbati,

This is with php7.2 on Ubuntu 14.04.

I will look into getting 0.10.0 in the next couple of days. I was just waiting for the .deb release first.

You understood correctly. It crashes with that code added, and no other code to load any integrations.

If 0.10.0 doesn't work, I will try to get a docker compose file ready with a Magento 1 demo.

[danelowe]

@labbati I spent the day creating a test app running in docker with test data and a fresh install of Magento.

Unfortunately I cannot replicate the issue in this docker container.

I can so far only replicate in our production server (with numerous services installed directly in Ubuntu).

Is there any way you can think of to debug further in this environment?

[labbati]

Hi, thanks for taking time to look into this 🙇

I was just waiting for the .deb release first.

.deb is loaded, it was my mistake during first hours from the upload.

Is there any way you can think of to debug further in this environment?

Typically a good starting point is to match php -i and php -m for the two instances, or at least approaching a perfect match.

[labbati]

Hi @danelowe, since our last message we introduced a few versions which, among other things, fixed a number of memory leaks we have found in our integration.

If you could give it a chance and test in your magento app it would be great! Based on the time of your latest comment you MAY already be on 0.10+, in that case the update would be smooth. If you are on a <0.10 please look into the migration guide.

Thanks 😃

[danelowe]

Hi @labbati,

I installed 0.13.1 php extension on our server via the .deb package (left the composer dependency at 0.10.0).

The same code seems to cause our site to crash

        dd_trace('PDOStatement', 'execute', function () {
            $params = func_get_args();
            return call_user_func_array([$this, 'execute'], $params);
        });

I wondered if there is some way this code leads to a non-terminating loop, but I couldn't replicate it in another environment, and can't think of anything that would cause it in Magento's code base.

At the moment, I've just manually instrumented Magento's PDO adapter.

[SammyK]

Hey @danelowe! Thanks for the extra info! I believe the issue you're having with the PDO integration in Magento 1 is related to #284 which is a known issue that we are working on. :)

[labbati]

Hi @danelowe #284 has been merged long time ago and I think your issue should have been resolved. I am closing this right now, please feel free to re-open it if the issue is still unresolved.