New Relic transaction naming support for Jobs?

[epicwhale]

I was looking for pointers / support on how can each Job be reported as a separate New Relic transaction.

Right now in my New Relic, I see the bin/resque as one thick transaction with no further clarity on which job is taking how long..

Would like your thoughts on better integration with New Relic?

[danhunsaker]

Have your jobs call newrelic_name_transaction() (see https://docs.newrelic.com/docs/agents/php-agent/configuration/php-agent-api#api-name-wt for details). Other nifty things you can do using NewRelic PHP API calls include changing the app name on a per-run basis, defining custom metrics, and starting a new transaction before the script completes. PHP-Resque's hooks system can also come in handy for using some of these features to their furthest extent.

[epicwhale]

@danhunsaker thanks for the suggestions.. doing some R&D on this today. Do you think new Relic treats a fork of a process as a new transaction that begins and ends when the forked process (Job) ends? My fear is that it will treat the whole worker queue, polling and execution as one thick transaction.

So I think besides setting a name for the transaction, I will also explicitly need to call newrelic_end_transaction when the Job is being cleaned up.

This function causes the current transaction to end immediately, and will ship all of the metrics gathered thus far to the daemon unless the ignore parameter is set to true. In effect this call simulates what would happen when PHP terminates the current transaction. This is most commonly used in command line scripts that do some form of job queue processing. You would use this call at the end of processing a single job task, and begin a new transaction (see below) when a new task is pulled off the queue.

[epicwhale]

@danhunsaker setting the name of the transaction in the setup() { .. } section of the Job does make it a separate transaction in the New Relic dashboard, but it looks like it treats the whole worker as one transaction and not each of its individual jobs...

I tried using newrelic_end_transaction() in the tearDown block, but its somehow not working for me..

[danhunsaker]

You'll probably need to end the current transaction and start a new one in the setUp(), right before you name it. The process has already forked by this point, so you'll want to force it to use a completely new transaction instead of reusing the one from the worker. That should do what you're looking for.

The recommended usage in the NewRelic docs is for a worker that doesn't fork to run jobs. We need to do things a bit differently when forking is used, since the job process is a complete duplicate of the worker process when it starts, down to variable references and connections to external data sources. Telling NewRelic "I'm not the worker anymore; start a new transaction" decouples the job process from the worker so stats are passed per job correctly.

[epicwhale]

@danhunsaker Ok, sounds logical. Trying this out, starting the transaction and naming it in setup() and endTransaction in tearDown .

[epicwhale]

@danhunsaker hmm.. still no luck. It appears as a separate transaction, but still its execution time seems to be that of the worker... goes into hundreds of seconds!

I was looking at this custom boostrap file by @ruudk https://gist.github.com/ruudk/a28319b83cb8a08fd979 where he seems to be implementing new relic too.. but from what I read, he's setting a new transaction name to every worker (and not every job that's processed). What help would that be?

[danhunsaker]

It would let you tell apart which worker is doing which jobs, for one. Especially useful if each worker has a dedicated queue.

You're missing a step in your setUp(). You need to end the worker's transaction before you start the job's transaction. So that's two places with a transaction end, though the one in tearDown() is probably optional, since the NewRelic extension should automatically end the transaction when the forked process terminates.

[epicwhale]

@danhunsaker Got you, trying this now:

in setUp()

• End Transaction
• Start Transaction
• Set Transaction Name

in tearDown()

• End transaction

Shipped this onto servers. Will let you know the results in a bit.

[epicwhale]

@danhunsaker unfortunately, this doesn't work at all.. in fact now, no transaction appears in new relic itself. I've also raised a ticket with New Relic for support, but I doubt they'll be able to wrap their mind around the PHP version of resque.

[danhunsaker]

That's odd. That should work perfectly. I wonder what's causing issues there...

Probably best to roll back to the previous approach until NewRelic can take a look - at least that way you get some metrics, even if they are a bit misleading. NewRelic is actually pretty good at working these things out, and the HOWITWORKS.md I wrote should help them figure it out faster. The important bit is that Resque forks for each job, and the NewRelic extension may not understand that. Which will mean adding support for process forking in a future version of the extension (probably in the form of a function for reconnecting to the NewRelic daemon, which you'd call about the same place you reconnect to databases and so forth). If nothing else, we've found a documentation bug. 😄

[epicwhale]

@danhunsaker I have started a dialogue with New Relic's technical support. I've also linked them to our conversation file and the HOWITWORKS.md

Just to be absolutely sure, this (pasted below) is what my Job class looks like right now.. (striped down for verbosity sake). Do you mind taking a quick look and confirming I'm doing exactly what we discussed.

<?php

use BCC\ResqueBundle\ContainerAwareJob;

class MyJob extends ContainerAwareJob
{

    public function setUp()
    {
        // my initialization code

        \newrelic_end_of_transaction();
        \newrelic_start_transaction("my-app");
        \newrelic_name_transaction($transactionName);
    }

    public function run($args)
    {
        // my execution logic here
    }

    /**
     * {@inheritDoc}
     */
    public function tearDown()
    {
        \newrelic_end_of_transaction();
        parent::tearDown();
    }
}

[epicwhale]

Official reply from New Relic's team

I've just spoken to our PHP developers and unfortunately it seems we do not support forking PHP processes. Our Agent currently doesn't support this as it uses the PCNTL library.

[danhunsaker]

Yeah, about what I expected would be the response. Hopefully that changes in a later version.

Your code looks good, by the way. I don't use Symfony myself, so the BCC Bundle's use of ->run() threw me off for a moment, but having checked their code, it does make sense that way. It adapts the Ruby approach (essentially, serialize the object when you add it to the queue, then unserialize it in the worker when you pull it off the queue, so it doesn't strictly have function arguments so much as object properties that are properly set on either end; we could do something similar in PHP, but the way we do it instead is a bit cleaner and more compatible with queueing jobs in one language and running them in the other) to the PHP approach (where it's a bit more complicated to auto-initialize objects, so we instead have an arguments array, a setUp(), and a tearDown(), with the understanding that setUp() will use $this->args to properly set up the object, perform() will use the job class's own properties to do its work, and tearDown() will ensure everything gets properly cleaned up) in a bit more intuitive a fashion (passing $this->args to $this->run() turns it back into a proper function, which is the obvious way to do things).

[epicwhale]

@danhunsaker ye, unfortunate. Will postback here if New Relic ever brings support for this feature.. but now monitoring the performance of which Job is taking long looks like a challenge to me.

Just to put your workable solution/opinion on record and as a point of reference in this thread, if New Relic transaction can't work per job, what do you think is your suggestion for the next best way of transaction tracing.

in the setup = create a new transaction + set a name and tear down = end transaction.

Or will this strategy be useless too unless I use one dedicated queue per Job type?

Thanks for the pointers so far.

[danhunsaker]

I haven't tested any of these configurations, so I can't say for sure, but that seems like a decent place to start. Something else you might do is newrelic_add_custom_parameter(), a special extra label that might further break down your transactions for you. Though in all honesty, perhaps you shouldn't rely on the transaction system, and instead initialize a variable (such as a property of the job object) to the start time of the job, then subtract that from the end time of the job, and pass the result into newrelic_custom_metric(). Then, just set up a custom dashboard to track the value, which you can name according to job class, or even ID if you're really insane (you only get 2000 metrics in the system before weird things start happening, so I wouldn't use the ID to name your metrics). Your best bet is probably to do both a custom metric per job class, and a custom parameter (or a handful) per job, so you can track how long a given class tends to take, but also identify specific jobs where a given class takes longer than usual to execute.

You might also look into the newrelic_add_custom_tracer() option. That may let you break out different classes into their own traces programmatically without much work on your part. Again, I've never tested any of this, so I can't say for certain how well any given setup will work, but there are a few options that might work, depending on how well the extension cooperates with forking code.

[rrhyne]

@epicwhale very curious about where you ended up with this!

[epicwhale]

@rrhyne No where. New Relic officially confirmed that this wont work as expected as pcntl_fork is not supported by New Relic's PHP Agent. If you are a New Relic customer, you should make a feature request in their support desk too, just in case they decide to hurry this up.