Detailed MongoDB/Mongoid instrumentation #16

Merged
merged 4 commits into from Feb 29, 2012

Projects

None yet
@titanous

This patch combines @bensymonds' wonderful MongoDB patch with Mongoid instrumentation to provide detailed transaction traces.

Despite instrumenting under the ActiveRecord and SQL metric categories, I'm still not seeing the Database section on my overall RPM graph show up. What should I do to get that?

Ben Symonds and others added some commits Oct 14, 2011
Ben Symonds Update for v1.4.0 of ruby mongo driver:
  - 'refresh' method has been refactored so the cursor refresh itself is in a new method 'send_get_more', so instrument that directly
  - add a comment explaining that cursor refresh is not covered by the mongo 'instrument' method, hence why we have to do it explicitly
  - 'instrument' method has been moved into separate Logging module
64b6408
Ben Symonds re-include Mongo::Logging after adding newrelic 84ee78d
Jonathan Rudenberg Merge remote-tracking branch 'bensymonds/mongo-1.4.0_update' into mon…
…goid-instrumentation

Conflicts:
	lib/rpm_contrib/instrumentation/mongo.rb
d0e475f
Jonathan Rudenberg Instrument Mongoid 5ba7563
@titanous

Looks like the first part of this patch is included in #15.

@samg
Member
samg commented Nov 23, 2011

tl;dr We need some Mongo community members to test this patch, and make sure it doesn't crash when combined with old version of Mongoid. Eventually we can do this internally, but it will be much faster if the community does it.

@rgabo - I would like to improved mongo instrumentation pulled into the main gem.

Here's the issue. We've received several pull requests for Mongo instrumentation. Most of them have only been tested against one version of of the Mongoid drivers, and it's not clear to me that pulling these into the main line won't break some people's apps, who are using incompatible versions of Mongoid. Crashing people's apps is a major support headache for us.

We would love to set up a variety of app environments, using various versions of Mongo and Mongoid and make sure that this code won't crash some apps, and does provide useful instrumentation. Unfortunately we're kept busy keeping up with the many, many app environments newrelic runs in. This type of environment setup and testing (which is fairly time consuming) for contrib instrumentation tends to take a back burner and be pushed off into the future.

But there is hope. If someone would test this instrumentation in various version of Mongo and Mongoid, and provide some reassurance that it will work in a variety of environments I'd be happy to pull this into the main gem. This may mean tightening up the the depends_on block to check for ::Mongo::Logging and other various constants that are referenced later, so we avoid crashing apps that use older versions of Mongoid.

I'm happy to provide feedback, guidance, and support to help this happen. I'm not a mongo user, but I can help make sure this gets pulled in, if others are willing to contribute some of the legwork.

@varchar
varchar commented Jan 18, 2012

Whats the latest on getting detailed mongoid logging into this gem?

@shingara

I test this patch and works like a charm +1 to merge it.

@thibaudgg

@shingara are you using it with Mongoid 2.4 and Mongo driver 1.5.2 ?

@shingara

Mongoid 2.3.4 and mongo 1.5.2. I try with mongoid 2.4 soon.

@shingara

It's works fine too with Mongoid 2.4.2

@thibaudgg

@shingara thanks, so I'll give it a try too.

@thibaudgg

Ok this patch works great for us too (Mongoid 2.4.1 & Mongo driver 1.5.2, on Heroku)

@samg
Member
samg commented Jan 23, 2012

I'm traveling the next couple days, but will look into merging this again this week. Any more testing or work on version compatibility that can be done in the meantime would be appreciated.

@jtescher

+1 mongoid-2.4.4 / mongo-1.5.2

@pithyless

I've tested this patch with:

Mongoid 2.4.5 / Mongo 1.6.0

Mongoid 2.2.2 / Mongo 1.5.2

+1 for helpful metrics and not crashing my apps :)

@samg samg merged commit 9c9ccd0 into newrelic:master Feb 29, 2012
@samg
Member
samg commented Feb 29, 2012

Merged. I'll try to get a new version of the gem cut in the next week or so.

Thanks for everyone's work on this.

@nickhoffman

That's fantastic news, @samg. Thanks for merging this in.

@iamnader
iamnader commented Mar 1, 2012

For those of using the heroku integration, will this instrumentation just start magically appearing?

@samg
Member
samg commented Mar 1, 2012

@iamnadar - No rpm_contrib is just like any other gem.

You can run this right now by putting something like this in your Gemfile:

gem 'rpm_contrib', :git => 'git://github.com/newrelic/rpm_contrib.git'
@iamnader
iamnader commented Mar 1, 2012

i previously had the rpm_contrib in my gemfile and that caused all sorts of problems. i was told to remove it because heroku auto-adds it

On Mar 1, 2012, at 10:17 AM, Sam Goldstein wrote:

@iamnadar - No rpm_contrib is just like any other gem.

You can run this right now by putting something like this in your Gemfile:

gem 'rpm_contrib', :git => 'git://github.com/newrelic/rpm_contrib.git'


Reply to this email directly or view it on GitHub:
#16 (comment)

@samg
Member
samg commented Mar 1, 2012

It's different between Bamboo and Cedar. On Cedar I believe you do all the gem management yourself. I'll look into Bamboo and get back to you.

@dblock
dblock commented Mar 5, 2012

With this code enabled, on a long running worker we start leaking (?) memory. Heroku complains:

2012-03-05T21:17:23+00:00 heroku[worker.1]: Process running mem=542M(106.1%)
2012-03-05T21:17:23+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)

Haven't debugged it any further, yet.

@samg
Member
samg commented Mar 19, 2012

This has been released a version 2.1.8.

@dblock We have some improvements to how the agent handles memory usage on long running workers, coming out later this month. You could try the beta we have out now, and see if it improves this behavior:
https://rubygems.org/gems/newrelic_rpm/versions/3.3.3.beta2

@dblock
dblock commented Mar 20, 2012

@samg will give it a try, could you please point to a commit that you think would fix my problem?

@samg
Member
samg commented Mar 20, 2012

@dblock - It's 0b2ba76 and 726f733. They should both be available in the beta referenced above.

@joe1chen

I'm getting an error with Mongoid 2.3.4 and Mongo 1.3.1:

Uncaught exception: uninitialized constant Mongo::Logging

This error only starts happening with version 2.1.8. The older version doesn't have this problem. It took me a while to track this down to here. Can we put a README message or something stating that 2.1.8 is only supported with Mongo driver 1.4 and above?

@samg
Member
samg commented Apr 10, 2012

@joe1chen - Sorry to hear that. What we'd like is for someone to pull the Mongo instrumentation into a separate gem (see README for more details), which would make it easier to deal with compatibility issues like this. Let me know if you've got any interest in tackling that (we'd help ;-). Otherwise I'd happily merge a pull request for an updated README.

Thanks.

@pithyless

@samg - If no one has got the ball rolling yet, I don't mind trying to put this together over the next few days.

I think this would be a great use case for Travis CI (run the tests against various Mongo + Mongoid builds). The only question is are you looking for separate newrelic-mongoid and newrelic-mongo gems? Or just one newrelic-mongo gem (w/ detection for Mongoid and eventually MongoMapper)?

@samg
Member
samg commented Apr 10, 2012

@pithyless - That would be awesome!

I'm open to either approach (separate or combined gems). I'd probably lean towards separate gems since that minimizes the amount of detection you have to do of what's installed in the app. We're discouraging the use of the DependencyDetection helpers for this purpose since these gems will be much more targeted than the monolithic rpm_contrib.

Look at https://github.com/evanphx/newrelic-redis for a good example.

Also feel free to email me (sam@newrelic.com) if you have questions or need pointers. Once you've got the projects extracted I'll update rpm_contrib to point to those projects.

Thanks!

@iamnader
iamnader commented May 8, 2012

@samg any word on getting detailed instrumentation onto the bamboo stack of heroku?

@dblock
dblock commented May 8, 2012

@iamnader We've been running this on bamboo just fine for a while now.

  gem "rpm_contrib", "2.1.8"
  gem "newrelic_rpm", "3.3.3"
@barmstrong

I'm having trouble getting this working with Heroku's cedar stack. I have the following versions:

gem 'mongoid',                       '2.4.3'
gem 'rpm_contrib',                   '2.1.9'
gem 'newrelic_rpm',                  '3.3.4'

But newrelic still only shows 'ruby' and 'request queueing' as the two components of a request. Has anyone got this working on heroku cedar stack? Thanks!

@dblock
dblock commented May 12, 2012

@barmstrong In New Relic, Menu, Diagnostics, Transaction Traces is the only place you're going to see those. Double check whether you have it in there.

@barmstrong

Interesting, that seemed to work mostly.

It's a paid feature so I upgraded on NewRelic. Now I see transaction traces, which is cool because I see the COUNT query took the most time:
http://cl.ly/3o243b0U0x1z2e3H3c0Q

Is there a way to view the actual query though? On the "trace details" tab I tried expanding the actual query and it seems to show an abstracted version of it with question marks in it.
http://cl.ly/0J2c2W0g3P2e392d1k1O

Is this to try and intelligently group the same types of queries together? Anyway, this is definitely way better than what I had before. Thanks for the help!

@samg
Member
samg commented May 13, 2012

@barmstrong - The ?s in the query are new relic's SQL obfuscation feature. By default we remove values from the SQL to avoid sending sensitive data (ids, usernames, etc) to the service. You can turn this off in your newrelic.yml. Also the obfuscation feature is set up to handle mysql, postgres, and sqlite style SQL so it looks like it's going a little haywire with the Mongo style non-SQL queries.

You can also define your own SQL obfuscation routine via the #set_sql_obfuscation method.
http://newrelic.github.com/rpm/NewRelic/Agent/Database/Obfuscator.html#method-i-set_sql_obfuscator

This is something the mongo instrumentation could do for mongo queries.

See https://github.com/projectdx/better_newrelic_sql_obfuscator for another example of how to do this.

@samg
Member
samg commented May 22, 2012

We (@newrelic) were just talking about mongo support across our agents, and we're very curious to look at some newrelic applications that are using this instrumentation. If anyone is willing to send me a link to their newrelic account (sam@newrelic.com) and possibly talk with us about how you're using mongo/newrelic I'd appreciate it.

Thanks!

@barmstrong

@samg I'd be interested in chatting more with you guys. Would love to see mongo instrumentation as a first class citizen in rpm. You can email me from my github profile. Thanks!

@jrdioko
jrdioko commented Aug 7, 2013

@samg Have there been any updates in the past year to this SQL obfuscation issue with mongo, or is the recommended way still to set a custom SQL obfuscation method?

@samg
Member
samg commented Aug 8, 2013

@jrdioko - Not really. The example I provided above is a good reference for how to do this yourself.

As an aside, we are planning to tackle mongo instrumentation in the agent this year, so expect some improvements there that don't involve shoehorning it into the SQL UI.

If you want to be notified when Mongo related features do launch the best thing would be to email support@newrelic.com (or open a ticket at support.newrelic.com) and ask them to put you on the list of users interested in that feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment