Automated ruby wrapper to optimize GC variables #122

Closed
rylwin opened this Issue Feb 2, 2012 · 7 comments

Comments

Projects
None yet
2 participants
@rylwin
Contributor

rylwin commented Feb 2, 2012

I've just spent a while messing around with creating a wrapper for ruby to set GC variables as described in this blog post (http://www.coffeepowered.net/2009/06/13/fine-tuning-your-garbage-collector/).

I'd like to automate the modification of passenger.conf and the creation of the ruby wrapper script. Is this something that I should write as a moonshine plugin or is this useful (common) enough to write into moonshine itself?

@technicalpickles

This comment has been minimized.

Show comment Hide comment
@technicalpickles

technicalpickles Feb 2, 2012

Contributor

Good question. I could see it going either way.

One thought that occurred to me is that it'd almost make sense to have those always be defined in the environment, so you don't have to specifically recall to use this wrapper script. Also, less changing of the passenger configuration.

What do you think?

Contributor

technicalpickles commented Feb 2, 2012

Good question. I could see it going either way.

One thought that occurred to me is that it'd almost make sense to have those always be defined in the environment, so you don't have to specifically recall to use this wrapper script. Also, less changing of the passenger configuration.

What do you think?

@rylwin

This comment has been minimized.

Show comment Hide comment
@rylwin

rylwin Feb 2, 2012

Contributor

I like the idea of defining the GC settings in the environment, but it turns out that this really isn't achievable. After some trial and error I started searching and came across this post: http://blog.phusion.nl/2008/12/16/passing-environment-variables-to-ruby-from-phusion-passenger/

Given the wrapper seems necessary there are two "features" that moonshine would need to support to automate this:

  1. Make PassengerRuby in passenger.conf.erb a configurable variable (default can still be /usr/bin/ruby).
  2. Add a recipe to create the wrapper script with settings that could be provided in moonshine.yml or the app manifest

It seems like item 1 would be best to change in moonshine itself, and it would be backwards compatible. Item 2 I'm not sure if it's better off as a plugin or part of moonshine, I guess it depends on how many people would want to customize these GC settings. I see it as critical given the huge performance boost I received by tweaking the GC settings, but perhaps that's just me.

Contributor

rylwin commented Feb 2, 2012

I like the idea of defining the GC settings in the environment, but it turns out that this really isn't achievable. After some trial and error I started searching and came across this post: http://blog.phusion.nl/2008/12/16/passing-environment-variables-to-ruby-from-phusion-passenger/

Given the wrapper seems necessary there are two "features" that moonshine would need to support to automate this:

  1. Make PassengerRuby in passenger.conf.erb a configurable variable (default can still be /usr/bin/ruby).
  2. Add a recipe to create the wrapper script with settings that could be provided in moonshine.yml or the app manifest

It seems like item 1 would be best to change in moonshine itself, and it would be backwards compatible. Item 2 I'm not sure if it's better off as a plugin or part of moonshine, I guess it depends on how many people would want to customize these GC settings. I see it as critical given the huge performance boost I received by tweaking the GC settings, but perhaps that's just me.

@rylwin

This comment has been minimized.

Show comment Hide comment
@rylwin

rylwin Feb 2, 2012

Contributor

Perhaps I'm making this too complex. Maybe the only thing that would have to change in moonshine is item 1 from my previous post: make PassengerRuby path configurable. Then you could just make the wrapper script and put in in ./lib and set PassengerRuby to point to /home/user/deploy_path/lib/ruby-with-env

And making PassengerRuby configurable can also be done by moving passenger.conf.erb to app/manifests/templates.

So now I have myself wondering, is it worth making a change to moonshine to make all of this easier?

Contributor

rylwin commented Feb 2, 2012

Perhaps I'm making this too complex. Maybe the only thing that would have to change in moonshine is item 1 from my previous post: make PassengerRuby path configurable. Then you could just make the wrapper script and put in in ./lib and set PassengerRuby to point to /home/user/deploy_path/lib/ruby-with-env

And making PassengerRuby configurable can also be done by moving passenger.conf.erb to app/manifests/templates.

So now I have myself wondering, is it worth making a change to moonshine to make all of this easier?

@technicalpickles

This comment has been minimized.

Show comment Hide comment
@technicalpickles

technicalpickles Feb 2, 2012

Contributor

It's a pretty easy thing to make configurable, so I don't think there's anything to lose.

However, it's worth thinking about how you would be tuning this thing. It's easy enough to expose configuring the GC variables, but the real challenge is if there's a reasonable set of defaults, and if not, how can you lead people to it? For mysql, for example, mysqltuner.pl can lead you to enlightenment. Not sure if there's something equivalent.

Contributor

technicalpickles commented Feb 2, 2012

It's a pretty easy thing to make configurable, so I don't think there's anything to lose.

However, it's worth thinking about how you would be tuning this thing. It's easy enough to expose configuring the GC variables, but the real challenge is if there's a reasonable set of defaults, and if not, how can you lead people to it? For mysql, for example, mysqltuner.pl can lead you to enlightenment. Not sure if there's something equivalent.

@rylwin

This comment has been minimized.

Show comment Hide comment
@rylwin

rylwin Feb 3, 2012

Contributor

I think it's possible to select defaults that, though not optimized for your specific rails app, would yield better performance than the ruby defaults. For example, just copying/pasting the config given in the link in the OP led to ruby creating 3 heaps on startup instead of the 9 it does by default--big performance boost. I then tweaked further and was able to get down to creating a single heap.

I got the relevant data to tune using https://github.com/mogest/scrap, which gives you GC stats by going to http://yoururl.com/stats/scrap. I suppose I could pull out the relevant stats provided by scrap into a rake task that could load the environment to get to the same GC data (since seeing the heap stats on startup seems to be enough to tweak the settings) and provide a recommendation based on that.

I would think the recommendation would be based on getting down to allocating a single heap on startup with some additional buffer (~10%?). To do this, one just needs to count the number of objects allocated when the Rails app starts and add a bit of buffer to that number--that would handle what's probably the most important GC setting and could be provided by the rake task.

The other settings are either based off of this number or would involve tracking long-term app performance. For the latter it may be better to decide upon some reasonable defaults (e.g., there seems to be a consensus that RUBY_HEAP_SLOTS_GROWTH_FACTOR should be 1 for a Rails app).

Contributor

rylwin commented Feb 3, 2012

I think it's possible to select defaults that, though not optimized for your specific rails app, would yield better performance than the ruby defaults. For example, just copying/pasting the config given in the link in the OP led to ruby creating 3 heaps on startup instead of the 9 it does by default--big performance boost. I then tweaked further and was able to get down to creating a single heap.

I got the relevant data to tune using https://github.com/mogest/scrap, which gives you GC stats by going to http://yoururl.com/stats/scrap. I suppose I could pull out the relevant stats provided by scrap into a rake task that could load the environment to get to the same GC data (since seeing the heap stats on startup seems to be enough to tweak the settings) and provide a recommendation based on that.

I would think the recommendation would be based on getting down to allocating a single heap on startup with some additional buffer (~10%?). To do this, one just needs to count the number of objects allocated when the Rails app starts and add a bit of buffer to that number--that would handle what's probably the most important GC setting and could be provided by the rake task.

The other settings are either based off of this number or would involve tracking long-term app performance. For the latter it may be better to decide upon some reasonable defaults (e.g., there seems to be a consensus that RUBY_HEAP_SLOTS_GROWTH_FACTOR should be 1 for a Rails app).

@technicalpickles

This comment has been minimized.

Show comment Hide comment
@technicalpickles

technicalpickles May 14, 2012

Contributor

Sorry for the belated update, but support for setting these via apache's envvars was added back in 20e16c7.

I would still like to figure something out for defaults eventually, or at least some automated/methodical way to do tuning. I think we got enough to close this out for now though :)

Contributor

technicalpickles commented May 14, 2012

Sorry for the belated update, but support for setting these via apache's envvars was added back in 20e16c7.

I would still like to figure something out for defaults eventually, or at least some automated/methodical way to do tuning. I think we got enough to close this out for now though :)

@rylwin

This comment has been minimized.

Show comment Hide comment
@rylwin

rylwin May 26, 2012

Contributor

Awesome. Thanks for letting me know!

When I get the time I'll play around with setting the envvars via apache to make sure it works (for some reason I feel like I tried this once and didn't get it to work...but maybe I wasn't setting it right).

Automatic tuning would be cool. I guess the general concept would be to have a script that would launch a server instance (preferably in the production mode) and make a request to see how many objects were created to get the app server up and running. Then this number could be used to back into some reasonable settings.

Contributor

rylwin commented May 26, 2012

Awesome. Thanks for letting me know!

When I get the time I'll play around with setting the envvars via apache to make sure it works (for some reason I feel like I tried this once and didn't get it to work...but maybe I wasn't setting it right).

Automatic tuning would be cool. I guess the general concept would be to have a script that would launch a server instance (preferably in the production mode) and make a request to see how many objects were created to get the app server up and running. Then this number could be used to back into some reasonable settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment