Sentry doesn't seem to be recording errors when started via Rails runner

[nfm]

It seems like Sentry doesn't record exceptions for some reason when running code via Rails runner. I've put together a minimal example Rails app to reproduce the issue: https://github.com/nfm/raven_rails_runner_bug

Basically, running any code via Rails runner that causes an exception (eg. ./bin/rails runner '1 / 0') isn't then capturing the exception for me. This happens in production, where other kinds of exceptions are being recorded correctly (from the webserver, but also from rake tasks and delayed jobs). It also happens in development if I configure Raven's environments to include development.

The example repo should be pretty straightforward but if not let me know, I'm happy to clarify what's going wrong however I can. Cheers :)

[nateberkopec]

Thanks for the reproducing case, I'll take a look.

[nateberkopec]

It looks like we don't really have a clear place to "hook in" Sentry in this context. We automatically load in Rake and Rack Middlewares, neither of which is actually used when executing via the rails runner. Obviously, you can just modify all of your rails runner scripts to use a Raven.capture block like so:

./bin/rails runner 'Raven.capture { 1 / 0 }'

I'm not sure if there's anything we can do about this.

[dcramer]

We do this in Python by monkey patching the base code for command runners.

Airbrake seems to do something that might be related:

https://github.com/airbrake/airbrake/blob/master/lib/airbrake/rails3_tasks.rb

Though its not clear to me if instance_eval actually has to do with this. If we had some example code of how one of these commands is generated I imagine we can come up with something clever, even if it's a hook that you have to inject.