Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Fix Postgres CPU burn #573
According to Heroku, we have a query that accounts for 80% of all time spent in postgres
Where did it come from? Turns out, it's not in my code. It was added by a rails validation.
But, you might be thinking. We only have 2,000 repos in the whole database, why is it updating so much?
Well, it turns out this validation is called on EVERY update.
For example this sidekiq task:
Going to remove the validation and do it in the DB instead, which is better anyway. However it doesn't give us the nice errors of a rails validation (I don't think). Would be nice to integrate this in the future.
Having tried this route in the past in my own apps and being surprised by the effects, I'll note that the difference goes beyond the nice errors. Unless something has changed recently in ActiveRecord, the failure behavior on
Previously, if you tried to save a non-unique record, ActiveRecord would find the conflicting record, add a validation error, and and
With this change, if you try to save a non-unique record, ActiveRecord just knows that it tried to do a DB operation and it blew up, so even the non-bang version of
You seem to have accounted for some of this in the changed unit test, but I thought it would be worth noting in case this is something you (or other readers who found this PR) haven't experienced in the past. It can cause unexpected and surprising failures down the line, generally at the whole-request level. It's been bad enough for me that I decided to always match DB constraints with AR validations, but maybe performance effects are more important here. Generally, YMMV.
@amcaplan that's a great note and a great point.
In the controller action which is the only place we allow for Repo creation, i'm was already doing a search in the DB first to make sure the record didn't already exist. In theory there could be a race condition, between that and the time the record uniqueness check actually happens. In practice there's only 2K records here.
Here's the query
So basically i'm counting on the user never seeing that exception.
I actually first wrapped a bunch of
It would be cool if we could tell ActiveRecord when we've enabled a uniqueness constraint (or other constraint) and it could coerce that into behavior that is closer to what we come to expect from
The challenge there is that
In the meantime, I'm glad this has a single known use case with an extremely unlikely error. Given what I've seen, the race condition would cause a 500 with (hopefully) not much information sent back to the client, but that's a pretty tight race condition, and actually is the same race condition as ActiveModel validations would have anyway (which is why it's always recommended to pair with a DB constraint)!