Skip to content

Commit

Permalink
Fix Postgres CPU burn
Browse files Browse the repository at this point in the history
According to Heroku, we have a query that accounts for 80% of all time spent in postgres

```
$ heroku pg:outliers
 total_exec_time  | prop_exec_time |   ncalls    |   sync_io_time   |                                                                                       query
------------------+----------------+-------------+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 3790:50:52.62102 | 80.2%          | 100,727,265 | 727:08:40.969477 | SELECT  ? AS one FROM "repos" WHERE LOWER("repos"."name") = LOWER($1) AND ("repos"."id" != $2) AND "repos"."user_name" = $3 LIMIT $4
```

Where did it come from? Turns out, it's not in my code. It was added by a rails validation.

But, you might be thinking. We only have 2,000 repos in the whole database, why is it updating so much?

Well, it turns out this validation is called on EVERY update.

For example this sidekiq task:

```
Jun 29 07:00:32 issuetriage app/scheduler.8183:  [ActiveJob] Enqueued PopulateIssuesJob (Job ID: 9e04e63f-a515-4dcd-947f-0f777e56dd1b) to Sidekiq(default) with arguments: #<GlobalID:0x00000004f98a68 @uri=#<URI::GID gid://code-triage/Repo/1008>>
Jun 29 07:11:05 issuetriage app/worker.1:  [ActiveJob] [PopulateIssuesJob] [9e04e63f-a515-4dcd-947f-0f777e56dd1b] Performing PopulateIssuesJob (Job ID: 9e04e63f-a515-4dcd-947f-0f777e56dd1b) from Sidekiq(default) with arguments: #<GlobalID:0x007fd098f6bdf8 @uri=#<URI::GID gid://code-triage/Repo/1008>>
Jun 29 07:11:05 issuetriage app/worker.1:  [ActiveJob] [PopulateIssuesJob] [9e04e63f-a515-4dcd-947f-0f777e56dd1b]   User Load (10.4ms)  SELECT  "users".* FROM "users" WHERE ("users"."github_access_token" IS NOT NULL) ORDER BY RANDOM() LIMIT $1
Jun 29 07:11:05 issuetriage app/worker.1:  [ActiveJob] [PopulateIssuesJob] [9e04e63f-a515-4dcd-947f-0f777e56dd1b]    (35.4ms)  BEGIN
Jun 29 07:11:05 issuetriage app/worker.1:  [ActiveJob] [PopulateIssuesJob] [9e04e63f-a515-4dcd-947f-0f777e56dd1b]   Repo Exists (352.9ms)  SELECT  1 AS one FROM "repos" WHERE LOWER("repos"."name") = LOWER($1) AND ("repos"."id" != $2) AND "repos"."user_name" = $3 LIMIT $4
Jun 29 07:11:05 issuetriage app/worker.1:  [ActiveJob] [PopulateIssuesJob] [9e04e63f-a515-4dcd-947f-0f777e56dd1b]   SQL (3.7ms)  UPDATE "repos" SET "github_error_msg" = $1, "updated_at" = $2 WHERE "repos"."id" = $3
Jun 29 07:11:05 issuetriage app/worker.1:  [ActiveJob] [PopulateIssuesJob] [9e04e63f-a515-4dcd-947f-0f777e56dd1b]    (4.5ms)  COMMIT
Jun 29 07:11:05 issuetriage app/worker.1:  [ActiveJob] [PopulateIssuesJob] [9e04e63f-a515-4dcd-947f-0f777e56dd1b] Performed PopulateIssuesJob (Job ID: 9e04e63f-a515-4dcd-947f-0f777e56dd1b) from Sidekiq(default) in 629.22ms
```

Going to remove the validation and do it in the DB instead, which is better anyway. However it doesn't give us the nice errors of a rails validation (I don't think). Would be nice to integrate this in the future.

Ideally `validates` would see if there is a unique index and use that instead, give us a warning if no unique index existed, because race conditions.
  • Loading branch information
schneems committed Jun 29, 2017
1 parent 3cb61c0 commit d96cf44
Show file tree
Hide file tree
Showing 3 changed files with 9 additions and 2 deletions.
3 changes: 2 additions & 1 deletion app/models/repo.rb
Expand Up @@ -2,7 +2,8 @@

class Repo < ActiveRecord::Base
validate :github_url_exists, on: :create
validates :name, uniqueness: {scope: :user_name, case_sensitive: false }

# Now done at the DB level # validates :name, uniqueness: {scope: :user_name, case_sensitive: false }
validates :name, :user_name, presence: true

has_many :issues
Expand Down
5 changes: 5 additions & 0 deletions db/migrate/20170629151616_add_unique_index_to_repos.rb
@@ -0,0 +1,5 @@
class AddUniqueIndexToRepos < ActiveRecord::Migration[5.1]
def change
add_index :repos, [:name, :user_name], :unique => true
end
end
3 changes: 2 additions & 1 deletion db/schema.rb
Expand Up @@ -10,7 +10,7 @@
#
# It's strongly recommended that you check this file into your version control system.

ActiveRecord::Schema.define(version: 20170627024417) do
ActiveRecord::Schema.define(version: 20170629151616) do

# These are extensions that must be enabled in order to support this database
enable_extension "plpgsql"
Expand Down Expand Up @@ -153,6 +153,7 @@
t.integer "stars_count", default: 0
t.index ["issues_count"], name: "index_repos_on_issues_count"
t.index ["language"], name: "index_repos_on_language"
t.index ["name", "user_name"], name: "index_repos_on_name_and_user_name", unique: true
t.index ["name"], name: "index_repos_on_name"
t.index ["user_name"], name: "index_repos_on_user_name"
end
Expand Down

0 comments on commit d96cf44

Please sign in to comment.