Permalink
Browse files

Removing DB column without downtime: cleanup, dropping indexes first

Based on an email from Heroku PG support:

> I don't think the transaction was at fault at all - dropping columns requires holding some heavy locks that mean other work on the table is unlikely to proceed. Generally, I'd recommend dropping each index touching the column with DROP INDEX CONCURRENTLY before dropping the column itself.
>
> The DROP column commands requiring such an exclusive lock means that it can queue up behind other operations running against the database at that time. For example, if an analytical query is running, the DROP can queue behind that query, even if that query takes several hours.
>
> I suspect the timing of the DROP query interacted badly with another query running at the same time, causing the locking you saw.
>
> Removing transactions when dropping columns is not advised - it wouldn't have helped (as far as I can tell) in this case, it's a timing issue only.
>
> Thanks
>
> Tom Crayford
> Heroku Postgres
  • Loading branch information...
henrik committed May 26, 2017
1 parent c342852 commit 0ec78de6fd3b36aa9a6349b05e1d952be747718c
Showing with 12 additions and 7 deletions.
  1. +12 −7 deploy_without_downtime/README.md
@@ -45,9 +45,11 @@ Specifically:
* **Removing columns** is never safe.
The old app will attempt to use the cached column name and will break things.
Deploy removals in two steps:
Deploy removals in multiple steps:
* (Optional) Deploy 0: Make the column nullable if it isn't already:
* Make sure the column is not actually in use by any code.
* (Optional) Deploy 1: Make the column nullable if it isn't already (being aware that [it's dangerous on big tables](http://stackoverflow.com/q/42070628/6962):
```ruby
class MyMigration < ActiveRecord::Migration
@@ -59,25 +61,28 @@ Specifically:
This is so you can safely ignore the column without breaking the creation of new records.
* Deploy 1: Make the app ignore the column:
* Deploy 2: Make the app ignore the column:
``` ruby
class Item < ActiveRecord::base
# Assuming we use https://github.com/henrik/fixme – change the date to some future one
FIXME "2015-12-01: Don't forget to remove this code when the column is gone"
def self.columns
super.reject { |c| c.name == "description" }
end
# Or, if the project includes this convenience method:
ignore_column :description
# … the rest of the class
end
```
Put that method **at the very top of the class** or you risk errors like "undefined method `type' for nil:NilClass".
* Deploy 2: A migration to remove the column. The old app will no longer have the column name cached.
* Deploy 3: A migration to remove the column. The old app will no longer have the column name cached.
In a big table, you should drop each index touching the column with `DROP INDEX CONCURRENTLY` (TODO: provide Ruby code for this) before dropping the column itself, to avoid locking issues.
* Deploy 3: Remove the code that ignored the column. If migrations run before the app code reloads (e.g. not on a standard Heroku setup), step 2 and 3 can be combined in one.
* Deploy 4: Remove the code that ignored the column. If migrations run before the app code reloads (e.g. not on a standard Heroku setup), step 2 and 3 can be combined in one.
If you get "PG::InFailedSqlTransaction" errors, you may be on Rails 4 and need [this monkeypatch](https://github.com/rails/rails/issues/12330#issuecomment-244930976).

0 comments on commit 0ec78de

Please sign in to comment.