New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change Default Primary Keys to BIGINT #26266

Merged
merged 2 commits into from Dec 5, 2016

Conversation

@jmccartie
Contributor

jmccartie commented Aug 24, 2016

Friends don't let friends use INT as a primary key.

— Schneems (@schneems) May 13, 2016

Summary

Per a conversation with @sgrif: changes default primary keys from Integer to BIGINT for both Postgresql and MySQL. Leaves behavior alone for SQLite since this database does not provide support for BIGINT primary keys.

Other Information

For obvious reasons, this also requires foreign keys to change from integer to bigints. As a result the test suite's schema.rb has been change in the necessary places.

I'll squash and add a CHANGELOG entry once the rest looks ok...

@rails-bot

This comment has been minimized.

Show comment
Hide comment
@rails-bot

rails-bot Aug 24, 2016

Thanks for the pull request, and welcome! The Rails team is excited to review your changes, and you should hear from @kaspth (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

This repository is being automatically checked for code quality issues using Code Climate. You can see results for this analysis in the PR status below. Newly introduced issues should be fixed before a Pull Request is considered ready to review.

Please see the contribution instructions for more information.

rails-bot commented Aug 24, 2016

Thanks for the pull request, and welcome! The Rails team is excited to review your changes, and you should hear from @kaspth (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

This repository is being automatically checked for code quality issues using Code Climate. You can see results for this analysis in the PR status below. Newly introduced issues should be fixed before a Pull Request is considered ready to review.

Please see the contribution instructions for more information.

@sgrif sgrif assigned sgrif and unassigned kaspth Aug 24, 2016

@sgrif

This comment has been minimized.

Show comment
Hide comment
@sgrif

sgrif Aug 24, 2016

Member

We'll need to make sure that migrations written against 5.0 and later don't have the type of their primary key changed.

Member

sgrif commented Aug 24, 2016

We'll need to make sure that migrations written against 5.0 and later don't have the type of their primary key changed.

@jmccartie

This comment has been minimized.

Show comment
Hide comment
@jmccartie

jmccartie Aug 24, 2016

Contributor

@sgrif Can you recommend a best-practice for ensuring that? Have we done something like that in the past that I can learn from?

Contributor

jmccartie commented Aug 24, 2016

@sgrif Can you recommend a best-practice for ensuring that? Have we done something like that in the past that I can learn from?

@jmccartie

This comment has been minimized.

Show comment
Hide comment
@jmccartie

jmccartie Aug 24, 2016

Contributor

@sgrif Thanks.

Contributor

jmccartie commented Aug 24, 2016

@sgrif Thanks.

@jmccartie

This comment has been minimized.

Show comment
Hide comment
@jmccartie

jmccartie Aug 24, 2016

Contributor

@sgrif So override the necessary methods inside class V5_0 ?

Contributor

jmccartie commented Aug 24, 2016

@sgrif So override the necessary methods inside class V5_0 ?

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Aug 24, 2016

Member

This is a dup of #24962. While this implementation may be more complete it is really a bad OSS etiquette to finish other people PR without given them a chance to finish it or proper credits.

My recommendation:

  1. pull #24962 commits in this PR.
  2. Work on top of that commits
Member

rafaelfranca commented Aug 24, 2016

This is a dup of #24962. While this implementation may be more complete it is really a bad OSS etiquette to finish other people PR without given them a chance to finish it or proper credits.

My recommendation:

  1. pull #24962 commits in this PR.
  2. Work on top of that commits
@jmccartie

This comment has been minimized.

Show comment
Hide comment
@jmccartie

jmccartie Aug 24, 2016

Contributor

@rafaelfranca Ah - thanks for showing me that. I searched for open PR's, but that didn't turn up for me.

I work at Heroku with @rwz, so I'll cycle around with him and see if we can tag team this.

Contributor

jmccartie commented Aug 24, 2016

@rafaelfranca Ah - thanks for showing me that. I searched for open PR's, but that didn't turn up for me.

I work at Heroku with @rwz, so I'll cycle around with him and see if we can tag team this.

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca
Member

rafaelfranca commented Aug 24, 2016

👍

@jmccartie jmccartie closed this Aug 24, 2016

@jmccartie jmccartie reopened this Aug 24, 2016

@jmccartie

This comment has been minimized.

Show comment
Hide comment
@jmccartie

jmccartie Aug 24, 2016

Contributor

I've now entered into the 3rd layer of git rebase hell...

Contributor

jmccartie commented Aug 24, 2016

I've now entered into the 3rd layer of git rebase hell...

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Aug 24, 2016

Member

I've now entered into the 3rd layer of git rebase hell...

lol. This is really tough.

Member

rafaelfranca commented Aug 24, 2016

I've now entered into the 3rd layer of git rebase hell...

lol. This is really tough.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth
Member

kaspth commented Aug 24, 2016

r? @sgrif

@schneems

This comment has been minimized.

Show comment
Hide comment
@schneems

schneems Aug 25, 2016

Member

Looks like tests are passing now

Member

schneems commented Aug 25, 2016

Looks like tests are passing now

@sgrif

View changes

Show outdated Hide outdated activerecord/lib/active_record/migration/compatibility.rb
@jmccartie

This comment has been minimized.

Show comment
Hide comment
@jmccartie

jmccartie Aug 25, 2016

Contributor

@sgrif Added a test for MySQL compatibility. The logic still queues off the adapter name, though. We could move that logic into a default on each adapter, but then we have some sort of "legacy_primary_key" method on the adapter itself, which I liked much less than the ternary.

Thoughts?

Contributor

jmccartie commented Aug 25, 2016

@sgrif Added a test for MySQL compatibility. The logic still queues off the adapter name, though. We could move that logic into a default on each adapter, but then we have some sort of "legacy_primary_key" method on the adapter itself, which I liked much less than the ternary.

Thoughts?

@sgrif

This comment has been minimized.

Show comment
Hide comment
@sgrif

sgrif Aug 26, 2016

Member

Can we just make the PG adapter do the right thing when the PK type is integer?

Member

sgrif commented Aug 26, 2016

Can we just make the PG adapter do the right thing when the PK type is integer?

@jmccartie

This comment has been minimized.

Show comment
Hide comment
@jmccartie

jmccartie Aug 26, 2016

Contributor

@sgrif I'm not sure. Both adapters need the logic since we need to ensure MySQL uses integer (and not bigint) and Postgresql uses serial (and not bigserial)

Contributor

jmccartie commented Aug 26, 2016

@sgrif I'm not sure. Both adapters need the logic since we need to ensure MySQL uses integer (and not bigint) and Postgresql uses serial (and not bigserial)

@sgrif

This comment has been minimized.

Show comment
Hide comment
@sgrif

sgrif Aug 26, 2016

Member

How about just :integer?

On Thu, Aug 25, 2016 at 11:04 PM Jon McCartie notifications@github.com
wrote:

@sgrif https://github.com/sgrif Something like this?

options[:id] ||= connection.class::LEGACY_PK_TYPE


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#26266 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABdWK2-R2Rj-aRqV77WLaJaf0AC5MeBYks5qjlfYgaJpZM4JrgwZ
.

Member

sgrif commented Aug 26, 2016

How about just :integer?

On Thu, Aug 25, 2016 at 11:04 PM Jon McCartie notifications@github.com
wrote:

@sgrif https://github.com/sgrif Something like this?

options[:id] ||= connection.class::LEGACY_PK_TYPE


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#26266 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABdWK2-R2Rj-aRqV77WLaJaf0AC5MeBYks5qjlfYgaJpZM4JrgwZ
.

@jmccartie

This comment has been minimized.

Show comment
Hide comment
@jmccartie

jmccartie Aug 26, 2016

Contributor

@sgrif

Because Postgresql needs serial, which is an ...

image

.... oh, right ... an integer.

Done.

Contributor

jmccartie commented Aug 26, 2016

@sgrif

Because Postgresql needs serial, which is an ...

image

.... oh, right ... an integer.

Done.

@matthewd

This comment has been minimized.

Show comment
Hide comment
@matthewd

matthewd Aug 27, 2016

Member

Pulling over my concern from #24962 (comment):

Forcing explicit type declarations on FKs is not going to fly, though... and migration versioning is not enough to solve that one for people who are upgrading: they're going to end up with some tables using int4 and some using int8 PKs.

Member

matthewd commented Aug 27, 2016

Pulling over my concern from #24962 (comment):

Forcing explicit type declarations on FKs is not going to fly, though... and migration versioning is not enough to solve that one for people who are upgrading: they're going to end up with some tables using int4 and some using int8 PKs.

@jmccartie

This comment has been minimized.

Show comment
Hide comment
@jmccartie

jmccartie Aug 27, 2016

Contributor

@matthewd I get that. Any idea on how to resolve that? Ditch it and go the rails:upgrade path like you suggested in #24962? Other?

Contributor

jmccartie commented Aug 27, 2016

@matthewd I get that. Any idea on how to resolve that? Ditch it and go the rails:upgrade path like you suggested in #24962? Other?

@jmccartie jmccartie closed this Aug 28, 2016

@jmccartie jmccartie reopened this Aug 28, 2016

@rafaelfranca

View changes

Show outdated Hide outdated ...verecord/lib/active_record/connection_adapters/abstract_mysql_adapter.rb
@grosser

This comment has been minimized.

Show comment
Hide comment
@grosser

grosser May 2, 2017

Contributor

loading my old schema now blows up because foreign keys don't match:

ActiveRecord::MismatchedForeignKey: Column `environment_id` on table `deploy_groups` has a type of `int(11)`.
This does not match column `id` on `environments`, which has type `bigint(20)`.
To resolve this issue, change the type of the `environment_id` column on `deploy_groups` to be :integer. (For example `t.integer environment_id`).
Original message: Mysql2::Error: Can't create table 'samson_test.#sql-3b_2c' (errno: 150): ALTER TABLE `deploy_groups` ADD CONSTRAINT `fk_rails_09c461f421`
FOREIGN KEY (`environment_id`)
  REFERENCES `environments` (`id`)

any way of reverting this default so I can upgrade and migrate after ?

Contributor

grosser commented May 2, 2017

loading my old schema now blows up because foreign keys don't match:

ActiveRecord::MismatchedForeignKey: Column `environment_id` on table `deploy_groups` has a type of `int(11)`.
This does not match column `id` on `environments`, which has type `bigint(20)`.
To resolve this issue, change the type of the `environment_id` column on `deploy_groups` to be :integer. (For example `t.integer environment_id`).
Original message: Mysql2::Error: Can't create table 'samson_test.#sql-3b_2c' (errno: 150): ALTER TABLE `deploy_groups` ADD CONSTRAINT `fk_rails_09c461f421`
FOREIGN KEY (`environment_id`)
  REFERENCES `environments` (`id`)

any way of reverting this default so I can upgrade and migrate after ?

@jmccartie

This comment has been minimized.

Show comment
Hide comment
@jmccartie

jmccartie May 2, 2017

Contributor

@grosser You'll want to either 1) upgrade your existing columns, or 2) Change the migration to force an int primary key.

  create_table :users, id: :integer do
Contributor

jmccartie commented May 2, 2017

@grosser You'll want to either 1) upgrade your existing columns, or 2) Change the migration to force an int primary key.

  create_table :users, id: :integer do

grosser added a commit to zendesk/samson that referenced this pull request May 2, 2017

remove foreign keys
we only used 1 so pretty inconsistent / surprising ... also breaks rails 5.1 migrations see
rails/rails#26266 (comment)

@grosser grosser referenced this pull request May 2, 2017

Merged

remove foreign keys #1960

grosser added a commit to zendesk/samson that referenced this pull request May 2, 2017

remove foreign keys
we only used 1 so pretty inconsistent / surprising ... also breaks rails 5.1 migrations see
rails/rails#26266 (comment)

@zwolf zwolf referenced this pull request Jun 20, 2017

Merged

Rails5 upgrade #40

@lowjoel lowjoel referenced this pull request Jul 28, 2017

Open

ActiveRecord 5.1 Support #13

abicky added a commit to abicky/ridgepole that referenced this pull request Aug 19, 2017

Use also bigint as the default type of references
The default type of references is bigint in Rails 5.1.
cf. rails/rails#26266

abicky added a commit to abicky/ridgepole that referenced this pull request Aug 19, 2017

Use also bigint as the default type of references
The default type of references is bigint in Rails 5.1.
cf. rails/rails#26266

suginoy added a commit to suginoy/rails that referenced this pull request Oct 28, 2017

[ci skip]Update the documentation about the primary key type
Replace the primary key type `integer` in docs with `bigint`.

ref #26266

albertoalmagro added a commit to albertoalmagro/rails that referenced this pull request Dec 3, 2017

Include reference key type exception for SQLite
Before, all primary keys had type integer by default and the
documentation reflected that. After #26266 MySQL and PostgreSQL
started using +:bigint+ as default type, while SQLite continued
using +:integer+ as big integer is not supported.

At #31007 documentation was updated to +:bigint+, but this
was not accurate because SQLite does not support it. This
commit reflects this exception.
@arielscherman

This comment has been minimized.

Show comment
Hide comment
@arielscherman

arielscherman Mar 28, 2018

What if you just can't change the schema? I am facing the same issue, as described here and changing the schema is not an option for me.
What other thing can I do to avoid the mismatch exception?

arielscherman commented Mar 28, 2018

What if you just can't change the schema? I am facing the same issue, as described here and changing the schema is not an option for me.
What other thing can I do to avoid the mismatch exception?

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Mar 28, 2018

Member

You don't need to change the schema. Just that new tables will have bigint ids by default but you can still specify. After upgrading your application to 5.1, make sure you dump the new schema.rb. After that you will not have that problem anymore.

Member

rafaelfranca commented Mar 28, 2018

You don't need to change the schema. Just that new tables will have bigint ids by default but you can still specify. After upgrading your application to 5.1, make sure you dump the new schema.rb. After that you will not have that problem anymore.

@arielscherman

This comment has been minimized.

Show comment
Hide comment
@arielscherman

arielscherman Mar 28, 2018

The thing is that I am loading a schema from another Rails 4.0 app that's currently working.
So when I load the schema to create a temp database on runtime (for testing), it fails because I'm doing it on Rails 5.1 with a schema generated with Rails 4.0, so Primary Keys are BigInts, but Foreign Keys are not.

arielscherman commented Mar 28, 2018

The thing is that I am loading a schema from another Rails 4.0 app that's currently working.
So when I load the schema to create a temp database on runtime (for testing), it fails because I'm doing it on Rails 5.1 with a schema generated with Rails 4.0, so Primary Keys are BigInts, but Foreign Keys are not.

@arielscherman

This comment has been minimized.

Show comment
Hide comment
@arielscherman

arielscherman Mar 28, 2018

I am not having issues with the schema generated in Rails 5.0. My issues are for schemas generated on v4, but being loaded on v5.1. I am not (and I cannot) updating tables. I am just loading a schema file from another app, and trying to create a database from it.

arielscherman commented Mar 28, 2018

I am not having issues with the schema generated in Rails 5.0. My issues are for schemas generated on v4, but being loaded on v5.1. I am not (and I cannot) updating tables. I am just loading a schema file from another app, and trying to create a database from it.

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Mar 28, 2018

Member

The schema generated by a Rails 4 application can't be interpreted in the same way in Rails 5. We don't support that. You seems to be better off with the strucutre.sql format.

Member

rafaelfranca commented Mar 28, 2018

The schema generated by a Rails 4 application can't be interpreted in the same way in Rails 5. We don't support that. You seems to be better off with the strucutre.sql format.

@zachaysan

This comment has been minimized.

Show comment
Hide comment
@zachaysan

zachaysan May 8, 2018

Contributor

Just a quick anecdote regarding a project I'm on that started before this PR:

The team had missed that new keys were in bigint form, but everything continued to work as expected even though the foreign keys were the wrong size. Rails does application foreign keys, not DB level, which I suspect is why everything kept working. Updating the foreign keys to match the new size of the primary keys was fairly easy, but if I hadn't caught it before they scaled up to multiple DB servers, etc, they could have been burned or at least had a much more painful migration since it would have taken much longer.

Most people starting a new rails project will probably miss that bigint is the new normal for ids, and if they're writing their own migrations they'll continue to type :integer thinking that that is what foreign keys should be typed.

What I recommend is to have a warning when the migration runs if a table is created with a non-bigint type for a column that ends in _id this way full control is available to the developer, but the mismatch wont burn people just as they're starting to scale, and realistically almost all foreign keys end in _id.

Regardless, I'm a huge fan of this PR. Thanks @jmccartie, @rwz, and everyone else involved.

Contributor

zachaysan commented May 8, 2018

Just a quick anecdote regarding a project I'm on that started before this PR:

The team had missed that new keys were in bigint form, but everything continued to work as expected even though the foreign keys were the wrong size. Rails does application foreign keys, not DB level, which I suspect is why everything kept working. Updating the foreign keys to match the new size of the primary keys was fairly easy, but if I hadn't caught it before they scaled up to multiple DB servers, etc, they could have been burned or at least had a much more painful migration since it would have taken much longer.

Most people starting a new rails project will probably miss that bigint is the new normal for ids, and if they're writing their own migrations they'll continue to type :integer thinking that that is what foreign keys should be typed.

What I recommend is to have a warning when the migration runs if a table is created with a non-bigint type for a column that ends in _id this way full control is available to the developer, but the mismatch wont burn people just as they're starting to scale, and realistically almost all foreign keys end in _id.

Regardless, I'm a huge fan of this PR. Thanks @jmccartie, @rwz, and everyone else involved.

@pyrabbit

This comment has been minimized.

Show comment
Hide comment
@pyrabbit

pyrabbit Aug 8, 2018

This change caught me by surprise when migrating from 5.0 -> 5.1.

I couldn't simply change the column type without removing the existing foreign keys. Errors were being raised from MySql preventing me from changing columns that had foreign keys. Obviously this is by design. I am not sure if this is the most efficient way possible but I had to do the following:

  1. remove all existing foreign keys using remove_foreign_key
  2. change all the row id's and reference id's to bigint making sure I also added auto_increment: true on the primary keys [I am using MySql 5.5.4].
  3. create foreign keys again

This adds up to one fairly large migration so becareful as the add_foreign_key and remove_foreign_key methods do not work when running rails db:rollback

If there was a more efficient way to do this, please share.

pyrabbit commented Aug 8, 2018

This change caught me by surprise when migrating from 5.0 -> 5.1.

I couldn't simply change the column type without removing the existing foreign keys. Errors were being raised from MySql preventing me from changing columns that had foreign keys. Obviously this is by design. I am not sure if this is the most efficient way possible but I had to do the following:

  1. remove all existing foreign keys using remove_foreign_key
  2. change all the row id's and reference id's to bigint making sure I also added auto_increment: true on the primary keys [I am using MySql 5.5.4].
  3. create foreign keys again

This adds up to one fairly large migration so becareful as the add_foreign_key and remove_foreign_key methods do not work when running rails db:rollback

If there was a more efficient way to do this, please share.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment