Basic support for adding and removing foreign keys

[senny]

We had this as a GSoC idea but there was no accepted student to work on it. Because I was really looking forward to see this in rails/rails I made a first draft.

This patch adds basic support for adding and removing foreign keys using the migration DSL:

add_foreign_key :from, :to # will add the key to `to_id`
add_foreign_key :from, :to, column: "fk_id" # with specific column

add_foreign_key :from, :to, on_delete: :restrict # to specify ON DELETE RESTRICT
add_foreign_key :from, :to, on_delete: :cascade # to specify ON DELETE CASCADE
add_foreign_key :from, :to, on_delete: :nullify # to specify ON DELETE SET NULL

add_foreign_key :from, :to, on_update: :cascade # to specify ON UPDATE CASCADE
add_foreign_key :from, :to, on_update: :nullify # to specify ON UPDATE SET NULL

add_foreign_key :from, :to, name: "my_fk" # with specific name
add_foreign_key :from, :to, primary_key: "special_pk" # will reference `special_pk` from `:to`

Removing a foreign key works mostly the same:

remove_foreign_key :from, :to # will remove foreign key on `to_id`
remove_foreign_key :from, column: "fk_id" # will remove foreign key on `fk_id`
remove_foreign_key :from, name: "my_fk" # will remove foreign key named `my_fk`

Foreign keys are dumped after the indexes to schema.rb. The options name and column are always dumped. dependent only when set.

add_foreign_key is reversible, remove_foreign_key is not.

There is a connection#supports_foreign_keys? method to determine wether the adapter does support the API.

TODO:

• Write documentation
• Decide wether add-/remove_foreign_key should be no-op when supports_foreign_keys? is false
• Make sure it works with automatic test schema maintenance (ActiveRecord::Migration.maintain_test_schema! doesn't work with structure.sql #15394)
• Schema dumping must ensure the tables are created in the right order
• Handle long identifiers.
• rename dependent to on_delete and introduce on_update
• rename when a table / column is renamed. (because of digest names, this is no longer needed)

Future Work (not this PR):

• Support foreign keys when creating a table (This would allow SQLite to use FK's as well)

This patch was hugely inspired by foreigner. Credits to @matthuhiggins ❤️

[rafaelfranca]

Thank you. One thing to make work is the automatic test schema could you check if it is working now? I know it doesn't with foreigner

[fxn]

I have not checked the patch but something in the example code caught my attention.

Foreign keys have ON DELETE CASCADE, and ON UPDATE CASCADE. While :dependent resonates to associations, I see a few drawbacks:

• It is suspicious that there's only one possible value (other than nil).
• It doesn't generalize to ON UPDATE CASCADE.

What about two explicit flags on_delete_cascade: true, and on_update_cascade?

Coincidentally I have heavily used ON UPDATE CASCADE recently in a database merge where I had to move IDs.

[senny]

@fxn the current implementation does not allow for ON UPDATE specifications. The ON DELETE case is broader than the examples:

          def dependency_sql(dependency)
            case dependency
              when :nullify then "ON DELETE SET NULL"
              when :delete  then "ON DELETE CASCADE"
              when :restrict then "ON DELETE RESTRICT"
              else ""
            end
          end

A simple boolean is not enough. This also varies per adapter. PostgreSQL for example has a default of NO ACTION.

[fxn]

Ah, I see didn't remember that.

Why isn't ON UPDATE supported?

[senny]

@fxn because I did not implement it yet 😅

[fxn]

Haha :).

So yes, at the database schema level on_update and on_delete seem to be a better vocabulary to me than :dependent. Or you could have both, but if the DSL has only :dependent I see no equivalent choice for ON UPDATE.

[senny]

@fxn agreed. Will update accordingly.

[matthuhiggins]

I might as well chime in!

• I regret using :dependent as the option name. Since the option was named after has_many's option of the same name, it created expectations for users that it would work the same. :on_delete is much better.
• I always turned down any support for on_update because I viewed foreign keys as a database constraint rather than a way to change the behavior of the application. foreigner did support this through :options: https://github.com/matthuhiggins/foreigner#database-specific-options
• Similar to above, there will always be database specific constraint types, so you'll probably need the :options option regardless. (Example: Support for Postgres NOT VALID options for foreign key constraints matthuhiggins/foreigner#136)
• This is your first and last time to decide on the default constraint name when no name is provided. "foreign_key_#{from_name}_on_#{column_names * '_'}" better mirrors the naming convention used by add_index.
• Since add_foreign_key supplants the use of add_index in many cases, users will expect the foreign key methods to work much like the index migration methods. This goes for things such as creating a foreign key during create_table, along with renaming foreign keys.
• Related to above, it's important to note that adding a foreign key to a column without an index will introduce a new index to that column. This results in some annoyances:

change_table :widgets do |t|
  t.foreign_key :factories
end
# widgets.factory_id now has a foreign key and an index.

change_table :widgets do |t|
  t.remove_foreign_key :factories
end
# widgets still has an index on factory_id

• The whole sqlite thing is probably worth addressing as a different beast. I have not checked recently, but I believe that the only way to add a foreign key to an existing sqlite table is to drop and recreate the table. I'm sure that this could be done automatically, but it would probably convoluted.

Anyways, those are the things I learned over the last 5 years of trying to keep foreigner as simple as possible. You can get a general gist of problems that you'll need to address or say no to by looking at https://github.com/matthuhiggins/foreigner/issues.

[senny]

@matthuhiggins thanks so much for sharing your insight. Will investigate further to provide a sound starting point. I mostly agree with your points, I'm skeptical about the fk naming though. We had many issues because identifiers that got too long. I'd like to stick to a short naming when possible. Any thoughts on that?

[fxn]

Regarding ON UPDATE my take is that databases are who define what are FK constraints for. It's their field rather than ours.

FKs protect you from orphan records in the common case, but also allow you to define what to do when something happens to the parent record. In the use case I mentioned before, I need to issue dozens of statements like this:

UPDATE foos SET id = id + delta;

and ON UPDATE CASCADE is the blessed way to keep the FKs in children tables in sync automatically.

I believe that if we add support for FKs then we need to adapt to whatever FKs mean to database writers.

[matthuhiggins]

Your current naming is completely fine too. Yes, I am aware of the naming limits, in particular with postgres. It is less likely to occur with foreign keys, since it is on only one column rather than multiple. Your choice is completely fine (hey now, it was my choice too). There is also fk_#{table_name}_on_#{column_name}.

Great work btw. I hope that some form of this makes it in.

[matthuhiggins]

@fxn That is completely reasonable. I was stating where I personally drew the line, and therefore did not support. Similarly, I never supported composite foreign keys. Per your philosophy, should composite foreign keys be supported?

@senny Make sure that this issue does not exist with your new implementation: matthuhiggins/foreigner#110

[senny]

@matthuhiggins thanks for the issue reference. I think this is the behavior @rafaelfranca was talking about. As we now have automatic schema maintenance for test runs, this is even more important. I haven't settled on a solution yet but will investigate dropping the database and not the tables.

[fxn]

@matthuhiggins Oh absolutely, regading your feedback. Just to be clear I was also talking from the point of view of a team member thinking about a new feature in Rails, without implying anything about your choices in foreigner, since that is your baby and I totally respect your choices there, couldn't be otherwise :).

Regarding composite FKs, I have never used them. In principle I'd say yes from the point of view of Active Record. If there's support for FKs I'd expect support for multiple columns. But then it also depends on practical matters, does it complicate the interface, is it worthwhile? So I'd say yes unless there are practical trade-offs that suggest not to support it.

In the case of ON UPDATE what I see is that it is totally analogous to ON DELETE. I imagine the implementation should be similar, and indeed treating both as equal (as they are in SQL) may even influence naming/interface.

[egilburg]

we already use foreigner in our code. is this patch fully compatible with existing code that uses foreigner's syntax?

[egilburg]

add_foreign_key is reversible, remove_foreign_key is not.

why not? remove_column is reversible in rails 4 provided it was defined with the params that add_column would usually require

[egilburg]

foreigner accepts arbitrary options like add_foreign_key(:comments, :posts, options: 'ON UPDATE DEFERRED'). WDYT of supporting that?

[senny]

I will probably add something of that sort but for now, I'd like to get the basic functionality working and merged.

[egilburg]

And of course, much ❤️ for proposing this to core Rails.

I always thought that "getting started quickly" and "convention over configuration" doesn't mean "throw basic sane db design out the window". Not having db-level foreign key constraints and only relying on validations/assocs just means a world of hurt when you have to deal with production maintenance, changing application code, data migrations, etc.

So IMO this is a much needed patch and should also have docs that encourage it's use to the same tune as indexes and null constraints.

[egilburg]

Also, one thing that bit me with foreigner is that rename_table, rename_column and friends doesn't rename existing foreign key names. This is especially painful when a db refactor results in a completely different table being named what another table was before. Is this something to consider tracking?

[senny]

@fxn Why do you think we need to provide a complete solution for FK's? When I look at other methods from the migration DSL it becomes obvious that we only support a small subset of what would be possible. You can always drop down to execute.

I'm not opposed to build to a more complete solution but I think we should be shipping this as soon as we hit 80 / 20.

[fxn]

@senny note that my answer had some ifs :).

[senny]

@egilburg there is basic compatibility between the current implementation and foreigner. The name of the options might vary (currently we have :on_delete and not :dependent). You should be able to migrate over though:

1.) remove foreigner
2.) remove legacy migrations
3.) dump a new schema.rb to have the correct syntax reflected

I will look into the renaming stuff. I know we do this for indexes.

[matthewd]

else ArgumentError ?

[VanTanev]

One thing possibly worth considering: composite key foreign keys.

[rafaelfranca]

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Really awesome work as usual 👏 👏 👏 👏

[fnando]

<3 <3 <3

Thanks a lot, @senny!

[fabianoleittes]

Good work @senny, tks! 👍

[shamanime]

Nicely done! 👍

[senny]

thanks guys ❤️

[thibaudgg]

Nice, really useful. Thanks from Switzerland @senny!

[tilsammans]

Well done, love this. If anything can we make sure all the work by @matthuhiggins is being incorporated. I use Foreigner in all apps, would be best to have a smooth transition between the two.

[senny]

@tilsammans as described above you can transition to this. There will be small differences in the API and especially in the naming of foreign keys.

A lot of this PR is based on the work of @matthuhiggins and his work on foreigner. Hopefully we can improve this first draft down the line. ❤️

[dosire]

Thanks for getting this into rails Yves! <3

[robin850]

Awesome work here! ❤️

[matthuhiggins]

Great stuff. I gladly relinquish my technical debt to Rails!

[Krule]

Finally! Thank you. ❤️