Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using `create_or_find_by` led to primary keys running out #35543

alexcameron89 opened this Issue Mar 8, 2019 · 21 comments


None yet
8 participants
Copy link

commented Mar 8, 2019

Steps to reproduce

While this doesn't fully assert the issue, it shows that the primary key can grow tremendously without actually writing new rows.

# frozen_string_literal: true

require "bundler/inline"

gemfile(true) do
  source ""

  git_source(:github) { |repo| "{repo}.git" }

  gem "rails", github: "rails/rails"
  gem "pg"

require "active_record"
require "minitest/autorun"
require "logger"

# This connection will do for database-independent bug reports.
ActiveRecord::Base.establish_connection(adapter: "postgresql", database: "rails")
ActiveRecord::Base.logger =

ActiveRecord::Schema.define do
  create_table :posts, force: true do |t|

  create_table :comments, force: true do |t|
    t.integer :post_id

  add_index :comments, :post_id, unique: true

class Post < ActiveRecord::Base
  has_many :comments

class Comment < ActiveRecord::Base
  belongs_to :post

class BugTest < Minitest::Test

  # When create fails, the autoincremented id does not roll back
  def test_primary_key_growth
    post_one = Post.create!

    comment_for_post_one = Comment.create_or_find_by(post: post_one)

    # Create fails for Post 1's second comment and it finds the same record for
    # first_comment_for_post_one
    1000.times do
      second_comment_for_post_one = Comment.create_or_find_by(post: post_one)
      assert_equal second_comment_for_post_one, comment_for_post_one

    post_two = Post.create!
    # This fails
    # Failure:
    #   BugTest#test_primary_key_growth [create_or_find_by.rb:56]:
    #   Expected: 2
    #     Actual: #<Comment id: 1002, post_id: 2>
    comment_for_post_two = Comment.create_or_find_by(post: post_two)
    assert_equal + 1,

   # This is nothing new, as the same thing can happen with `create`
  def test_create
    post_one = Post.create!

    comment_for_post_one = Comment.create(post: post_one)

    1000.times do
      assert_raises ActiveRecord::RecordNotUnique do
        second_comment_for_post_one = Comment.create(post: post_one)

    post_two = Post.create!
    comment_for_post_two = Comment.create(post: post_two)
    assert_equal + 1,


Each time create_or_find_by hits ActiveRecord::RecordNotUnique, it rolls back, but the auto-increment does not. The fact that auto-increment does not roll back seems to be expected behavior from PostgreSQL.

Still, the fact that this happens with create_or_find_by led to one of our database tables running out of primary keys in production, preventing the creation of any new rows.

System configuration

Rails version:

Ruby version:


This comment has been minimized.

Copy link

commented Mar 9, 2019

Hey @alexcameron89 👋, glad to have you back again 😄

I'm unsure what we can really do about this. Just for completeness, is this also a problem with the old find_or_create_by with a rescue ActiveRecord::RecordNotUnique?


This comment has been minimized.

Copy link
Member Author

commented Mar 11, 2019

Thank you @kaspth, I'm glad to be back! 😄

Just for completeness, is this also a problem with the old find_or_create_by with a rescue ActiveRecord::RecordNotUnique

Yes, this will happen in the case of the data race of find_or_create_by, and in general any time a create fails at the database level. But given that it's a data race issue in find_or_create_by, the majority of uses cases are likely to be a successful find or create, leaving the failed create and incrementing primary key issue with a much smaller footprint.

My concern with create_or_find_by is that the growing primary key issue is brought front and center because failing at the database level is promoted. In the best case, create succeeds more often than not, and a growing primary key is not much of an issue. In the worst case, create fails often because the item already exists, leading to the issue we had.


I can already see that create_or_find_by may be seen as a possible jump in performance over find_or_create_by for some people, and it may be adopted without thought or knowledge of consequences of this issue that they may hit.

All in all, I feel that create_or_find_by has a dangerous side effect that can lead to large production issues. I think it might be worth considering removing the method or replacing it with a safer method (I don't know what that might be). For instance, is there much more gain from create_or_find_by than doing the retry logic that was documented before? Could that become its own method that users don't have to re-create in their applications?


This comment has been minimized.

Copy link

commented Mar 11, 2019

All in all, I feel that create_or_find_by has a dangerous side effect that can lead to large production issues.

But only on Postgres, correct? Or did SQLite and MySQL auto-increment the primary key without rolling back too? At the very least we should mention this in the documentation.

I'm not entirely sure an outright removal is the right course of action. This seems somewhat par for the course when we're talking uniqueness and Postgres. E.g. this could happen with a failed uniqueness validation that's retried as you said, create_or_find_by just exacerbates the issue. How did you work around this in your app? Was it enough to go back to find_or_create_by + rescue?

@dhh what's your take on this? As I understand it, create_or_find_by has been a fine extraction for Basecamp 3.


This comment has been minimized.

Copy link

commented Mar 11, 2019

It's certainly good to point out this gotcha in the documentation, at the very least. We use MySQL at Basecamp and are happy users of this method in production.

@alexcameron98 Can you share your concerns with ID auto-incremental growth in general? We're defaulting id columns to bigint, so we have room for 9,223,372,036,854,775,807 values in pgsql. Doesn't seem so likely that this method is going to make scarcely anyone run out of room? Even if you called this method 10 million times per day, it would take 2.5 billion years to run out of IDs 😄


This comment has been minimized.

Copy link

commented Mar 11, 2019

it would take 2.5 billion years to run out of IDs

The only logical explanation is that @alexcameron89 is speaking to us from 2.5 billion years in the future! 😄

@alexcameron89 I take it your column isn't a big int?


This comment has been minimized.

Copy link
Member Author

commented Mar 11, 2019

Column Type: Int

I take it your column isn't a big int?

That's a good callout, the table that was affected had int as primary key. In reality, this is likely to only affect int tables. But given that bigint didn't become a default until 5.0+, there's a high likelihood that this method will be used on many tables with int primary keys, and that's a little concerning.

Other affected DB's

Or did SQLite and MySQL auto-increment the primary key without rolling back too? At the very least we should mention this in the documentation.

It looks like Mysql is affected by it (script), but SQLite is not (script).


It's certainly good to point out this gotcha in the documentation, at the very least.

I'll push up a PR with a callout in the documentation.

The Future

The only logical explanation is that @alexcameron89 is speaking to us from 2.5 billion years in the future! 😄

It's true!bigbigbigint is the new hip thing these days. 😄 🤖


This comment has been minimized.

Copy link

commented Mar 11, 2019


This comment has been minimized.

Copy link

commented Mar 11, 2019

Perhaps we should deprecate having int for primary keys in Rails 6? Not sure about all the ramifications, but it could be good to something here. Or we could add a task to help generate migrations for it.


This comment has been minimized.

Copy link
Member Author

commented Mar 11, 2019

Perhaps we should deprecate having int for primary keys in Rails 6? Not sure about all the ramifications, but it could be good to something here. Or we could add a task to help generate migrations for it.

I think that's a really interesting idea. 👍


This comment has been minimized.

Copy link

commented Mar 11, 2019

We experienced an outage from create_or_find_by with a table on the order of a few million rows - big, but not growing at a rate that we'd have expected to exhaust the sequence. We had assumed the sequence roughly corresponded to the table size. Even then migrating from an int column to a bigint took a fair amount of downtime. Is anybody monitoring their database sequences? (I doubt it)

It's true that int PKs are a timebomb, but this method accelerates the clock significantly. Running out of ints with a table of a few million wasn't expected. The sequence basically becomes a query count.

I've come around to the idea that find_or_create_or_find_by is a better behavior in our case (though I'd never suggest rails having that method).

I'm dubious of the general-purpose utility of create_or_find_by after having experienced that outage. There are already 4 bullet point drawbacks listed in the documentation for this method, this would be a 5th. Better to warn people than have them discover it "naturally" as we did.


This comment has been minimized.

Copy link

commented Mar 11, 2019


This comment has been minimized.

Copy link

commented Mar 11, 2019

The race condition it was designed for IS a problem for me, but find_or_create_or_find is a less-surprising solution. The race condition was a low-grade annoyance in our application's BugSnag, and so we applied create_or_find_by thinking it's no big deal. Months later we had an outage over an hour long because of this decision.

It's true it doesn't matter as much with bigint columns. Migrating to bigint columns can take a fair amount of work depending on your system - we took out service offline to isolate the database and ran the migration and it took about 45 minutes against a few million rows - it's rewriting the table and indexes which is expensive the naive way. Zero downtime migrations take a bit of prep that you may not be prepared for if you're thinking you're not going to run out of ids because your table is a few orders of magnitude smaller than the limit.

I'm happy it works for you in Basecamp. The people that will hit this are likely to be those upgrading pre-existing apps that have not converted everything to bigints.


This comment has been minimized.

Copy link

commented Mar 12, 2019


This comment has been minimized.

Copy link

commented Mar 12, 2019

I've come around to the idea that find_or_create_or_find_by is a better behavior in our case (though I'd never suggest rails having that method).

I'd like to consider making that the behaviour of find_or_create_by, if you're interested in PRing it.

To me the ideal would be that they both have similar[ly rare] failure scenarios, and the choice of find_or_create_by vs create_or_find_by would boil down to whether you, on average, expect the row to exist.. and thus which one is more likely to save you a query.

alexcameron89 added a commit to alexcameron89/rails that referenced this issue Mar 12, 2019

Document int Primary Key with create_or_find_by
This commit addresses the issue in
rails#35543 by making note of the
growing primary key issue with `create_or_find_by`.

This comment has been minimized.

Copy link

commented Mar 12, 2019

Was looking at this, in terms of PostgreSQL, there isn't an obvious way to change anything. Figure I should post the experiment in case any others find value in it.

require 'pg'

db = PG.connect dbname: 'josh_testing'
%I[exec exec_params].each do |name|
  db.define_singleton_method name do |*args, &block|
    super(*args, &block).to_a
    $!.set_backtrace caller.drop(1)

db.exec <<~SQL
  drop table if exists omghi;
  create table if not exists omghi (
    id serial primary key,
    val text unique

insert = -> val {
  db.exec_params <<~SQL, [val]
    insert into omghi (val) values ($1)
    returning *

# insert value
insert['a']                 # => [{"id"=>"1", "val"=>"a"}]

# failing to insert increments the sequence
insert['a'] rescue $!.class # => PG::UniqueViolation
insert['b']                 # => [{"id"=>"3", "val"=>"b"}]

# an explicit transaction changes nothing
db.exec 'begin transaction'
insert['a'] rescue $!.class # => PG::UniqueViolation
db.exec 'rollback transaction'
insert['c']                 # => [{"id"=>"5", "val"=>"c"}]

# "on conflict do nothing" changes nothing
db.exec <<~SQL
  insert into omghi (val) values ('a')
  on conflict do nothing
  returning *
insert['d']                 # => [{"id"=>"7", "val"=>"d"}]

# the db's values
db.exec('select * from omghi').to_a
                            # => [{"id"=>"1", "val"=>"a"},
                            #     {"id"=>"3", "val"=>"b"},
                            #     {"id"=>"5", "val"=>"c"},
                            #     {"id"=>"7", "val"=>"d"}]

This comment has been minimized.

Copy link

commented Mar 12, 2019

actually, there is a quick workaround for int to bigint migration in postgres:

SELECT * INTO users_temp FROM users;
ALTER TABLE users ALTER COLUMN id type bigint;
INSERT INTO users SELECT * FROM users_temp;
DROP TABLE users_temp;
COMMIT; -- to release lock

Takes seconds on a table with 1M records.


This comment has been minimized.

Copy link

commented Mar 12, 2019

I'm surprised by the unwillingness to recognize the problem that this issue is attempting to address. While bigint columns certainly pushes the ultimate breaking problem down the road (billions of years, even), it doesn't make the actual problem go away (unnecessary skipping of unused primary key values).

In my opinion, it'd be similar to saying:

You think it's a problem that too many straws are being opened and discarded unnecessarily before being used, and you're upset that the lake behind your house is getting filled with straws? I'm sorry you're experiencing this, but the lake behind my house got filled a long time ago, so I understand how you feel! Just start discarding your unused straws into the ocean, it's much bigger than your lake and you won't notice they're there until you're LONG gone!

While I'm probably more OCD than most about missing chunks of primary keys in my tables, I think a quick education in the docs around the scenarios in which create_or_find_by should be used instead of find_or_create_by would be appropriate, given the potential drawbacks. Rails is awesome for its accessibility to newer developers, and I would be concerned about create_or_find_by being accidentally misused when find_or_create_by would be more appropriate (especially if it were updated to behave like find_or_create_or_find_by, as has been suggested).


This comment has been minimized.

Copy link

commented Mar 12, 2019


This comment has been minimized.

Copy link

commented Mar 12, 2019

But if you're on bigint, then it won't matter, unless you for some bizarre reason considers incrementing a counter the same as littering.

I do consider it unnecessary litter in my db. Maybe I'm the anomaly, and if so, I apologize for the inaccurate comparison.

And if that's the case, you are, as in all other cases, utterly free not to use this feature.

In the case I'm not the anomaly, my concern isn't that I would use the feature inappropriately, but that countless others might; and that they'd eventually be upset at their unintentional littering of their own db.

alexcameron89 added a commit to alexcameron89/rails that referenced this issue Mar 13, 2019

Document int Primary Key with create_or_find_by
This commit addresses the issue in
rails#35543 by making note of the
growing primary key issue with `create_or_find_by`.

This comment has been minimized.

Copy link
Member Author

commented Mar 13, 2019

I've come around to the idea that find_or_create_or_find_by is a better behavior in our case (though I'd never suggest rails having that method).

I'd like to consider making that the behaviour of find_or_create_by, if you're interested in PRing it.

@matthewd I'm working on this and trying to find a good way to test it.

dhh added a commit that referenced this issue Mar 13, 2019

Document int Primary Key with create_or_find_by (#35573)
This commit addresses the issue in
#35543 by making note of the
growing primary key issue with `create_or_find_by`.

This comment has been minimized.

Copy link
Member Author

commented Mar 15, 2019

Since the mitigation to this issue was decided to be documenting the issue, and since that was merged in #35573, I'll close this issue.

Kiku-git added a commit to Kiku-git/rails that referenced this issue Mar 31, 2019

fix (#1)
* Add changelog entry for transliterate/parameterize accepting `locale` [ci skip]

* Add note about has_many associations callbacks [ci skip]

Add a note explaining when the has_many associations callbacks will be called or not.

* After rails#35281  and rails#35036 AV warns about not having the compiled container.
Use the suggested with_empty_template_cache to overcome warning

* Show friendly message to install action mailbox if the related table does not exist

- This is similar to the work done in rails#31534

* Add additional ERB DummyCompiler tests

This adds a few additional tests to the commits by eileencodes (rails#35497) and rafaelfranca (rails@cfa22f1). The additional tests cover several more ERB tag formatting cases such as multiline tags, conditional statements that result in duplicate keys, and multiple erb statements on a single line.

* Edit a changelog entry [ci skip]

* Change wording of some instances of 'opt out' [ci skip]

* Update small typo in documentation.

* Initialize `@default_timezone` and `@timestamp_decoder` in `add_pg_decoders`

Staled `@default_timezone` would cause an error on `reconnect!` after

Steps to reproduce:

% ARCONN=postgresql bin/test test/cases/adapter_test.rb test/cases/base_test.rb -n "/(?:test_attributes_on_dummy_time|test_reconnect_after_a_disconnect)$/" --seed 15849
Using postgresql
Run options: -n "/(?:test_attributes_on_dummy_time|test_reconnect_after_a_disconnect)$/" --seed 15849

# Running:


NoMethodError: undefined method `add_coder' for #<PG::TypeMapAllStrings:0x00007f85ab9dd5b8>
    /Users/kamipo/src/ `update_typemap_for_default_timezone'
    /Users/kamipo/src/ `exec_no_cache'
    /Users/kamipo/src/ `execute_and_clear'
    /Users/kamipo/src/ `add_pg_decoders'
    /Users/kamipo/src/ `connect'
    /Users/kamipo/src/ `rescue in block in reconnect!'
    /Users/kamipo/src/ `block in reconnect!'
    /Users/kamipo/.rbenv/versions/2.6.1/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
    /Users/kamipo/src/ `reconnect!'
    /Users/kamipo/src/ `block in <class:AdapterTestWithoutTransaction>'

* Remove unused `Row` class in `SelectManager`

The `Row` class is no longer used since d956772.

* Update URL in [ci skip]

* Update valid link for `signing release tag` from `git-scm` book
* Use `en`(English) instead of `tr`(Turkish) documentation link

* Remove Marshal support from SchemaCache

YAML has been used to serialize the schema cache ever since 2016 with
Rails 5.1: 4c00c6e

* Modernize size calculation in Schema Cache

Not looking for other contributions like this, but I took the liberty
since I was already working on this.

* Schema Cache: cache table indexes

Useful to not query for indexes when an application uses schema cache.

Ref rails#35546

* Document int Primary Key with create_or_find_by (rails#35573)

This commit addresses the issue in
rails#35543 by making note of the
growing primary key issue with `create_or_find_by`.

* Add test for runner command with `environment` option

* Give GeneratedAttributeMethods module a name

Currently GeneratedAttributeMethods is a module builder class, an
instance of which is included in every AR class. OTOH,
GeneratedAssociatedMethods is assigned to a constant under the model
namespace. This is inconsistent and looks strange in the list of

There is no particular reason *not* to assign a constant for this (very
important) module under the model namespace, so that's what this commit

Previous to this change, ancestors for an AR class looked like this:

=> [User (call 'User.connection' to establish a connection),

With this change, they look like this:

=> [User (call 'User.connection' to establish a connection),

The previously named `GeneratedAttributeMethods` module builder class is
renamed `GeneratedAttributeMethodsBuilder` to emphasize that this is not
a module but a class.

* Fix rubocop violations

* [skip ci] Rails 5.1+ supports bigint primary key

Follow up rails#35573

* Indentation >>

* Fix typo s/ActiveSupport::Timezone/ActiveSupport::TimeZone/ [ci skip]

* Revert "Remove Marshal support from SchemaCache"

This reverts commit 65f2eea.

* Fix gemfile.lock

* Add secret token for action mailbox tests

We need to create a secret token in the application so that the travis
build doesn't error when trying to create a file in the tmp directory.


* Fix secret_key_base for Railties

This was missed in the security fix for local dev. CI doesn't have a tmp
directory in the apps built for testing so these end up failing. This
adds the secret_key_base so we don't need to generate one.

* support slice assignment on SafeBuffer

* Make application work without tmp directory

The tmp directory is added to version control in the newly created
application. This was added in Rails 5.0.0(rails@f06ce4c).

However, applications created before that are not guaranteed to have the
tmp directory. If the tmp directory does not exist, writing to the key file
raise error.

This is a bit incompatible. So I fixed that create the directory before
writing a key.

* GCS service: skip unnecessary bucket lookups

* Raise exception when building invalid mime type

This allows mime types in the form text/html, text/*, or */*

This required a few minor test/code changes where previously nil was
used as a mime string.

* bumps Zeitwerk

* Properly expand the environment's name in all commands

Since 3777701, the environment's name is
automatically expanded in console and dbconsole commands.
In order to match the behavior between the commands, fixes it to have the
same behavior of all the commands.

This behavior is defined in `EnvironmentArgument`. Since
`EnvironmentArgument` also defines the environment option, it is reused.

However, since desc was not content that can be used in all comments,
fixed desc to be defined for each command.

* Fix for migration active_storage migration

*ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "active_storage_attachments" does not exist*

Updating to rails 6.0.0.beta3 if the command rake app:update is used a new migration file is generated:

# db/migrate/add_foreign_key_constraint_to_active_storage_attachments_for_blob_id.active_storage.rb
class AddForeignKeyConstraintToActiveStorageAttachmentsForBlobId < ActiveRecord::Migration[6.0]
  def up
    unless foreign_key_exists?(:active_storage_attachments, column: :blob_id)
  		add_foreign_key :active_storage_attachments, :active_storage_blobs, column: :blob_id

**If the project does not have previously installed active storage** this mgiration causes an error:

    ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "active_storage_attachments" does not exist

**Error output:**

    PG::UndefinedTable: ERROR:  relation "active_storage_attachments" does not exist
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/postgresql/database_statements.rb:90:in `async_exec'
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/postgresql/database_statements.rb:90:in `block (2 levels) in execute'
    activesupport-6.0.0.beta3/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
    activesupport-6.0.0.beta3/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
    activesupport-6.0.0.beta3/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/postgresql/database_statements.rb:89:in `block in execute'
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/abstract_adapter.rb:643:in `block (2 levels) in log'
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/abstract_adapter.rb:642:in `block in log'
    activesupport-6.0.0.beta3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/abstract_adapter.rb:633:in `log'
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/postgresql/database_statements.rb:88:in `execute'
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/abstract/schema_statements.rb:977:in `add_foreign_key'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:875:in `block in method_missing'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:843:in `block in say_with_time'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:843:in `say_with_time'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:864:in `method_missing'
    /db/migrate/20190315102607_add_foreign_key_constraint_to_active_storage_attachments_for_blob_id.active_storage.rb:5:in `up'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:816:in `exec_migration'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:797:in `block (2 levels) in migrate'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:796:in `block in migrate'
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:795:in `migrate'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:985:in `migrate'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:1293:in `block in execute_migration_in_transaction'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:1344:in `block in ddl_transaction'
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `block in transaction'
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/abstract/transaction.rb:272:in `block in within_new_transaction'
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/abstract/transaction.rb:270:in `within_new_transaction'
    activerecord-6.0.0.beta3/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'
    activerecord-6.0.0.beta3/lib/active_record/transactions.rb:212:in `transaction'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:1344:in `ddl_transaction'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:1292:in `execute_migration_in_transaction'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:1264:in `block in migrate_without_lock'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:1263:in `each'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:1263:in `migrate_without_lock'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:1211:in `block in migrate'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:1364:in `with_advisory_lock'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:1211:in `migrate'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:1044:in `up'
    activerecord-6.0.0.beta3/lib/active_record/migration.rb:1019:in `migrate'
    activerecord-6.0.0.beta3/lib/active_record/tasks/database_tasks.rb:191:in `migrate'
    activerecord-6.0.0.beta3/lib/active_record/railties/databases.rake:78:in `block (3 levels) in <main>'
    activerecord-6.0.0.beta3/lib/active_record/railties/databases.rake:76:in `each'
    activerecord-6.0.0.beta3/lib/active_record/railties/databases.rake:76:in `block (2 levels) in <main>'
    railties-6.0.0.beta3/lib/rails/commands/rake/rake_command.rb:23:in `block in perform'
    railties-6.0.0.beta3/lib/rails/commands/rake/rake_command.rb:20:in `perform'
    railties-6.0.0.beta3/lib/rails/command.rb:48:in `invoke'
    railties-6.0.0.beta3/lib/rails/commands.rb:18:in `<main>'
    bootsnap-1.4.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
    bootsnap-1.4.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
    bootsnap-1.4.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:83:in `register'
    bootsnap-1.4.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
    bootsnap-1.4.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
    activesupport-6.0.0.beta3/lib/active_support/dependencies.rb:297:in `block in require'
    activesupport-6.0.0.beta3/lib/active_support/dependencies.rb:263:in `load_dependency'
    activesupport-6.0.0.beta3/lib/active_support/dependencies.rb:297:in `require'
    bin/rails:4:in `<main>'

* Make Template::Resolver always cache

All actionview caches are already cleared at the start of each request
(when Resolver.caching is false) by PerExecutionDigestCacheExpiry, which
calls LookupContext::DetailsKey.clear (which clears all caches).

Because caches are always cleared per-request in dev, we shouldn't need
this extra logic to compare mtimes and conditionally reload templates.

This should make templates slightly faster in development (particularly
multiple renders of the same template)

* Remove updated_at from Templates

* Match evented checker behavior on dir with no exts

When FileUpdateChecker is passed a directory and given an empty array of
extensions to match on, it will match any extension.

Previously, EventedFileUpdateChecker would never match any files when
given an empty array. This commit makes it EventedFileUpdateChecker
match FileUpdateChecker, and watch all extensions when given an empty

* Rename `ActionView::Base#run` to `#_run`

There was a recent change by @tenderlove to Action view which introduced
`ActionView::Base#run` [1].

We ran into an issue with our application because one of the core
concepts in our domain model is a `Run` which is exposed in most of our
views as a helper method, which now conflicts with this new method.

Although this is a public method it is not really meant to be part of
the public API.

In order to discourage public use of this method and to reduce the
chances of this method conflicting with helper methods we can prefix
this method with an underscore, renaming this method to `_run`.

[1] rails@c740ebd

* Update docs for 'parameterize()' [ci skip]

* Engines are reloaded in Zeitwerk mode [closes rails#35618]

* Add -e/--environment option to `rails initializers`

This allows specifying the environment as would any other rails commands.

* Support Optimizer Hints

We as Arm Treasure Data are using Optimizer Hints with a monkey patch
especially in order to use `MAX_EXECUTION_TIME` (refer rails#31129).


class Job < ApplicationRecord
  default_scope { optimizer_hints("MAX_EXECUTION_TIME(50000) NO_INDEX_MERGE(jobs)") }

Optimizer Hints is supported not only for MySQL but also for most
databases (PostgreSQL on RDS, Oracle, SQL Server, etc), it is really
helpful to turn heavy queries for large scale applications.

* Raise UnknownAttributeError when unknown column is passed to insert_all and friends.

* `ActiveRecord::Result#to_hash` has been renamed to `to_a` [ci skip]

See rails#33912.

* Extract `truncate` and `truncate_tables` into database statements

This is to easier make `truncate_tables` to bulk statements.

* Fix warning: instance variable @optimizer_hints not initialized

* Make `truncate_tables` to bulk statements


   (16.4ms)  TRUNCATE TABLE `author_addresses`
   (20.5ms)  TRUNCATE TABLE `authors`
   (19.4ms)  TRUNCATE TABLE `posts`


  Truncate Tables (19.5ms)  TRUNCATE TABLE `author_addresses`;

* Ensure `execute_batch` discards remaining results

* Fix undefined local variable or method `discard_remaining_results'

* SQLite3: Make fixture loading to bulk statements

* Don't override @set_cookies on CookieJar#update_cookies_from_jar'

When building the cookie_jar for the current test request.
It was possible for this method to override keys currently being set on the test itself.
In situations such as when making two requests mixing creating the cookie on the test and the controller.

* Tweak `truncate_tables`

* Remove redundant `table_names.empty?`
* Early return in `truncate_tables` since it is already deeply nested
* Move `truncate_tables` out from between `exec_delete` and `exec_update`

* Add test case for unscoping `:optimizer_hints`

* Add `:reselect` and `:optimizer_hints` to the list of finder methods

These are introduced after Rails 6.0.0 beta2.

[ci skip]

* Ruby 2.4 and later support native Unicode case mappings

Here is only place where we use `mb_chars` internally.

* Revert "Pass HTML responses as plain-text in rails-ujs"

This reverts commit 48e44ed.

See discussion in rails#32287

For HTML content in `ajax:success` handlers, `event.detail[0]` should
be an `HTMLDocument` instance.

* Add regression test for HTML content in rails-ujs

See discussion in rails#32287

* Get rid of `Arel::Nodes::Values`

That is completely covered by `Arel::Nodes::ValuesList`.

Related rails/arel#484.

* Suggest 'strict-origin' Referrer-Policy header

* Fix updating rich text via nested attributes

Closes rails#35159.

* Add test case to prevent possible SQL injection

* Fix Time#advance to work with dates before 1001-03-07

In rails#10634 the behavior of Time#advance was changed to maintain a
proleptic gregorian calendar for dates before calendar reform. However
it didn't full address dates a long time before calendar reform and
they gradually drift away from the proleptic calendar the further you
go back in time. Fix this by always converting the date to gregorian
before calling advance which sets the reform date to -infinity.

* Added release notes for changes in Railties [ci skip] (rails#35622)

[Alex Kitchens, Prathamesh Sonpatki]

* Don't expose internal `type_casted_binds` method

Internal usage for the method as public has removed at rails#29623.

* Ignore nil in LookupContext#formats=

This also removes the mutation we were performing on the values being
passed in.

* Make uniq in LookupContext#formats=

Having a format listed twice had no effect. This is mostly helpful to
avoid an extra format when assigning [:html, "*/*"]

* Raise in LookupContext#formats= on invalid format

This is a developer quality of life improvement, to ensure that unknown
formats aren't assigned (which it would previously accept, but wouldn't
work 100% correctly due to caching).

* Extract `sanitize_as_sql_comment` from SQL visitor into connection

Probably that is useful for any other feature as well.

* Remove virtual_path from fallback templates

* Fix database configuration when adding another config level

This is kind of hard to explain but if you have a database config with
another level like this:

    database: "my db"
      statement_timeout: 1000

the database configurations code would chooke on the `variables` level
because it didn't know what to do with it.

We'd see the following error:

`block in find_db_config': undefined method `env_name' for [nil]:Array

The problem here is that Rails does correctly identify this as not a
real configuration but returns `[nil]` along with the others. We need to
make sure to flatten the array and remove all the `nil`'s before
returning the `configurations` objects.

Fixes rails#35646

* Extract column check in values_list

`values_list` is quite long and does far too many things. This also
eagerly extract the keys in initialize instead of having to worry about
calling them multiple times.

* Simplify values_list with more composition

This also prevents insert_all from leaking its attributes checks.

* Remove roflscaling

roflscaling (using frozen string constants instead of literal
strings) was added in 2012, before frozen string literals were
added in Ruby 2.3.  Now that Rails no longer supports Ruby <2.3,
and all of these files use frozen string literals, there is
no reason to keep the roflscaling.

This does not delete or deprecate the related constants. Such
a change can be made in a later commit.

* Use symbol for mail preview format, not string

* Rename invalid_types to invalid_values

* Revert "Remove `javascripts` and `javascript_engine` options for generators"

* Improve "raises on invalid format assignment" test

* Fix a ContentNegotiation test description

* depend on Zeitwerk 1.4.0

* Traducir del: inglés
Checking for the existence of only one table

* update_at/on note for ActiveRecord::Relation.update_all (rails#35461)

* update_at/on note for ActiveRecord::Relation.update_all

* Verbatim updated_at/on

* Update regular expression for checking valid MIME type

MIME Type validation regular expression does not allow for MIME types initialized with strings that contain parameters after the MIME type name.

* Instrument middleware processing

Adds ActiveSupport::Notifications instrumentation of the processing of
each middleware in the stack.

* Remove roflscaling constants

* Re-add Template#updated_at as deprecated

* Fix test broken by 04ae0b0

This test was trying to set the exception_app in the wrapper proxy
instead in the middleware itself.

* Avoid assigning [nil] to formats

* Don't compact formats

* Revert "Revert "Remove `javascripts` and `javascript_engine` options for generators""

* Add mention about `process_middleware.action_dispatch` in the guide [ci skip]

We added ActiveSupport::Notifications instrumentation of the processing of each middleware in the stack,
See 04ae0b0.

* Make aliases of `database` option in generators work

Thor automatically adds `-` if aliases do not start with `-`.

But Thor follows a convention of one-dash-one-letter options.
So, even if `-` is added to `db`, it does not work.

Follow up rails#34021.

* Bulk Insert: Reuse indexes for unique_by

I found `:unique_by` with `:columns` and `:where` inside it tough to
grasp. The documentation only mentioned indexes and partial indexes.
So why duplicate a model's indexes in an insert_all/upsert_all call
when we can just look it up?

This has the added benefit of raising if no index is found, such that
people can't insert thousands of records without relying on an index
of some form.

* [ci skip] Documentation pass for insert_all etc.

* Document option forwarding in ActiveRecord::Base.attribute

This has been supported for a while but we didn't have documentation
for it.

* Address rubocop offences

* Fix document formatting of Database configuration classes [ci skip]

* Show options as list.
* Fix incorrect quoting.

* Add `unknown` method to list of methods to write current log [ci skip]

* Prefer render template: in tests

Many tests were using `render file:`, but were only testing the
behaviour of `render template:` (file: just allows more paths/ is less
secure then template:).

The reason for so many `render file:` is probably that they were the old

This commit replaces `render file:` with `render template:` anywhere the
test wasn't specifically interested in using `render file:`.

* Fix server restart test on Puma 3.12.1

Since puma/puma#1700, the default host is
correctly used. So `localhost` is used instead of ``.

As a result, the log output on restart is changed, and the restart test
fails on Puma 3.12.1.

Specify binding explicitly to avoid being affected by Puma changes.

* Add Relation#annotate for SQL commenting

This patch has two main portions:

1. Add SQL comment support to Arel via Arel::Nodes::Comment.
2. Implement a Relation#annotate method on top of that.

== Adding SQL comment support

Adds a new Arel::Nodes::Comment node that represents an optional SQL
comment and teachers the relevant visitors how to handle it.

Comment nodes may be added to the basic CRUD statement nodes and set
through any of the four (Select|Insert|Update|Delete)Manager objects.

For example:

    manager =
    manager.table table
    manager.to_sql # UPDATE "users" /* annotation */

This new node type will be used by ActiveRecord::Relation to enable
query annotation via SQL comments.

== Implementing the Relation#annotate method

Implements `ActiveRecord::Relation#annotate`, which accepts a comment
string that will be appeneded to any queries generated by the relation.

Some examples:

    relation = Post.where(id: 123).annotate("metadata string")
    # SELECT "posts".* FROM "posts" WHERE "posts"."id" = 123
    # LIMIT 1 /* metadata string */

    class Tag < ActiveRecord::Base
      scope :foo_annotated, -> { annotate("foo") }
    # SELECT "tags".* FROM "tags" LIMIT 1 /* foo */ /* bar */

Also wires up the plumbing so this works with `#update_all` and
`#delete_all` as well.

This feature is useful for instrumentation and general analysis of
queries generated at runtime.

* Fix release template

* Fix announce script

* Update AS::Notifications::Instrumenter#instrument

  * Update #instrument to make passing a block optional. This will let users
    leverage #instrument for messaging in addition to instrumentation.

* Update CHANGELOG and docs
with change to ActiveSupport::Notifications::Instrumenter#instrument

* Update instrumentation guides w/optional block

* Fix rails guides generation

Was getting `formats=': Invalid formats: "html"

* Fix unintended autosave on has_one through association

Fixes rails#35680

The problem occurred, when a `has one through` association contains
a foreign key (it belongs to the intermediate association).

For example, Comment belongs to Post, Post belongs to Author, and Author
`has_one :first_comment, through: :first_post`.

In this case, the value of the foreign key is comparing with the original
record, and since they are likely different, the association is marked
as changed. So it updates every time when the origin record updates.


* Update CHANGELOGs for 6.0.0.beta3 release

* Drop microseconds in job argument assertions

* Add `secret_key_base` when creating new credential file

Since `secret_key_base` is expected to be included in credential file,
`secret_key_base` should be included even if re-create the file. This is
the same behavior as creating a new app.
When env is specified, it may be unnecessary, so I added it only when not
specifying env.

* Update comment for how secret key is calculated

This updates the comment to reflect how the secret key is generated
since 4c74358

Fixes rails#35717

* [ci skip] Fixed typo

* Add config.disable_sandbox option to Rails console

A long-running `rails console --sandbox` could cause a database server
to become out-of-memory as it's holding on to changes that happen on the

Given that it's common for Ruby on Rails application with huge
traffic to have separate write database and read database, we should
allow the developers to disable this sandbox option to prevent someone
from accidentally causing the Denial-of-Service on their server.

* Depend on Zeitwerk 1.4.2

* Avoid creating ActionText::RichText records unnecessarily

Assigning a has_one association for a persisted record saves the change immediately, so attempting to read a rich-text attribute on a persisted record without a corresponding ActionText::RichText would eagerly create one. Avoid assigning the rich text association to fix.

* Add default `config.action_view.raise_on_missing_translations` [ci skip]

* Use webdrivers instead of chromedriver-helper for AV UJS tests

* Use webdrivers instead of chromedriver-helper in new apps

* Cleanup guide for configuring config.disable_sandbox and related changelog [ci skip] (rails#35733)

* Format 'RETURNING' text in the docs [ci skip]

* Reword test names in credentials_test.rb

* [ci skip] Fixed typo

* Add default value `ActiveSupport::Deprecation.silenced` [ci skip]

* Add the `Mime::Type::InvalidMimeType` error in the default rescue_response:

- rails#35604 introduced a vulnerability fix
  to raise an error in case the `HTTP_ACCEPT` headers contains malformated
  mime type.

  This will cause applications to throw a 500 if a User Agent sends an
  invalid header.

  This PR adds the `InvalidMimeType` in the default `rescue_responses` from
  the ExceptionWrapper and will return a 406. I looked up the HTTP/1.1
  RFC and it doesn't stand what should be returned when the UA
  sends malformated mime type. Decided to get 406 as it seemed to be the
  status the better suited for this.

* Use `assert_queries(0)` instead of `assert_no_queries` to ignore metadata queries

Fix rails#35665

$ ARCONN=mysql2 bin/test test/cases/scoping/named_scoping_test.rb test/cases/tasks/database_tasks_test.rb test/cases/associations/cascaded_eager_loading_test.rb test/cases/associations/eager_singularization_test.rb -n "/^(?:NamedScopingTest#(?:test_many_should_not_fire_query_if_scope_loaded)|ActiveRecord::DatabaseTasksDumpSchemaCacheTest#(?:test_dump_schema_cache)|CascadedEagerLoadingTest#(?:test_eager_association_loading_with_has_many_sti_and_subclasses)|EagerSingularizationTest#(?:test_eager_no_extra_singularization_has_many_through_belongs_to))$/" --seed 16818
Using mysql2
Run options: -n "/^(?:NamedScopingTest#(?:test_many_should_not_fire_query_if_scope_loaded)|ActiveRecord::DatabaseTasksDumpSchemaCacheTest#(?:test_dump_schema_cache)|CascadedEagerLoadingTest#(?:test_eager_association_loading_with_has_many_sti_and_subclasses)|EagerSingularizationTest#(?:test_eager_no_extra_singularization_has_many_through_belongs_to))$/" --seed 16818


CascadedEagerLoadingTest#test_eager_association_loading_with_has_many_sti_and_subclasses [/home/yahonda/git/rails/activerecord/test/cases/associations/cascaded_eager_loading_test.rb:124]:
1 instead of 0 queries were executed.
Expected: 0
  Actual: 1

bin/test test/cases/associations/cascaded_eager_loading_test.rb:119

Finished in 6.894609s, 0.5802 runs/s, 1.0153 assertions/s.
4 runs, 7 assertions, 1 failures, 0 errors, 0 skips

* Fix CI failure due to remaining tagging records

`TRUNCATE TABLE posts` also resets `AUTO_INCREMENT`. If newly created a
post, it is wrongly associated with remaining tagging records.
To un-associate remaining tagging record, use `post.create_tagging!`

Fixes rails#35751.

* Spelling error

* Use weak references in descendants tracker

It allows anonymous subclasses to be garbage collected.

* Raise if resource custom params contain colons

After this change it's not possible anymore to configure routes
like this:

    routes.draw do
      resources :users, param: "name/:sneaky"

Fixes rails#30467.

* Aligned the order of the arguments of render_template and render_with_layout

* Remove useless = [ci skip]

* bumps Zeitwerk and Bootsnap

* includes bootsnap 1.4.2-java in Gemfile.lock

Was just pushed.

* Deprecate custom patterns for PathResolver

Custom glob patterns tie the implementation (Using Dir.glob) to the API
we provide.

It also doesn't really work. extract_handler_and_format_and_variant
expects the handler, format, and variant to be at the end of the
template path, and in the same order as they are in the default pattern.

This deprecates specifying a custom path for FileSystemResolver and
removes the pattern argument of OptimizedFileSystemResolver#initialize,
which does not work with a custom pattern.

* Bump RuboCop to 0.66.0

### Summary

RuboCop 0.66.0 has been released.

And rubocop-0-66 channel is available in Code Climate.

RuboCop 0.66.0 fixed the false negative to indentation for
modifier. And this PR applied the auto-correction fixed by it.

In addtion, this PR is also updating the following 4 gems that
RuboCop depends on.

- Update Psych gem ... rubocop-hq/rubocop#6766
- Update Parser gem to that supports Ruby 2.5.5 and 2.6.2 ...
- Remove powerpack gem ... rubocop-hq/rubocop#6806
- Update unicode-display_width gem ... rubocop-hq/rubocop#6813

* Schema version documentation (rails#35762)

* Update

* Change year to match others

[Ken Greeff + Rafael Mendonça França]

* Don't encode in secure_compare for speedup

Hex encoding is base 16 which makes the original input twice as big. With this change less time need to be spent in fixed_length_secure_compare.

* Don't change `collation_connection` in the current connection

We have a test case for `collation_connection` session variable, so it
should not be changed in any other test.

Fixes rails#35458.

* Introduce Template::File as new render file:

The previous behaviour of render file: was essentially the same as
render template:, except that templates can be specified as an absolute
path on the filesystem.

This makes sense for historic reasons, but now render file: is almost
exclusively used to render raw files (not .erb) like public/404.html. In
addition to complicating the code in template/resolver.rb, I think the
current behaviour is surprising to developers.

This commit deprecates the existing "lookup a template from anywhere"
behaviour and replaces it with "render this file exactly as it is on
disk". Handlers will no longer be used (it will render the same as if
the :raw handler was used), but formats (.html, .xml, etc) will still be
detected (and will default to :plain).

The existing render file: behaviour was the path through which Rails
apps were vulnerable in the recent CVE-2019-5418. Although the
vulnerability has been patched in a fully backwards-compatible way, I
think it's a strong hint that we should drop the existing
previously-vulnerable behaviour if it isn't a benefit to developers.

* Remove :all symbol from Mime::ALL

.all isn't a valid file extension, so it shouldn't used as a symbol.
This also makes Mime::ALL better match how */* is parsed from an Accept

* Fix bug in Range comparisons when comparing to excluded-end Range

(1..10).cover?(1...11) => false

(1..10).cover?(1...11) => true

See for the commit against Ruby core that added
support for Range arguments, with similar handling of this case.

* [ci skip] Fixed testing guides typo `fourty` -> `forty`

* Add ActiveStorage::Service#open

* Add load hook for ActiveStorage::Attachment

* Add rich_text field to model generators

* Fix annotated typo

* Rename `i18n_full_message` config option to `i18n_customize_full_message`

- I feel `i18n_customize_full_messages` explains the meaning of the
  config better.
- Followup of rails#32956

* Add Active Model release notes for Rails 6 [ci skip] (rails#35790)

* Fixed the test description for i18n-customize-full-message after rename in rails#35789

* Type cast falsy boolean symbols on boolean attribute as false

Before 34cc301, type casting by boolean attribute when querying is a
no-op, so finding by truthy boolean string (i.e.
`where(value: "true") # => value = 'true'`) didn't work as expected
(matches it to FALSE in MySQL rails#32624). By type casting is ensured, a
value on boolean attribute is always serialized to TRUE or FALSE.

In PostgreSQL, `where(value: :false) # => value = 'false'` was a valid
SQL, so 34cc301 is a regresson for PostgreSQL since all symbol values
are serialized as TRUE.

I'd say using `:false` is mostly a developer's mistake (user's input
basically comes as a string), but `:false` on boolean attribute is
serialized as TRUE is not a desirable behavior for anybody.

This allows falsy boolean symbols as false, i.e.
`klass.create(value: :false).value? # => false` and
`where(value: :false) # => value = FALSE`.

Fixes rails#35676.

* Add `ActiveRecord::Relation#extract_associated` for extracting associated record (rails#35784)

* Add `ActiveRecord::Relation#extract_associated` for extracting associated records from a relation

* [ci skip] Updated docs to reflect index: true option not available as column modifier

* Fix callbacks on has_many :through associations (rails#33249)

When adding a child record via a has_many :through association,
build_through_record would previously build the join record, and then
assign the child record and source_type option to it.  Because the
before_add and after_add callbacks are called as part of build, however,
this caused the callbacks to receive incomplete records, specifically
without the other end of the has_many :through association.  Collecting
all attributes before building the join record ensures the callbacks
receive the fully constructed record.

* [ci skip] Add ActiveRecord::Relation#extract_associated method to the

* fixes eager loading edge case in :zeitwerk mode

* optimizes eager loading in :zeitwerk mode

During initialization, the eager load paths of engines are unshifted
into AS::Dependencies.autoload_paths. After that, the collection is
frozen. (See the initializers in railties/lib/rails/engine.rb.)

Hence, there is no eager load path that is not an autoload path too, and
so the array difference in the deleted code is always an empty array.

Just do nothing.

* [ci skip] Fix typo directores -> directories

* Specify Active Storage's dependency on Active Job

* Require railties for all Active Storage dependencies

* Tweaks CHANGELOGs and docs [ci skip]

* add leading `#` before `=>` since hash rocket is valid Ruby code
* add backticks
* remove trailing spaces
* and more

* Follow up tweaks b89a3e7 [ci skip]

* use backticks instead of `+`
* and more (e.g. missed replacing `Array#excluding` and
`Enumerable#excluding` in b89a3e7)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.