Rails 7.1 Transactional tests, threaded code and schema caching can cause deadlocks

[dannyfallon]

Steps to reproduce

# frozen_string_literal: true

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"

  gem "rails"
  gem "sqlite3", "~> 1.4"
end

require "active_record"
require "minitest/autorun"
require "logger"

# Have to use a persisted database rather than `:memory:`
ActiveRecord::Base.establish_connection(adapter: "sqlite3", pool: 5, database: "deadlocker.sqlite3")
ActiveRecord::Base.logger = Logger.new(STDOUT)

ActiveRecord::Schema.define do
  create_table :posts, force: true do |t|
  end

  create_table :comments, force: true do |t|
    t.text :body
  end
end

class Comment < ActiveRecord::Base
end

class Post < ActiveRecord::Base
  before_save :deadlock

  def deadlock
    Thread.new do
      ::Comment.new(body: "deadlock_on_fetching_schema_cache_columns").save!
    end

    # Naively sleep for 1 second to give the thread a chance to start and fetch
    # the schema cache lock then block on the `LoadInterlockAwareMonitor`
    sleep(1)

    ::Comment.new(body: "deadlock_on_obtaining_schema_cache_lock").save!
  end

end

class BugTest < Minitest::Test
  def test_deadlock
    # Mimic what happens in "full Rails" where we're running tests in a
    # transaction and the connection pool always uses the same connection
    ActiveRecord::Base.connection_pool.lock_thread = true
    ActiveRecord::Base.transaction do
      puts "Attempting deadlock"
      Post.create!
    end
  end
end

Expected behavior

I expect the test to pass and not deadlock.

Actual behavior

The main test thread has obtained the connection's LoadInterlockAwareMonitor lock in order to execute Post#save!. It's now the only thread allowed to execute SQL queries.

Another thread has been spawned that has obtained the Comment model's schema cache monitor lock in order to load the schema cache. This thread cannot execute the query to obtain the table's columns and blocks:

rails/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb

Line 1000 in 747a03b

@lock.synchronize do

The main thread meanwhile now requires access to Comment and also attempts to load the schema cache. It cannot obtain the schema cache monitor lock and blocks:

rails/activerecord/lib/active_record/model_schema.rb

Line 560 in 747a03b

@load_schema_monitor.synchronize do

System configuration

Rails version: 7.1.3.3

Ruby version: 3.2.2

[justinko]

If you're threading like that then you're gonna have to disable use_transactional_tests.