Write queries in Sidekiq jobs/workers fail randomly in Rails 6 when using multiple databases

[cestbalez]

Ruby version: 2.6.2
Sidekiq / Pro / Enterprise version(s): 5.2.7 / Pro: 4.0.5

There is an issue with Sidekiq jobs/workers that write to the database in Rails 6 when in multi-threaded environments with multiple databases. The write queries fail randomly with the following error: Write query attempted while in read-only mode....

This happens even when specifying the connections in application_record.rb: connects_to: database: { writing: :primary, reading: :replica } and directly in the job/worker code by wrapping the write queries in

ActiveRecord::Base.connected_to(role: :writing) do
  (...)
end

It turns out that this is related to a known issue in Rails 6 RC2 (rails/rails#36830), and a fix was included in Rails 6.0.0 (rails/rails#36843), but it didn't completely solve the problem. There's a new fix coming in Rails 6.0.1 (rails/rails#37065), but until then, the problem can be solved by specifying not to prevent writes with the following:

ActiveRecord::Base.connected_to(role: :writing) do
  ActiveRecord::Base.connection_handler.while_preventing_writes(false) do
    (...)
  end
end

This is not exclusive for Sidekiq jobs/workers, but that is where I have encountered it most frequently. Follow the links above for more info about the issue.

[cestbalez]

It turns out this didn't solve it after all. The problem returned last night, so if anyone has any insight it would be appreciated.