Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AsyncAdapter causes ActiveStorage attachment to hang in db:seed #34939

Closed
lbrito1 opened this Issue Jan 15, 2019 · 1 comment

Comments

Projects
None yet
2 participants
@lbrito1
Copy link

lbrito1 commented Jan 15, 2019

TLDR trying to attach two or more files using ActiveStorage in a db:seed task hangs after the first attachment.

I suspect that ActiveJob::QueueAdapters::AsyncAdapter is not releasing the DB lock, which causes the Rake task to hang during the second call to attach.

Rails has config.active_job.queue_adapter = :async by default, so this issue happens in a fresh Rails 5.2.2 app. When setting config.active_job.queue_adapter = :inline, the task completes with no errors.

Might be related to #33500 .

Steps to reproduce

git clone https://github.com/lbrito1/active-storage-bug
cd active-storage-bug
bundle
rails db:migrate
rails db:seed # will hang here

Although I was able to reproduce this many times with the steps above on different machines, it becomes non-reproduceable after the first run, even after clearing tmp/* and storage/* and running db:drop (i.e. rails db:seed runs normally).
To reproduce more than once on the same machine, I had to resort to Docker:

docker run -it ruby
apt-get update && apt-get install -y nodejs
# git clone ...

Expected behavior

Rake exits.

Actual behavior

Rake hangs.

System configuration

Rails version: 5.2.2

Ruby version: 2.6.0

Solution

I think we can config.active_job.queue_adapter = :inline in development.rb. Non-development envs seem more troublesome to me, though: as far as I can tell they're probably also affected by this.

Traces

This is the trace I get if I interrupt the task after a while with ctrl+c:

/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/concurrency/share_lock.rb:221:in `wait_for'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/concurrency/share_lock.rb:190:in `block in yield_shares'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/concurrency/share_lock.rb:189:in `yield_shares'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/sqlite3_adapter.rb:213:in `block in exec_query'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/sqlite3_adapter.rb:212:in `exec_query'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:122:in `exec_insert'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:154:in `insert'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `insert'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/persistence.rb:185:in `_insert_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/persistence.rb:732:in `_create_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/counter_cache.rb:184:in `_create_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/locking/optimistic.rb:70:in `_create_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/attribute_methods/dirty.rb:140:in `_create_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/callbacks.rb:346:in `block in _create_record'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/callbacks.rb:132:in `run_callbacks'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/callbacks.rb:816:in `_run_create_callbacks'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/callbacks.rb:346:in `_create_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/timestamp.rb:102:in `_create_record'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/persistence.rb:703:in `create_or_update'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/callbacks.rb:342:in `block in create_or_update'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/callbacks.rb:132:in `run_callbacks'
/usr/local/bundle/gems/activesupport-5.2.2/lib/active_support/callbacks.rb:816:in `_run_save_callbacks'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/callbacks.rb:342:in `create_or_update'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/persistence.rb:306:in `save!'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/validations.rb:52:in `save!'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/transactions.rb:315:in `block in save!'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:259:in `block in transaction'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:259:in `transaction'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/transactions.rb:212:in `transaction'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/transactions.rb:315:in `save!'
/usr/local/bundle/gems/activerecord-5.2.2/lib/active_record/suppressor.rb:48:in `save!'
/usr/local/bundle/gems/activestorage-5.2.2/app/models/active_storage/blob.rb:65:in `tap'
/usr/local/bundle/gems/activestorage-5.2.2/app/models/active_storage/blob.rb:65:in `create_after_upload!'
/usr/local/bundle/gems/activestorage-5.2.2/lib/active_storage/attached.rb:28:in `create_blob_from'
/usr/local/bundle/gems/activestorage-5.2.2/lib/active_storage/attached/one.rb:24:in `attach'
/active-storage-bug/db/seeds.rb:7:in `<main>'
@gmcgibbon

This comment has been minimized.

Copy link
Member

gmcgibbon commented Jan 16, 2019

So according to logs it is enqueuing ActiveStorage::AnalyzeJob from here:

blob.analyze_later unless blob.analyzed?

I think it makes the most sense to change ActiveJob's queue adapter to inline while loading the seed file:

load(seed_file) if seed_file

I'll see if I can fix this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.