sidekiq not processing queues in 3.16

[lsh-0]

I'm trying to import our articles from crossref. I can see the crossref import jobs accumulating in sidekiq, but nothing happens - Sidekiq isn't processing anything.
Sidekiq is running with no errors or warning or much of anything in the log file.
I remove all other jobs in all other queues, stop and start sidekiq but nothing.

Import jobs:

Queues:

[mfenner]

Thanks, I will have a look.

[lsh-0]

There may be a fix in the works, probably related to file permissions. I'll
update this big in the morning.
On 11 Feb 2015 21:49, "Martin Fenner" notifications@github.com wrote:

Thanks, I will have a look.

—
Reply to this email directly or view it on GitHub
#246 (comment)
.

[mfenner]

File permissions would indeed be a good explanation, as I have exactly the same code running in multiple lagotto instances.

[lsh-0]

seeing the same behaviour on an AWS instance now.

the user being used to run sidekiq has full rwx on the /var/www/lagotto and it's subdirs where sidekiq is storing logs and pids, so not being limited by permissions there.

HOWEVER

I am seeing activity in the sidekiq log (attached). Once an hour at 10mins past the hour it appears to queue ~8 jobs and finish them. This is the hourly cron. The number of processed jobs goes up, but so does the number enqueued - this means it will never reach the end of the 'critical' queue.

The daily cron job is giving an error that appears to be related to the database:

$ RAILS_ENV=production bundle exec rake cron:daily
rake aborted!
ActiveRecord::SubclassNotFound: The single-table inheritance mechanism failed to locate the subclass: 'ArticleNotUpdatedError'. This error is raised because the column 'type' is reserved for
 storing the class in case of inheritance. Please rename this column if you didn't intend it to be used for storing the inheritance class or overwrite Filter.inheritance_column to use anothe
r column for that information.   
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/inheritance.rb:186:in `rescue in find_sti_class'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/inheritance.rb:180:in `find_sti_class'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/inheritance.rb:169:in `discriminate_class_for_record'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/persistence.rb:67:in `instantiate'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/querying.rb:50:in `block (2 levels) in find_by_sql'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/result.rb:51:in `block in each'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/result.rb:51:in `each'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/result.rb:51:in `each'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/querying.rb:50:in `map'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/querying.rb:50:in `block in find_by_sql'
/var/lib/gems/2.1.0/gems/activesupport-4.2.0/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/querying.rb:49:in `find_by_sql'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/relation.rb:638:in `exec_queries'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/relation.rb:514:in `load'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/relation.rb:243:in `to_a'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/relation/delegation.rb:46:in `each'
/srv/lagotto/app/models/filter.rb:43:in `run'
/srv/lagotto/lib/tasks/filter.rake:4:in `block (2 levels) in <top (required)>'
/srv/lagotto/lib/tasks/cron.rake:16:in `block (2 levels) in <top (required)>'
NameError: uninitialized constant ArticleNotUpdatedError
/var/lib/gems/2.1.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:261:in `const_get'
/var/lib/gems/2.1.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:261:in `block in constantize'
/var/lib/gems/2.1.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:259:in `each'
/var/lib/gems/2.1.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:259:in `inject'
/var/lib/gems/2.1.0/gems/activesupport-4.2.0/lib/active_support/inflector/methods.rb:259:in `constantize'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/inheritance.rb:181:in `find_sti_class'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/inheritance.rb:169:in `discriminate_class_for_record'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/persistence.rb:67:in `instantiate'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/querying.rb:50:in `block (2 levels) in find_by_sql'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/result.rb:51:in `block in each'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/result.rb:51:in `each'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/result.rb:51:in `each'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/querying.rb:50:in `map'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/querying.rb:50:in `block in find_by_sql'
/var/lib/gems/2.1.0/gems/activesupport-4.2.0/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/querying.rb:49:in `find_by_sql'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/relation.rb:638:in `exec_queries'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/relation.rb:514:in `load'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/relation.rb:243:in `to_a'
/var/lib/gems/2.1.0/gems/activerecord-4.2.0/lib/active_record/relation/delegation.rb:46:in `each'
/srv/lagotto/app/models/filter.rb:43:in `run'
/srv/lagotto/lib/tasks/filter.rake:4:in `block (2 levels) in <top (required)>'
/srv/lagotto/lib/tasks/cron.rake:16:in `block (2 levels) in <top (required)>'
Tasks: TOP => filter:all
(See full trace by running task with --trace)

All migrations have been run.

Does the daily cron event process the jobs in the 'high' queue and the hourly event those in the 'critical' queue?

[mfenner]

@lsh-0 I have identified two issues. The first is an issue with passing ENV variables to the Sidekiq configuration file config/sidekiq.yml - the commit to master above should fix that.

The other issue is the path to the Sidekiq config, log and pid files. I have rewritten the logic to be more robust for different server configurations.

Let me know if you still have problems after deploying the latest code from the master branch. I will turn this into a hotfix if it resolves the issues.

[lsh-0]

I'll update now and let you know

On Tue, Feb 17, 2015 at 10:26 AM, Martin Fenner notifications@github.com
wrote:

@lsh-0 https://github.com/lsh-0 I have identified two issues. The first
is an issue with passing ENV variables to the Sidekiq configuration file
config/sidekiq.yml - the commit to master above should fix that.

The other issue is the path to the Sidekiq config, log and pid files. I
have rewritten the logic to be more robust for different server
configurations.

Let me know if you still have problems after deploying the latest code
from the master branch. I will turn this into a hotfix if it resolves the
issues.

—
Reply to this email directly or view it on GitHub
#246 (comment)
.

Luke Skibinski
devops - eLife

eLife Sciences Publications, Ltd is a limited liability non-profit
non-stock corporation incorporated in the State of Delaware, USA, with
company number 5030732, and is registered in the UK with company number
FC030576 and branch number BR015634 at the address First Floor, 24 Hills
Road, Cambridge CB2 1JP.

[lsh-0]

after shifting to master, the command bundle exec rake sidekiq:start still doesn't appear to process jobs, however, running bundle exec sidekiq -c 2 did churn through the ~3000 queued jobs in a few minutes. This previously wasn't working for me on 3.16.

The daily cron is still giving me the same error as above though:

ActiveRecord::SubclassNotFound: The single-table inheritance mechanism failed to locate the subclass: 'ArticleNotUpdatedError'. This error is raised because the column 'type' is reserved for storing the class in case of inheritance. Please rename this column if you didn't intend it to be used for storing the inheritance class or overwrite Filter.inheritance_column to use another column for that information.

I can raise this as a separate bug if you'd like.

[mfenner]

Ok, some progress. To run sidekiq manually (-d to run as daemon), use this command from your root folder:

RAILS_ENV=production bundle exec sidekiq -C ./config/sidekiq.yml -d

And make sure you have CONCURRENCY in your .env file and restarted the server. The concurrency (-c 2 in your example, typical value is 25) is now read when Rails starts.

[lsh-0]

I killed off the existing sidekiq process and then ran that command, which appears to work.

I killed the process off again and did what my script is running (RAILS_ENV=production bundle exec rake sidekiq:start) and that appears to work now too.

[mfenner]

Closing the issue. Please reopen if there are still issues.