Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ConcurrencyError interrupted waiting for mutex: null #2245

Closed
phrinx opened this issue Nov 26, 2014 · 6 comments
Closed

ConcurrencyError interrupted waiting for mutex: null #2245

phrinx opened this issue Nov 26, 2014 · 6 comments

Comments

@phrinx
Copy link

@phrinx phrinx commented Nov 26, 2014

While running concurrent sidekiq jobs (with 75 threads configured on my machine) whereby some of them attempt to log errors I see once in a while the following stacktrace (the app was packaged as a executable jar with warbler):

org/jruby/ext/thread/Mutex.java:94:in `lock'
  /tmp/jruby8416221603871102217extract/jruby-stdlib-complete-1.7.16.jar!/META-INF/jruby.home/lib/ruby/1.9/monitor.rb:185:in `mon_enter'
    @mon_mutex.lock
  /tmp/jruby8416221603871102217extract/jruby-stdlib-complete-1.7.16.jar!/META-INF/jruby.home/lib/ruby/1.9/monitor.rb:209:in `mon_synchronize'
    mon_enter
  /srv/app.jar!/gems/log4r-1.1.10/lib/log4r/outputter/outputter.rb:130:in `synch'
    def synch; synchronize { yield } end
  /srv/app.jar!/gems/log4r-1.1.10/lib/log4r/outputter/outputter.rb:110:in `canonical_log'
    synch { write(format(logevent)) }
  (eval):3:in `error'
  (eval):8:in `error'
  org/jruby/RubyArray.java:1613:in `each'
  (eval):8:in `error'
  lib/helpers/logging.rb:139:in `Log'
  lib/helpers/sidekiq_middleware.rb:13:in `call'
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/middleware/chain.rb:124:in `invoke'
    chain.shift.call(*args, &traverse_chain)
  /srv/app.jar!/gems/sidekiq-pro-1.7.4/lib/sidekiq/batch/middleware.rb:26:in `call'
    yield
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/middleware/chain.rb:124:in `invoke'
    chain.shift.call(*args, &traverse_chain)
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
    yield
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/middleware/chain.rb:124:in `invoke'
    chain.shift.call(*args, &traverse_chain)
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/middleware/server/logging.rb:11:in `call'
    yield
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/logging.rb:22:in `with_context'
    yield
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/logging.rb:20:in `with_context'
    begin
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/middleware/server/logging.rb:7:in `call'
    Sidekiq::Logging.with_context("#{worker.class.to_s} JID-#{item['jid']}") do
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/middleware/chain.rb:124:in `invoke'
    chain.shift.call(*args, &traverse_chain)
  org/jruby/RubyProc.java:271:in `call'
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/middleware/chain.rb:127:in `invoke'
    traverse_chain.call
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/processor.rb:51:in `process'
    Sidekiq.server_middleware.invoke(worker, msg, queue) do
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/processor.rb:94:in `stats'
    yield
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/processor.rb:93:in `stats'
    begin
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/processor.rb:50:in `process'
    stats(worker, msg, queue) do
  /srv/app.jar!/gems/sidekiq-3.2.5/lib/sidekiq/processor.rb:44:in `process'
    begin
  org/jruby/RubyKernel.java:1952:in `public_send'
  /srv/app.jar!/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `dispatch'
    obj.public_send(@method, *@arguments, &_block)
  /srv/app.jar!/gems/celluloid-0.15.2/lib/celluloid/calls.rb:122:in `dispatch'
    super(obj)
  /srv/app.jar!/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:in `handle_message'
    message.dispatch(@subject)
  /srv/app.jar!/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in `task'
    yield
  /srv/app.jar!/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:in `initialize'
    yield
  /srv/app.jar!/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:47:in `initialize'
    begin
  /srv/app.jar!/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `create'
    yield
@rtyler
Copy link

@rtyler rtyler commented Nov 26, 2014

@mkristian would it be possible for you to work with @phrinx to figure this out when you have a chance?

@mkristian
Copy link
Member

@mkristian mkristian commented Nov 26, 2014

@phrinx the actual error message of the stacktrace is missing.

1 similar comment
@mkristian
Copy link
Member

@mkristian mkristian commented Nov 26, 2014

@phrinx the actual error message of the stacktrace is missing.

@phrinx
Copy link
Author

@phrinx phrinx commented Nov 26, 2014

@mkristian the error message is the subject of the issue ;)

@mkristian
Copy link
Member

@mkristian mkristian commented Dec 1, 2014

hey - another null exception message and no java stacktrace ;)

the stacktrace would at least show where the interrupt comes from.

@kares
Copy link
Member

@kares kares commented Jun 22, 2017

believe 2501c07 is expected to have resolved the issue on 9K. and since 1.7.x is EOL I am closing this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants