Sidekiq error Uninitialized Constant

[juand]

I'm having an issue with sidekiq. Basically we're getting NameError: uninitialized constant on our sidekiq setup which is causing a large number of jobs to fail.

The error log says:

NameError: uninitialized constant GameUser::Lock /data/@myapp/releases/20130321230952/app/models/game_user.rb:71:in `node_calls_base_get_user' 
/data/@myapp/shared/bundled_gems/ruby/1.9.1/gems/sidekiq-2.8.0/lib/sidekiq/processor.rb:45:in `block (3 levels) in process' 
/data/@myapp/shared/bundled_gems/ruby/1.9.1/gems/sidekiq-2.8.0/lib/sidekiq/middleware/chain.rb:109:in `call'

The code is here:

# app/models/game_user.rb
def self.node_calls_base_get_user(serial, game_name)
  if Lock.get("user:#{id}") # Set up lock to prevent multiple users to be created      
    Lock.delete("user:#{id}")
  end
  return false
end

Lock is defined in a library:

# lib/lock.rb
class Lock
  def self.get(key)
    lock = CACHE.add("lock:#{key}", 1, 5) # Let lock autoexpire after 5 seconds
    return true
  end
end

And the lib/ folder is automatically loaded with the rest of the configurations.

module Myapp
  class Application < Rails::Application
    ...
    # Custom directories with classes and modules you want to be autoloadable.
    config.autoload_paths += %W(#{config.root}/lib)
    ...    
  end
end

I have no idea why this is happening. It seems to happen more frequently when we deploy, but it seems to happen often enough otherwise.

I've been following the following thread: #331 but it doesn't seem to offer a solution besides adding the lib folder to the autoload_paths.

I'm using:

gem 'rails', '3.2.13' 
gem 'sidekiq', '>= 2.7.5'

Any help would be greatly apreciated.

[bensie]

Add a require 'lock' at the top of your GameUser class.

[juand]

Hi @bensie This seems terribly impractical. Lock is used in more classes than this. Additionally this is not a unique error. We're getting the for a variety of libraries. My impression was that adding the lib folder under autoload should have fixed it but we're finding out that's not the case.

[bensie]

Autoload itself as well as a number of libraries out there are not thread-safe. You need to ensure that all your code is eager loaded.

[juand]

Do you know why it would not happen all the time? It seems to happen 1:100 or jobs. This are all very common jobs that use the same functions most of the time, so I'd suspect we'd be getting more errors coming in if the threads were colliding.

Is there anything we can do to fix it besides the eager loading? Is there a like a global solution for this? Having to require every single library needed for a class seems to defeat some of the purpose to Rails.

[mperham]

Use require_dependency 'lock' to ensure Rails will reload your class in development mode.

[juand]

@mperham the but issues is again that it's not a practical solution, since it would mean we'd need to eager load for each class all the dependencies. If autoload is not safe, is there another way to load the entire dependency tree safely?

Edit:
You had also recommended using the autoload feature on a previous thread, so i'm not entirely sure if it's then the best way to do it.

[mperham]

Ruby requires eager loading when using threads because require itself is not thread-safe. You can try adding lib to the config.eager_load_paths.

[juand]

From the docs, it looks like this will be safe on production.

config.eager_load_paths accepts an array of paths from which Rails will eager load on boot if cache classes is enabled. Defaults to every folder in the app directory of the application.

But you sound a bit concern of using it. Are there any reservations to using this on a production environment? Cached classes are enabled for production by default on rails for production environments, so we should be able to use this.

Edit:

Also, require_dependency is recommended mostly for development. What are you thought about using that on prod?

[mperham]

Typically lib has a lot of junk in it, things like Rake tasks in lib/tasks that dont need to be loaded in your app. But you can judge for yourself if it's worth it.

[kakubei]

I've got eager loading for lib set in application.rb

config.autoload_paths += %W(#{config.root}/lib)
config.eager_load_paths += %W(#{config.root}/lib) # this is for Sidekiq

and am still getting this same error for classes that once worked. Now all of them give me the Uninitialized Constant error in development.

Rails 3.2.13
Sidekiq 2.12.0

Trace:

MacBookPro:adminHeroku alexd$ env WORKER_PROCESS=1 bundle exec sidekiq -c 1 -r './config/boot.rb' - e worker
2013-05-29T12:59:24Z 9028 TID-ox3o0bzz0 INFO: Booting Sidekiq 2.12.0 using redis://localhost:6379/0 with options {}
2013-05-29T12:59:24Z 9028 TID-ox3o0bzz0 INFO: Running in ruby 1.9.3p286 (2012-10-12 revision 37165) [x86_64-darwin12.2.0]
2013-05-29T12:59:24Z 9028 TID-ox3o0bzz0 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2013-05-29T12:59:24Z 9028 TID-ox3o0bzz0 INFO: Starting processing, hit Ctrl-C to stop
2013-05-29T12:59:30Z 9028 TID-ox3oaso3g WARN: {"retry"=>true, "queue"=>"default", "class"=>"TransferBatchSidekiq", "args"=>[1, "game"], "jid"=>"f16826bb9bfa5339f4706a7e"}
2013-05-29T12:59:30Z 9028 TID-ox3oaso3g WARN: uninitialized constant TransferBatchSidekiq
2013-05-29T12:59:30Z 9028 TID-ox3oaso3g WARN: /Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/activesupport-3.2.13/lib/active_support/inflector/methods.rb:230:in block in constantize' /Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/activesupport-3.2.13/lib/active_support/inflector/methods.rb:229:ineach'
/Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/activesupport-3.2.13/lib/active_support/inflector/methods.rb:229:in constantize' /Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/activesupport-3.2.13/lib/active_support/core_ext/string/inflections.rb:54:inconstantize'
/Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/sidekiq-2.12.0/lib/sidekiq/processor.rb:43:in process' /Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/celluloid-0.14.0/lib/celluloid/calls.rb:25:inpublic_send'
/Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/celluloid-0.14.0/lib/celluloid/calls.rb:25:in dispatch' /Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/celluloid-0.14.0/lib/celluloid/calls.rb:125:indispatch'
/Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/celluloid-0.14.0/lib/celluloid/actor.rb:328:in block in handle_message' /Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/celluloid-0.14.0/lib/celluloid/tasks.rb:42:inblock in initialize'
/Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/celluloid-0.14.0/lib/celluloid/tasks/task_thread.rb:20:in block in create' /Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/celluloid-0.14.0/lib/celluloid/internal_pool.rb:59:incall'
/Users/alexd/.rbenv/versions/1.9.3-p286/lib/ruby/gems/1.9.1/gems/celluloid-0.14.0/lib/celluloid/internal_pool.rb:59:in `block in create'

[juand]

The more I read about it, the more I realize Rails is just not thread
safe...

In our application it was pretty impossible to do what James Miller
originally suggested because of the way dependencies were built. However, I
would strongly recommend you going that way if possible.

Basically individually requiring the classes you need for each task.

The above fix worked on our project though we limited it to only eager_load
rather than both autoload and eager_load.

Did you make sure it's on the application.rb? or at least in the
development.rb files?

Juan Delgado
Server Engineer
jdelgado@quarkgames.com
Quarkgames.com

On Wed, May 29, 2013 at 6:00 AM, kakubei notifications@github.com wrote:

I've got eager loading for lib set in application.rb

config.autoload_paths += %W(#{config.root}/lib)
config.eager_load_paths += %W(#{config.root}/lib) # this is for Sidekiq

and am still getting this same error for classes that once worked. Now all
of them give me the Uninitialized Constant error in development.

Rails 3.2.13
Sidekiq 2.12.0

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/789#issuecomment-18614420
.

[mperham]

@juand Rails is thread-safe. Autoloading is not. Use require_dependency to pull in code eagerly while still keeping compatible with code reloading.

[juand]

@mperham thanks for correcting me. Isn't Autoloading a basic configuration
for Rails though? I feel like this is probably one of those things that
people just take for granted in Rails and gloss over. It certainly happen
to me which is why I opened this thread to begin with.

I think that with Rails even though it does support thread safety, it's
something you need create your application around. If it's an existing
application that you're just dropping something like sidekiq in, you're
bound to encounter issues like this, unlike an earlang framework where the
entire language is thread-safe out the box.

This was my first time delving into multi-threading so I do apologize if
this has been discussed before, I just find this conversation very enriching.

[mperham]

Yep, unfortunately autoloading doesn't work well except for code in app/; it's tricky to set up correctly for code in lib/. Unfortunately there's nothing I can do about it except educate and guide people to use best practices and the tools available.

[kakubei]

I'm a little confused here, I don't think the issue I'm seeing is due to lack of a require_dependency, I believe it's something else since the same setup works on Heroku and even locally in Development in the Padrino Console the classes that Sidekiq is complaining about are recognized properly. Just when I try to run a job locally I get these errors.

Not really sure where the problem is, any help would be appreciated. Thanks.

[ghost]

@kakubei, isn't it because by default, in development.rb you have config.cache_classes = false? The docs say that Rails only eager loads if config.cache_classes = true.

http://guides.rubyonrails.org/v3.2.13/configuring.html#rails-general-configuration

config.eager_load_paths accepts an array of paths from which Rails will eager load on boot if cache classes is enabled. Defaults to every folder in the app directory of the application.

This kinda sucks though, because it's nice to have caching off in development.

[pokrovskyy]

I'm having this kind of error popping up spontaneously on my production workers. I would say rather rarely. It only sometimes happens upon worker boot-up. Missing classes differ much:

uninitialized constant ActiveRecord::Associations::BelongsToAssociation
uninitialized constant ActiveRecord::Associations::HasManyAssociation
uninitialized constant ActiveRecord::Associations::Preloader::BelongsTo
uninitialized constant ActiveRecord::Core::LegacyYamlAdapter
uninitialized constant ActiveRecord::Associations::HasManyThroughAssociation
uninitialized constant ActiveRecord::Associations::JoinDependency::JoinBase
etc.

It really seems like the worker has not yet fully loaded all the requirements, but Sidekiq part already has and starting pulling up jobs and throwing them into the threads. Not sure if it's Sidekiq issue or not, but it really seems like it starts working a little too early, like it should just wait for the requirements to load up. Could not reproduce this in dev env, only happens in high-load production. Any thoughts?

[seuros]

@pokrovskyy What versions of the gems do you have ? did you have the latest version on production ?

[bhaveshf-cuelogic]

So what's the ideal solution for this problem ? We are also facing this issue in our app with Ruby 2.4.5 & Rails 5.2.0. We have our rails app running in one container and sidekiq running in another container.

We have some ruby classes in app/apis/services/<MyClassFile> which are being called from a worker file placed in app/workers/<folder>/<MyWorkerClass>. Whatever lines of code was calling MyClassFile from MyWorkerClass, we tried to move in models so that it gets autoloaded but still the problem exists.

This is for Sidekiq version: v5.2.5 & Sidekiq cron v1.1.0

[seuros]

You are not the autoloader correctly . If have workers inside a folder, they should be namespaced with the same name of the folder , or required manually.

Services::MyClass

[bhaveshf-cuelogic]

@seuros : Following is the trimmed down code of my current problem

app/apis/services/notification/type1_notifier.rb

module Services
  module Notification
    class Type1Notifier
      def initialize()
      ....
      end
    end
  end
end

app/workers/notifications/my_worker.rb

module Notifications
  class MyWorker
    include Sidekiq::Worker
    def perform
       Services::Notification::Type1Notifier.new(...)
    end
  end
end

[mperham]

@bhaveshf-cuelogic https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting#autoloading

You can ask for help and people are welcome to help but I make no attempt to solve autoloading problems. It's always an app problem, not a Sidekiq issue; the best thing to do is review the Rails autoloading guide.

https://guides.rubyonrails.org/autoloading_and_reloading_constants.html