Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Broken autoloading classes in lib #80

Closed
canavese opened this Issue · 13 comments

6 participants

Paul Canavese Mike Perham Fredrik Björk Ryan LeCompte Joshua Sierles Michael Madison
Paul Canavese

I'm new to sidekiq, so it's quite possible that I missed something here, but I'm having issues with autoloading classes in my lib directory. Rails 3.2.2. The lib directory is in my Rails config.autoload_paths, and I haven't had problems outside of sidekiq.

I'm running with this: bundle exec sidekiq -c 25

I've had the problem with multiple lib classes, so it's not an issue with just one. I also double-checked that the file names match the class. Example stack trace below. The file referenced does indeed define the named class (matching case and all).

Adding a require in the calling file fixes the problem, but that's obviously annoying.

Ideas?

/Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:503:in `load_missing_constant': Expected /Users/canavese/workspace/neighboring2/lib/datetime_extractor.rb to define DatetimeExtractor (LoadError)
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:192:in `block in const_missing'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:190:in `each'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:190:in `const_missing'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/aws-s3-0.6.2/lib/aws/s3/extensions.rb:206:in `const_missing_from_s3_library'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:514:in `load_missing_constant'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:192:in `block in const_missing'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:190:in `each'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:190:in `const_missing'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/aws-s3-0.6.2/lib/aws/s3/extensions.rb:206:in `const_missing_from_s3_library'
    from /Users/canavese/workspace/neighboring2/app/models/publish/publish_analyzer.rb:63:in `block in analyze'
Mike Perham
Owner

I suspect you have other load issues which are presenting themselves as this error.

Mike Perham
Owner

I just tried your scenario locally and it worked as expected. Let me know if you learn anything more that points to a Sidekiq issue.

Mike Perham mperham closed this
Paul Canavese

Thanks for looking at it a bit. It definitely wasn't a problem with all auto-loading anything from lib. It seems inconsistent and may be related to load order. But I'll ping you again if I can figure out a more direct association with sidekiq, but you may very well be right that it's a general load issue that I just happen to only be hitting in sidekiq.

Fredrik Björk

I'm also seeing autoload issues in my Sidekiq workers at random. It doesn't happen in other processes (web, rake tasks etc). The solution seems to be to require the files instead of relying on autoloading for now.

Ryan LeCompte
Collaborator

Right, autoloading in Ruby is not thread-safe. It's best to eagerly load classes in third-party libraries if you're using them in a multithreaded fashion (i.e., via Sidekiq). I am doing the following eager-loading for the following gems:

  # eagerly load classes for aws-sdk gem
  AWS.eager_autoload!

  # eagerly load classes for faraday gem
  Faraday.load_autoloaded_constants
Mike Perham
Owner

I call Rails.application.eager_load! to eager load your app/models before the Sidekiq Processors are even created but my guess is that Rails does not eager load lib/ and other directories manually added to the autoload_path.

Fredrik Björk

Wonder if the following would work?

config.eager_load_paths += %W(#{config.root}/lib/stuff)
Mike Perham
Owner

That looks like a brilliant plan to me.

Joshua Sierles

Are there clear best practices thread safety? Are we required to call Rails.application.eager_load!, and if so, where? Should config.threadsafe! be enabled?

Mike Perham
Owner

@jsierles You need to do nothing.

Sidekiq automatically calls eager_load! when it boots your application. Sidekiq doesn't explicitly invoke threadsafe! but since there's no web part of Sidekiq, there's no reloading happening at the end of a web request.

If you have threading or loading problems, open an issue and we'll figure out the problem.

Joshua Sierles
Michael Madison

Does this mean any gem needs to be installed in local lib directory?

Michael Madison

I don't know if this is relevant, but I'm having a weird problem with very simple sidekiq workers end up resulting in const_missing. it never gets inside the worker.

[177, 186] in /Users/epipheus/.rvm/gems/ruby-1.9.3-p327/gems/activesupport-4.0.0.rc1/lib/active_support/dependencies.rb
177 def const_missing(const_name)
178 # The interpreter does not pass nesting information, and in the
179 # case of anonymous modules we cannot even make the trade-off of
180 # assuming their name reflects the nesting. Resort to Object as
181 # the only meaningful guess we can make.
=> 182 from_mod = anonymous? ? ::Object : self
183 Dependencies.load_missing_constant(from_mod, const_name)
184 end

Karl Baum kbaum referenced this issue in activerecord-hackery/squeel
Closed

Polymorphic join unable to determine type #260

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.