Broken autoloading classes in lib #80

Closed
canavese opened this Issue Mar 15, 2012 · 13 comments

Comments

6 participants

I'm new to sidekiq, so it's quite possible that I missed something here, but I'm having issues with autoloading classes in my lib directory. Rails 3.2.2. The lib directory is in my Rails config.autoload_paths, and I haven't had problems outside of sidekiq.

I'm running with this: bundle exec sidekiq -c 25

I've had the problem with multiple lib classes, so it's not an issue with just one. I also double-checked that the file names match the class. Example stack trace below. The file referenced does indeed define the named class (matching case and all).

Adding a require in the calling file fixes the problem, but that's obviously annoying.

Ideas?

/Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:503:in `load_missing_constant': Expected /Users/canavese/workspace/neighboring2/lib/datetime_extractor.rb to define DatetimeExtractor (LoadError)
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:192:in `block in const_missing'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:190:in `each'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:190:in `const_missing'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/aws-s3-0.6.2/lib/aws/s3/extensions.rb:206:in `const_missing_from_s3_library'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:514:in `load_missing_constant'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:192:in `block in const_missing'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:190:in `each'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:190:in `const_missing'
    from /Users/canavese/.rvm/gems/ruby-1.9.3-p0@neighboring/gems/aws-s3-0.6.2/lib/aws/s3/extensions.rb:206:in `const_missing_from_s3_library'
    from /Users/canavese/workspace/neighboring2/app/models/publish/publish_analyzer.rb:63:in `block in analyze'
Owner

mperham commented Mar 15, 2012

I suspect you have other load issues which are presenting themselves as this error.

Owner

mperham commented Mar 20, 2012

I just tried your scenario locally and it worked as expected. Let me know if you learn anything more that points to a Sidekiq issue.

mperham closed this Mar 20, 2012

Thanks for looking at it a bit. It definitely wasn't a problem with all auto-loading anything from lib. It seems inconsistent and may be related to load order. But I'll ping you again if I can figure out a more direct association with sidekiq, but you may very well be right that it's a general load issue that I just happen to only be hitting in sidekiq.

Contributor

fbjork commented Apr 9, 2012

I'm also seeing autoload issues in my Sidekiq workers at random. It doesn't happen in other processes (web, rake tasks etc). The solution seems to be to require the files instead of relying on autoloading for now.

Contributor

ryanlecompte commented Apr 9, 2012

Right, autoloading in Ruby is not thread-safe. It's best to eagerly load classes in third-party libraries if you're using them in a multithreaded fashion (i.e., via Sidekiq). I am doing the following eager-loading for the following gems:

  # eagerly load classes for aws-sdk gem
  AWS.eager_autoload!

  # eagerly load classes for faraday gem
  Faraday.load_autoloaded_constants
Owner

mperham commented Apr 9, 2012

I call Rails.application.eager_load! to eager load your app/models before the Sidekiq Processors are even created but my guess is that Rails does not eager load lib/ and other directories manually added to the autoload_path.

Contributor

fbjork commented Apr 9, 2012

Wonder if the following would work?

config.eager_load_paths += %W(#{config.root}/lib/stuff)
Owner

mperham commented Apr 9, 2012

That looks like a brilliant plan to me.

Are there clear best practices thread safety? Are we required to call Rails.application.eager_load!, and if so, where? Should config.threadsafe! be enabled?

Owner

mperham commented May 10, 2012

@jsierles You need to do nothing.

Sidekiq automatically calls eager_load! when it boots your application. Sidekiq doesn't explicitly invoke threadsafe! but since there's no web part of Sidekiq, there's no reloading happening at the end of a web request.

If you have threading or loading problems, open an issue and we'll figure out the problem.

OK, thanks for the clarification.

On Thursday, May 10, 2012 at 5:24 PM, Mike Perham wrote:

@jsierles You need to do nothing.

Sidekiq automatically calls eager_load! when it boots your application. Sidekiq doesn't explicitly invoke threadsafe! but since there's no web part of Sidekiq, there's no reloading happening at the end of a web request.

If you have threading or loading problems, open an issue and we'll figure out the problem.


Reply to this email directly or view it on GitHub:
#80 (comment)

epipheus commented Jun 8, 2013

Does this mean any gem needs to be installed in local lib directory?

epipheus commented Jun 8, 2013

I don't know if this is relevant, but I'm having a weird problem with very simple sidekiq workers end up resulting in const_missing. it never gets inside the worker.

[177, 186] in /Users/epipheus/.rvm/gems/ruby-1.9.3-p327/gems/activesupport-4.0.0.rc1/lib/active_support/dependencies.rb
177 def const_missing(const_name)
178 # The interpreter does not pass nesting information, and in the
179 # case of anonymous modules we cannot even make the trade-off of
180 # assuming their name reflects the nesting. Resort to Object as
181 # the only meaningful guess we can make.
=> 182 from_mod = anonymous? ? ::Object : self
183 Dependencies.load_missing_constant(from_mod, const_name)
184 end

kbaum referenced this issue in activerecord-hackery/squeel Jul 27, 2013

Closed

Polymorphic join unable to determine type #260

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment