NoMethodError: undefined method on initial jobs #331
Comments
|
Does putting |
|
Ill have to check that but I dont see a similar resolution to the first one I listed NoMethodError: undefined method `initial_count_for' for #ActiveRecord::Associations::AliasTracker:0x000000064d68a8 I assume we don't have to be requiring Rails ActiveRecord libraries manually right? |
|
So yes, the HttpHelper class is in our lib dir and that removed that issue. Although I would like to add that it was inconsistent, it didn't happen every time we started our jobs. We'll see when it runs tonight. Unfortunately the ActiveRecord undefined method issues still happen and Im not sure how to explicitly require that where we need it. What do you think? Also why does requiring our HttpHelper class resolve the first issue? Does that explicit require help Sidekiq to ensure its included? Just curious. |
|
Add lib to Rails eager load directory set. Everyone does this but it's not enabled by default. |
config.autoload_paths += ["#{config.root}/lib"] |
|
Awesome thank you, ill let you know if the other issue still shows up after we add this. |
|
So I may have spoken too soon, this still seems to be happening. We are mainly seeing issues related to ActiveRecord at this point though. Stuff like this:
These errors showed up on a find by id and count SQL query respectively. The errors happen at a rate of 100s out of 1 million jobs so it is very rare but something we would like to get to the bottom of. Just to be clear, these million jobs are all exactly the same and the lines of code run fine for all of the other jobs. Also the errored jobs are not for the same objects each night, just randomly interspersed although it seems to happen mainly at the beginning of our nightly runs. Any ideas why the internals of ActiveRecord would not be loaded into memory when Sidekiq is processing a job? I won't reopen the issue, ill let you do that if you think something might be wrong with the Sidekiq internals. |
|
@tarcieri He's seeing mysterious undefined method errors in various core ActiveRecord components that are autoloaded. Could this be an autoloading race condition at fault on 1.9.2 or does that only happen on JRuby with parallel threading? |
|
autoload is not thread safe on 1.9.3. There are autoload bugs on 1.9.3/YARV that have only been fixed in Ruby 2.0/YARV. All code you need for your program should be loaded prior to spawning additional threads. |
|
This is a Rails 3.2 bug which does not eager load some of the ActiveRecord core API. The fix is in master and I've asked for a backport: rails/rails@8cf300c |
|
Yes, @tarcieri is correct. And please do not add lib to your eager_load directory. You should require whatever is in lib and that you want to use, as you would do with any other library. |
|
Done @ rails/rails@29fb31f |
|
@TheCorp you can upgrade to Rails 3.2.8 when it ships to get the fix. I believe they've already done a release candidate or two so it should be quite soon. |
|
This will not released with 3.2.8, the release branch was created before this commit. We will release at 3.2.9 |
|
Is there a workaround until 3.2.9 is available? I'm getting similar errors with methods like |
|
@Gurpartap Write an initializer that exercises ActiveRecord, loads models from the database, walks associations, etc so as much of AR as possible is loaded. It's a hack but it should work. The rails initializers are run single-threaded before Sidekiq starts all of its threads. |
|
Just did this and it worked perfectly. We also were seeing this issue with a few other gems like CarrierWave so the same fix worked as well to ensure it gets preloaded. Thanks. |
|
I'm using Rails 3.2.9 with Sidekiq 2.5.3 on Heroku. Still getting a missing method for first few job performs. Here's the log: Would this be related to this issue as it happens only during the quick first run? |
|
Yup its the same issue, just create an initializer that loads some models with ActiveRecord and you should be good to go. Its not finding the default_scope method so maybe try to make sure you are loading one of your models that has a default scope. |
|
Am I correct in thinking that the following happens because of the exact same problem? It seems as if the ActiveRecord style method chaining simply isn't working. I use @TheCorp, do you have an example on how to solve this with an initializer. |
|
I don't know if that's the same error as the ones everyone else is having above. I don't think there should ever be a method "id" on a Hash object? Is "spot" one of your models? What is "project"? Is that a relation? If so that should be an array or at least another model, it should never resolve to a Hash class I don't think. Most likely the project attribute is a serialized field since that's what would be Hash and then there is definitely no method "id" for that object. I can show you an example of the initializer but I doubt it would resolve your issue here. |
|
@TheCorp I solved the issue, a co-worker mentioned it was generally a bad idea to sent objects to Sidekiq, and instead to send ID's and get the object from the ID. This solved my issue. Still, if you have an example at the ready, it might help other people finding this topic. |
|
Ya Sidekiq can't be sent objects like DelayedJob can so that's definitely the case. As for how to resolve the issue, all you have to do is put a single line in your sidekiq initializer for each issue you are having with methods being missing, something like this: Anything simple like that will ensure the classes get loaded before Sidekiq uses them. |
|
I've been running into the same issue with Sidekiq, however I'm not sending objects around as @JeanMertz mentioned. Instead I'm passing by ID as is recommended, however when I go to call the ActiveRecord method update_attributes it fails with: This is the only thread where I've seen other people with the same issue. I'm on Rails 3.2.11 |
|
See the referenced issues for how to fix that. It's here: #331 |
|
@TheCorp , you just linked back to this issue :) It sounds like the main solution is an initializer with some arbitrary ActiveRecord request ... is that correct? |
|
Oh my bad sorry! Ya definitely, use the initializer method which does any arbitrary ActiveRecord call and that should make sure everything is in memory before any Sidekiq workers get to it. |
|
{"class":"PostmanWorker","args":["lmd"],"retry":true,"queue":"critical","jid":"04a7ac2fa0b59a792c6d72e4","created_at":1456753992.873091,"enqueued_at":1456754044.180575,"error_message":"undefined method `perform' for #<PostmanWorker:0x007fd28ce293a0 @jid="04a7ac2fa0b59a792c6d72e4">","error_class":"NoMethodError","failed_at":1456754009.4919481,"retry_count":1,"retried_at":1456754044.182247} Im also facing NoMethodError |
|
@biznickman did the arbitrary ActiveRecord call solve your problem? I am facing a problem similar to yours. Unfortunately, the ActiveRecord call in sidekiq initializer did not work for me. In config/initializers/sidekiq.rb, I added: Rails 4.2.1 |
So, not entirely sure about this issue but maybe you can help us track down the cause. I went through the existing open and closed issues and haven't found anything yet that seems related to this.
Basically right when we start Sidekiq workers up (on Heroku) the first few jobs always seem to error out with "NoMethodError" or "uninitialized constant" errors. For example:
NoMethodError: undefined method `initial_count_for' for #ActiveRecord::Associations::AliasTracker:0x000000064d68a8
NameError: uninitialized constant OneOfOurModels::HttpHelper
etc
initial_count_for is definitely a method in ActiveRecord and HttpHelper is a ruby library that should always be available. There are other instances of this if you want more examples. Basically it seems like the full Rails/Ruby environment hasn't been loaded into memory for the thread processing this Job?
After a minute (or less) all the jobs seem to process just fine so it seems like once things are warmed up and everything is in memory all of the jobs seem to have access to all of the functions that get mixed in. Here's our setup:
Ruby 1.9.2p290
Rails 3.2
Heroku Cedar
Let me know if this makes any sense or if you need more details about our config.
The text was updated successfully, but these errors were encountered: