Delayed job 3.0.1 not working, breaks on to_yaml #350

Closed
coneybeare opened this Issue Feb 4, 2012 · 22 comments

Projects

None yet
@coneybeare

Backtrace

[GEM_ROOT]/gems/delayed_job-3.0.1/lib/delayed/psych_ext.rb:6:in `encode_with'
[GEM_ROOT]/gems/activerecord-3.2.0/lib/active_record/base.rb:646:in `to_yaml'
/usr/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/syck/rubytypes.rb:19:in `node_export'
/usr/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/syck/rubytypes.rb:19:in `add'
/usr/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/syck/rubytypes.rb:19:in `block (3 levels) in to_yaml'
/usr/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/syck/rubytypes.rb:18:in `each'
/usr/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/syck/rubytypes.rb:18:in `block (2 levels) in to_yaml'
/usr/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/syck/rubytypes.rb:17:in `map'
/usr/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/syck/rubytypes.rb:17:in `block in to_yaml'
/usr/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/syck.rb:401:in `call'
/usr/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/syck.rb:401:in `emit'
/usr/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/syck.rb:401:in `quick_emit'
/usr/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/syck/rubytypes.rb:16:in `to_yaml'
[GEM_ROOT]/gems/delayed_job-3.0.1/lib/delayed/backend/base.rb:80:in `payload_object='
[GEM_ROOT]/gems/activerecord-3.2.0/lib/active_record/attribute_assignment.rb:85:in `block in assign_attributes'
[GEM_ROOT]/gems/activerecord-3.2.0/lib/active_record/attribute_assignment.rb:78:in `each'
[GEM_ROOT]/gems/activerecord-3.2.0/lib/active_record/attribute_assignment.rb:78:in `assign_attributes'
[GEM_ROOT]/gems/activerecord-3.2.0/lib/active_record/base.rb:488:in `initialize'
[GEM_ROOT]/gems/delayed_job-3.0.1/lib/delayed/backend/base.rb:28:in `new'
[GEM_ROOT]/gems/delayed_job-3.0.1/lib/delayed/backend/base.rb:28:in `enqueue'
[GEM_ROOT]/gems/delayed_job-3.0.1/lib/delayed/message_sending.rb:13:in `method_missing'
$ rails -v
Rails 3.2.0
$ ruby -v
ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-darwin11.3.0]

I have verified that the call works when called normally. The database migration for 3.0 has been applied.

@coneybeare

Any updates to this issue?

@duksis
duksis commented Feb 15, 2012

Could you provide a gist with the content of your Gemfile?

@mbhnyc
mbhnyc commented Feb 16, 2012

I'm seeing this as well on attempting to execute an asynchronous mailer delivery.

Bundle output: https://gist.github.com/760bbb3b995a57a6c6f3

Mailer: https://gist.github.com/8c6b49176939e8db182e

Fails on running
ProfileMailer.new_message_email(Profile.first).deliver

With a very similar error:
ArgumentError: wrong number of arguments (1 for 0)
from /Users/mhensrud/.rvm/gems/ruby-1.9.2-p290/gems/mail-2.3.0/lib/mail/message.rb:1714:in to_yaml' from /Users/mhensrud/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/yaml_tree.rb:55:inaccept'
from /Users/mhensrud/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/yaml_tree.rb:325:in `block in dump_ivars'

@patrick99e99

Yeah I am a little confused by this (in psych_ext.rb):

class ActiveRecord::Base
  def encode_with(coder)
    coder["attributes"] = @attributes
    coder.tag = ['!ruby/ActiveRecord', self.class.name].join(':')
  end
end

... ActiveRecord's has in it:

  # Hackery to accomodate Syck. Remove for 4.0.
  def to_yaml(opts = {}) #:nodoc:
    if YAML.const_defined?(:ENGINE) && !YAML::ENGINE.syck?
      super
    else
      coder = {}
      encode_with(coder)
      YAML.quick_emit(self, opts) do |out|
        out.map(taguri, to_yaml_style) do |map|
          coder.each { |k, v| map.add(k, v) }
        end 
      end 
    end 
  end 

So, in my case, YAML::ENGINE.yamler is syck, so encode_with() is being called with {} (an empty hash) as the argument... Then encode_with is being overwritten by delayed_job's psyche_ext.rb, and it's trying to call .tag= on that empty hash, which does not respond to that method name thus giving me:

NoMethodError: undefined method `tag=' for #<Hash:0x0000001ebfcd40>
from /home/deploy/rails_apps/myapp/shared/bundle/ruby/1.9.1/gems/delayed_job-3.0.1/lib/delayed/psych_ext.rb:6:in `encode_with'
from /home/deploy/rails_apps/myapp/shared/bundle/ruby/1.9.1/gems/activerecord-3.2.1/lib/active_record/base.rb:653:in `to_yaml'
from (irb):1
from /home/deploy/rails_apps/myapp/shared/bundle/ruby/1.9.1/gems/railties-3.2.1/lib/rails/commands/console.rb:47:in `start'
from /home/deploy/rails_apps/myapp/shared/bundle/ruby/1.9.1/gems/railties-3.2.1/lib/rails/commands/console.rb:8:in `start'
from /home/deploy/rails_apps/myapp/shared/bundle/ruby/1.9.1/gems/railties-3.2.1/lib/rails/commands.rb:41:in `<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'
@patrick99e99

Ok-- for me there were two issues here...

  1. we are using Syck instead of Psych... When the server launches, the gems are loaded before the environment / initializers, so delayed_job's yaml_ext.rb was hit first, and the YAML.parser.class was Psych..
require 'yaml'

if YAML.parser.class.name =~ /syck|yecht/i
  require File.expand_path('../syck_ext', __FILE__)
  require File.expand_path('../serialization/active_record', __FILE__)
else
  require File.expand_path('../psych_ext', __FILE__)
end

But then in our environment we were setting YAML::ENGINE.yamler = "syck"

... So ActiveRecord would know we were using Syck, but DelayedJob would still be thinking we're using Psych... So the solution was to put the YAML::ENGINE.yamler setter into boot.rb so that it got set prior to gems being loaded.

  1. Several months ago, Delayed Job's syck_ext.rb file had in it:
def yaml_tag_read_class(name)
  # Constantize the object so that ActiveSupport can attempt
  # its auto loading magic. Will raise LoadError if not successful.
  #
  # When requiring yaml, the parsers redefine the YAML constant. This causes an
  # issue with poorly formatted yaml, specifically in the case of a Bad Alias.
  # When you'd expect to see Syck::BadAlias, the name we're getting is
  # YAML::Syck::BadAlias and trying to constantize this results in an uninitialized constant Syck::Syck.
  name.gsub(/^YAML::/, '').constantize
  name
end

This method is now currently:

def yaml_tag_read_class(name)
  # Constantize the object so that ActiveSupport can attempt
  # its auto loading magic. Will raise LoadError if not successful.
  name.constantize
  name
end

... So the solution was to create a config/initializers/delayed_job.rb file and put in it:

# to fix issue with uninitialized constant Syck::Syck error
class Module

  def yaml_tag_read_class(name)
    # Constantize the object so that ActiveSupport can attempt
    # its auto loading magic. Will raise LoadError if not successful.
    name.gsub!(/^YAML::/, '') if name =~ /BadAlias/
    name.constantize
    name
  end 

end

and now everything is working... BTW, this uninitialized constant Syck::Syck error was being triggered by a serialized column being set in a before_save callback that was happening via an autosave association.

@coneybeare

For me, my setup does nothing to adjust the YAML at all so it should work out of the box.

@patrick99e99

Go into your rails console and type "YAML.parser.class" ... Is is Psych or Syck?

@coneybeare
1.9.3p0 :004 > YAML.parser.class
 => Syck::Parser 

Am I missing an initializer somewhere that forces it to be psych?

@patrick99e99

Yeah so this is the same thing that was happening for me.. when your rails app boots up, it is using Psych, and DelayedJob picks up on that and therefore uses the psych_ext.rb file instead of syck_ext.rb... but then somewhere during or after your app's initialization, the yaml parser is being set to Syck.

You could try putting in your environment.rb file:

require 'yaml'
YAML::ENGINE.yamler = 'psych'

either that or if you want to stick with syck, do what I did and put in boot.rb before your gemfile is picked up:

require 'yaml'
YAML::ENGINE.yamler = 'syck'
@coneybeare

Forcing psych in environment fixed it. Thanks.

@coneybeare coneybeare closed this Feb 17, 2012
@BBonifield

This should really be reopened. Modifying my boot.rb in this fashion is just a hack. At the very least, this issue should be noted in the project wiki.

@inspire22

+1 same problem for me, thanks for the discussion

@hrdwdmrbl

Yup, I still have issues with this too. But adding

require 'yaml'
YAML::ENGINE.yamler = 'psych'

to environment.rb worked. So +1 to @patrick99e99

@cschmitt
cschmitt commented Mar 6, 2013

I agree with this.. I have an out of the box setup and am getting this when attempting to execute tests. Any chance this will be fixed going forward

@lorenjohnson

+1 I'm using the boot.rb fix and was not having an issue and then started having it "out of the blue" and haven't yet figured-out what changed in our config to cause it. However, the boot.rb overrides before loading the Gemfile did fix it for us. I'll look into the deeper fix when/if I find time.

@jygrinberg

+1 for the boot.rb fix. After hours of bashing my head, this fixed it!

@cronuscronus

+1

Switching from psych to syck in boot.rb fixed the following madness:
"Job failed to load: private method `allocate' called for SolrObserver:Class. Handler" [rest of stack trace omitted]

I'm not sure why Psych fails for me with Ruby 1.9.3 and Rails 3.2.12?

@jhironaga

@cronuscronus, I had the same problem. Here's my limited understanding...

PROBLEM
I assume you were delaying a SolrObserver instance method. Because observers are singletons, you get that allocate() error when Delayed Job attempts to recreate the serialized singleton instance.

SOLUTION 1
Move the delay logic to class methods, so the class gets serialized instead. Example: https://coderwall.com/p/tieoea

SOLUTION 2
I was also able to revert to syck, but I have no idea how that gets around the nature of observers. o_O

Anyway, hope this helps in case you ever need to switch back to psych.

@cronuscronus

@jhironaga I agree that trying to delay a singleton method was a bad idea, and I've since moved the logic into non-singleton classes, and then moved the delay there from the observer.

After looking into Psych a little bit more, it looks like the revive method calls allocate:
https://github.com/tenderlove/psych/blob/v1.3.4/lib/psych/visitors/to_ruby.rb

Allocate requires that the object being allocated be an instance of class. Given that a singleton is not, this explains the exception..
http://ruby-doc.org/core-1.9.3/Class.html#method-i-allocate

Whether or not Psych's inability to load a singleton is a bug or not I don't know. Perhaps Aaron Patterson can answer that one.

@Djo
Djo commented May 15, 2014

Note for Ruby 2.0 / Rails:

YAML Syck engine support was dropped in Ruby 2.0 and it's available only by a gem 'syck'.
Switching to the engine could be done in boot.rb before loading all gems in application.rb:

require 'yaml'
require 'syck'
YAML::ENGINE.yamler = 'syck'

Gist with the whole boot.rb: https://gist.github.com/Djo/c8a6ca3ff5661d226276

@hrdwdmrbl

What's the reason DJ can't just abandon Syck? Seems like DJ is attempting to swim upstream.

@mconf-daileon mconf-daileon referenced this issue in mconf/mconf-web May 8, 2016
Closed

Replacement for delayed_job #437

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment