Einhorn incompatible with ruby 3.1 / psych 4

[jdelStrother]

Ruby version: 3.1.2
Rails version: 7.0.3
Sidekiq / Pro / Enterprise version(s): 6.5.1

I appreciate this isn't sidekiq's fault, but given that Einhorn is the officially recommended solution for rolling restarts in Sidekiq Enterprise, I was hoping to get some suggestions on how to deal with it...

We're in the middle of upgrading a ruby 3.0 installation to 3.1, which comes with Psych 4.
Einhorn successfully starts up in 3.1, but if you then try to reload it (einhornsh --execute upgrade), I get this output -

Jun 20 16:09:35 staging1-3 sidekiq[19159]: [MASTER 19159] INFO: Reloading einhorn master (/usr/local/bin/einhorn)...
Jun 20 16:09:35 staging1-3 sidekiq[19284]: Upgrading smoothly, as commanded
Jun 20 16:09:35 staging1-3 sidekiq[19159]: [MASTER 19286] INFO: Running under Ruby 3.1.2
Jun 20 16:09:35 staging1-3 sidekiq[19159]: [MASTER 19286] INFO: Using Bundler 2.3.14
Jun 20 16:09:35 staging1-3 sidekiq[19159]: [MASTER 19159] INFO: About to re-exec einhorn master as ["/app/audioboom/bin/einhorn-audioboom", ["--with-state-fd", "9", "--", "/usr/local/bin/bundle", "exec", "sidekiq", "-e", "staging", "-c", "10"], {:close_others=>false}]
Jun 20 16:09:35 staging1-3 sidekiq[19159]: /usr/local/lib/ruby/3.1.0/psych/class_loader.rb:99:in `find': Tried to load unspecified class: Set (Psych::DisallowedClass)
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/class_loader.rb:28:in `load'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:424:in `resolve_class'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:213:in `visit_Psych_Nodes_Mapping'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/visitor.rb:30:in `visit'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/visitor.rb:6:in `accept'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:35:in `accept'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:345:in `block in revive_hash'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `each'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `each_slice'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `revive_hash'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:167:in `visit_Psych_Nodes_Mapping'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/visitor.rb:30:in `visit'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/visitor.rb:6:in `accept'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:35:in `accept'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:345:in `block in revive_hash'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `each'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `each_slice'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `revive_hash'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:167:in `visit_Psych_Nodes_Mapping'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/visitor.rb:30:in `visit'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/visitor.rb:6:in `accept'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:35:in `accept'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:345:in `block in revive_hash'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `each'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `each_slice'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `revive_hash'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:167:in `visit_Psych_Nodes_Mapping'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/visitor.rb:30:in `visit'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/visitor.rb:6:in `accept'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:35:in `accept'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:345:in `block in revive_hash'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `each'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `each_slice'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `revive_hash'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:167:in `visit_Psych_Nodes_Mapping'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/visitor.rb:30:in `visit'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/visitor.rb:6:in `accept'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:35:in `accept'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:318:in `visit_Psych_Nodes_Document'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/visitor.rb:30:in `visit'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/visitor.rb:6:in `accept'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:35:in `accept'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych.rb:335:in `safe_load'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/psych.rb:370:in `load'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/gems/3.1.0/gems/einhorn-0.8.2/lib/einhorn.rb:94:in `restore_state'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/gems/3.1.0/gems/einhorn-0.8.2/bin/einhorn:304:in `block (2 levels) in <top (required)>'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/optparse.rb:1576:in `block in parse_in_order'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/optparse.rb:1559:in `catch'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/optparse.rb:1559:in `parse_in_order'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/3.1.0/optparse.rb:1553:in `order!'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/lib/ruby/gems/3.1.0/gems/einhorn-0.8.2/bin/einhorn:333:in `<top (required)>'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/bin/einhorn:25:in `load'
Jun 20 16:09:35 staging1-3 sidekiq[19159]:         from /usr/local/bin/einhorn:25:in `<main>'

Replacing YAML.load here with YAML.unsafe_load seems to work ok - there's a few other YAML.loads in the codebase, but AFAICT they're not trying to load any instances of Set.

Has anyone else run into this?

[jdelStrother]

It looks like if I upgraded psych from 4.0.3 to 4.0.4 that would probably fix it (ruby/psych#539). I'll try tomorrow & report back...

[EDIT: actually, scratch that - even with that pull request, 4.0.4 has safe loading by default. https://github.com/ruby/psych/pull/487 ]

[mperham]

I believe they need to switch to safe_load and whitelist the Set class as Sidekiq does here with Symbol:

https://github.com/mperham/sidekiq/blob/e830ecd7ac1539877de03e57fb4807765f5fa392/lib/sidekiq/cli.rb#L404

If they do that, all should work. If no one else is willing to maintain it, I will need to fork it and start my own version.

[petergoldstein]

@mperham For what it's worth, Stripe appears to have deprecated the project - https://github.com/stripe-archive/einhorn

[mperham]

I know, that’s why I just asked for new maintainers on Twitter. I’ll fork if I can’t get official blessing.

[mperham]

https://github.com/contribsys/einhorn/blob/main/Changes.md#100