Error "expected 64-byte String, got 3"

[ekampp]

I'm getting the following error:

SSHKit::Runner::ExecuteError: Exception while executing as <USER>@<IP>: expected 64-byte String, got 3

I have recently begun upgrading my ssh keys to the new ECDSA, and this is the server where it fails.

The initial error message was that I needed to add the ed25519 and bcrypt_pbkdf gems, which I did.

But this error message is less helpful.

I have tried manually connecting to the server with SSH which works fine.

Here's the Capistrano log:

  INFO ---------------------------------------------------------------------------
  INFO START 2018-10-23 22:12:25 +0200 cap production deploy
  INFO ---------------------------------------------------------------------------
  INFO [fb5e8e91] Running /usr/bin/env mkdir -p /tmp as <USER>@<IP>
 DEBUG [fb5e8e91] Command: /usr/bin/env mkdir -p /tmp

Here's the simplest replication code that I could come up with:

require 'sshkit'
require 'sshkit/dsl'
include SSHKit::DSL

on %w{<IP>}, in: :sequence, wait: 5 do |host|
  within "/path/to/rails/app" do
    as :deploy  do
      with rails_env: :production do
        runner "Rails.env"
      end
    end
  end
end

And here's the backtrace:

5: from ./sshkit_error_demo.rb:5:in `<main>'
4: from /Users/emil/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/sshkit-1.18.0/lib/sshkit/dsl.rb:6:in `on'
3: from /Users/emil/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/sshkit-1.18.0/lib/sshkit/coordinator.rb:21:in `each'
2: from /Users/emil/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/sshkit-1.18.0/lib/sshkit/runners/sequential.rb:22:in `execute'
1: from /Users/emil/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/sshkit-1.18.0/lib/sshkit/runners/sequential.rb:27:in `run_backend'
/Users/emil/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/sshkit-1.18.0/lib/sshkit/runners/sequential.rb:31:in `rescue in run_backend': Exception while executing on host api1.arctic-project.io: expected 64-byte String, got 3 (SSHKit::Runner::ExecuteError)

Thanks in advance

[ekampp]

Ps. It looks a little like something is missing from the error message. It looks "cut off" at the "3", but that is, in fact, the whole error message.

[dschwetz]

I just ran into this in one of my own projects. It seems that a recent change within the last couple of weeks (I haven't yet figured out what or where) requires the use of ED25519 keys instead of RSA keys when deploying via Capistrano (or, more specifically, when using sshkit). That may not be a fair assessment because these libraries seem to use system tools are that system dependent, so it may have been a recent update to the underlying version of my client OpenSSH or something on the server side. But it seems that the execute command in sshkit invokes a call stack that invariably terminates in a from_keypair call that lives in ed25519.rb (from the ed25519 gem). I've included my full stack trace below in case it helps. It just always seem to expect to negotiate using a ed25519 key, and throws an error when the parameters are derived from a RSA key.

Regardless, I fixed this in my project by generating a ED25519 key and adding it to my production server , updating my ~/.ssh/config with the new identity information. For anyone that stumbled here from google, see this blog for a how-to.

Good luck, I hope this helps.

My client is running OpenSSH 7.8p1. Full stack trace from the error I recieved (it was just executing which passenger on the production server) below:

Caused by:
ArgumentError: expected 64-byte String, got 3
/home/user/.rvm/gems/ruby-2.3.1/gems/ed25519-1.2.4/lib/ed25519/signing_key.rb:20:in `from_keypair'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/ed25519.rb:18:in `initialize'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/ed25519.rb:122:in `new'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/ed25519.rb:122:in `initialize'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/ed25519.rb:146:in `new'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/ed25519.rb:146:in `read'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/key_factory.rb:114:in `block in classify_key'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/key_factory.rb:61:in `load_data_private_key'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/key_factory.rb:44:in `load_private_key'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/key_manager.rb:142:in `sign'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/methods/publickey.rb:62:in `authenticate_with'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/methods/publickey.rb:20:in `block in authenticate'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/key_manager.rb:122:in `block in each_identity'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/key_manager.rb:119:in `each'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/key_manager.rb:119:in `each_identity'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/methods/publickey.rb:19:in `authenticate'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/session.rb:85:in `block in authenticate'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/session.rb:71:in `each'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh/authentication/session.rb:71:in `authenticate'
/home/user/.rvm/gems/ruby-2.3.1/gems/net-ssh-5.0.2/lib/net/ssh.rb:246:in `start'
/home/user/.rvm/gems/ruby-2.3.1/gems/sshkit-1.18.0/lib/sshkit/backends/connection_pool.rb:59:in `call'
/home/user/.rvm/gems/ruby-2.3.1/gems/sshkit-1.18.0/lib/sshkit/backends/connection_pool.rb:59:in `with'
/home/user/.rvm/gems/ruby-2.3.1/gems/sshkit-1.18.0/lib/sshkit/backends/netssh.rb:176:in `with_ssh'
/home/user/.rvm/gems/ruby-2.3.1/gems/sshkit-1.18.0/lib/sshkit/backends/netssh.rb:129:in `execute_command'
/home/user/.rvm/gems/ruby-2.3.1/gems/sshkit-1.18.0/lib/sshkit/backends/abstract.rb:145:in `block in create_command_and_execute'
/home/user/.rvm/gems/ruby-2.3.1/gems/sshkit-1.18.0/lib/sshkit/backends/abstract.rb:145:in `tap'
/home/user/.rvm/gems/ruby-2.3.1/gems/sshkit-1.18.0/lib/sshkit/backends/abstract.rb:145:in `create_command_and_execute'
/home/user/.rvm/gems/ruby-2.3.1/gems/sshkit-1.18.0/lib/sshkit/backends/abstract.rb:59:in `test'
/home/user/.rvm/gems/ruby-2.3.1/gems/capistrano-passenger-0.2.0/lib/capistrano/tasks/passenger.cap:42:in `block (3 levels) in <top (required)>'
/home/user/.rvm/gems/ruby-2.3.1/gems/sshkit-1.18.0/lib/sshkit/backends/abstract.rb:29:in `instance_exec'
/home/user/.rvm/gems/ruby-2.3.1/gems/sshkit-1.18.0/lib/sshkit/backends/abstract.rb:29:in `run'
/home/user/.rvm/gems/ruby-2.3.1/gems/sshkit-1.18.0/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
Tasks: TOP => rvm:hook => passenger:rvm:hook => passenger:test_which_passenger

[mattbrictson]

This is most likely an issue with net-ssh. I see you already opened an issue there (net-ssh/net-ssh#638), so I will close this one.

My take is that you have either a corrupt key in ~/.ssh or net-ssh is misinterpreting one of those keys as a being in ed25519 format, when it actually is something different.

I would start removing keys from ~/.ssh (move them aside, don't delete them) until the error stops. Then you will at least know which key is causing net-ssh to get confused.

[ekampp]

Hi @mattbrictson

This was not a problem in earlier versions, and this is not a problem with the actual SSH keys, which works fine when doing direct SSH authentication through openssh. So it seems extremely unlikely that my keys are misrepresenting and/or corrupt.

I don't disagree that it seems most likely to be a problem with net/ssh. But simply shuffling the responsibility back onto "it must be your keys" in order to close the issue seems cheap.

[mattbrictson]

Ouch! I was trying to offer some help and I think you're overreacting a bit here.

Before I wrote my comment I spent time digging into the net-ssh source code to troubleshoot your issue and came up with what I thought was the best course of action. When I said, "net-ssh is misinterpreting one of [your] keys". That does not mean it is your "fault". It means that net-ssh likely has a bug in its key-loading algorithm.

It sounds like we're in agreement that this is more likely a net-ssh bug, than an sshkit one. As a general rule we close issues that are due to upstream bugs (a policy that I think is fairly common in open source projects), hence my decision to close this one. My apologies if that came off the wrong way.

[ekampp]

Sorry if that came off as blunt! I don't know how much or little you debugged prior to closing. But neither did it seem that you're particularly interested in fixing the problem. That may be my misinterpretation.

Let's hope a solution is found within the net/ssh library.

[mszyndel]

Upgrading net-ssh from 5.0.2 to 5.1.0 solved this issue for me.

[sadhakbj]

ssh-add on terminal fixed my issue

[tvcam]

Upgrade net-ssh away from 5.0.2 fixed the issue.

[nosolopau]

I was facing this issue with net-ssh 4.0.1, updated it to 6.1.0 and it worked. Thanks @mszyndel!