deisctl fails to initialize ssh client with Vagrant 1.7

[mgartner]

I'm going through the latest Getting Started guide on deis.io for setting up a local vagrant cluster on OS X with v1.1.0, and when I got to Installing the Deis Platform any deisctl commands fail with:

Error: failed initializing SSH client: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

I've made sure to run add the vagrant ssh key with ssh-add ~/.vagrant.d/insecure_private_key.

I've looked through a couple of other issues where this error was mentioned, but nothing noted in those issues seems to help. Any help is appreciated.

[mboersma]

Does vagrant ssh deis-1 work? Perhaps try vagrant ssh-config deis-1 to verify that you have DEISCTL_TUNNEL set to the correct host/port and that it expects the vagrant insecure private key.

[pon]

I am in the same place as @mgartner - vagrant ssh deis-1 works fine, below is the output from my vagrant ssh-config deis-1

[mgartner]

I have the same results as @pon with vagrant ssh deis-1 and vagrant ssh-config deis-1.

[carmstrong]

I would suspect that your ssh-agent has many identities in it, and it's offering several incorrect ones before it gets to insecure_private_key. You could try removing all identities with ssh-add -D and then adding the insecure_private_key so it's the only one.

[mboersma]

@pon your vagrant ssh config is using a different private key than expected. Did you ssh-add that one, not ~/.vagrant.d/insecure_private_key? Here's my ssh config just for comparison:

$ vagrant ssh-config deis-1
Host deis-1
  HostName 127.0.0.1
  User core
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/matt/.vagrant.d/insecure_private_key
  IdentitiesOnly yes
  LogLevel FATAL

[carmstrong]

your vagrant ssh config is using a different private key than expected.

Good catch, @mboersma 👍

[mgartner]

Any idea why it would be using a different key by default? How would I change the key it uses? I tried adding config.ssh.private_key_path = '~/.vagrant.d/insecure_private_key' to the Vagrantfile with no luck.

[mboersma]

I'm not sure why vagrant would be using a key other than their default insecure_private_key. Did you see that too @mgartner in vagrant ssh-config deis-1?

It could be a vagrant packaging difference. I'm on OS X and I downloaded the installer from vagrantup.com. Was yours packaged differently, maybe from Homebrew?

Either way I don't think it's necessary to change vagrant's settings if vagrant ssh deis-1 already works. Let's verify that DEISCTL_TUNNEL=127.0.0.1:2222 (or whatever the SSH config is) and do ssh-add ~/workspace/deployments/deis/.vagrant/machines/deis-1/virtualbox/private_key (or wherever vagrant ssh-config deis-1 says the key is).

It's also a good idea to first have ssh-agent flush the keys it's serving and start over as @carmstrong said: ssh-add -D. We have seen that fix auth issues like this before.

[mgartner]

I installed Vagrant from vagrantup.com as well. I'll try reinstalling to be safe. What version of Vagrant are you using? I'm on 1.7.0.

I was able to get the SSH handshake to work after ssh-add'ing all three of the vagrant keys, but I think the fact that they were separate keys was creating more problems for me - not being able to connect to fleet.

I'll report back after the Vagrant re-install.

[mgartner]

No luck with the reinstall. After I add all 3 ssh keys I get the following when running deisctl list:

INFO client.go:291: Failed getting response from http://127.0.0.1:4001/: dial tcp 127.0.0.1:4001: connection refused
ERROR client.go:213: Unable to get result for {Get /_coreos.com/fleet/states}, retrying in 100ms
INFO client.go:291: Failed getting response from http://127.0.0.1:4001/: dial tcp 127.0.0.1:4001: connection refused
ERROR client.go:213: Unable to get result for {Get /_coreos.com/fleet/states}, retrying in 200ms
INFO client.go:291: Failed getting response from http://127.0.0.1:4001/: dial tcp 127.0.0.1:4001: connection refused
ERROR client.go:213: Unable to get result for {Get /_coreos.com/fleet/states}, retrying in 400ms
INFO client.go:291: Failed getting response from http://127.0.0.1:4001/: dial tcp 127.0.0.1:4001: connection refused
ERROR client.go:213: Unable to get result for {Get /_coreos.com/fleet/states}, retrying in 800ms
INFO client.go:291: Failed getting response from http://127.0.0.1:4001/: dial tcp 127.0.0.1:4001: connection refused
ERROR client.go:213: Unable to get result for {Get /_coreos.com/fleet/states}, retrying in 1s
INFO client.go:291: Failed getting response from http://127.0.0.1:4001/: dial tcp 127.0.0.1:4001: connection refused
ERROR client.go:213: Unable to get result for {Get /_coreos.com/fleet/states}, retrying in 1s
INFO client.go:291: Failed getting response from http://127.0.0.1:4001/: dial tcp 127.0.0.1:4001: connection refused
ERROR client.go:213: Unable to get result for {Get /_coreos.com/fleet/states}, retrying in 1s
INFO client.go:291: Failed getting response from http://127.0.0.1:4001/: dial tcp 127.0.0.1:4001: connection refused
ERROR client.go:213: Unable to get result for {Get /_coreos.com/fleet/states}, retrying in 1s
INFO client.go:291: Failed getting response from http://127.0.0.1:4001/: dial tcp 127.0.0.1:4001: connection refused
ERROR client.go:213: Unable to get result for {Get /_coreos.com/fleet/states}, retrying in 1s
INFO client.go:291: Failed getting response from http://127.0.0.1:4001/: dial tcp 127.0.0.1:4001: connection refused
ERROR client.go:213: Unable to get result for {Get /_coreos.com/fleet/states}, retrying in 1s

[mboersma]

$ vagrant --version
Vagrant 1.6.5

I didn't realize 1.7.0 was out, I hope it's not creating these issues. I'll give it a try.

The error message you just posted is actually an etcd error that probably indicates you didn't run make discovery-url before vagrant up. That usually happens when there is a stale or nonexistent etcd discovery URL. So ssh connectivity works, but the nodes aren't communicating with each other.

[mboersma]

Backwards compatibility between 1.x is not promised, and Vagrantfile syntax stability isn't promised until 2.0 final. Any backwards incompatibilities within 1.x will be clearly documented.

Clearly documented somewhere, I would imagine. Can't find it though. Last update by mitchellh on their blog was for 1.6.0. 😞

Update: found it https://github.com/mitchellh/vagrant/blob/v1.7.0/CHANGELOG.md#170-december-9-2014

[mboersma]

@mgartner and @pon, the issue here I believe is a backward-incompatible change made in Vagrant 1.7:

If the default insecure keypair is used, Vagrant will automatically replace it with a randomly generated keypair on first vagrant up. [GH-2608]

The workaround is to ssh-add one of these generated keys before using deisctl, or to install Vagrant 1.6.5.

[mgartner]

Nice find @mboersma! Thanks for your help in diagnosing this. Let me know if there is anything I can provide to help fix this.

[mboersma]

Since deisctl only needs SSH to 1 of the nodes, it's sufficient for now to retrieve the key path with vagrant ssh-config deis-1 and just ssh-add that one, not necessarily all 3+ keys. We should probably document how to generate your own key and use it in the Vagrantfile, so this workflow is more similar to other providers. I'll make a PR for that.