New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSH host key: is a #106

Closed
hackzilla opened this Issue Aug 11, 2014 · 30 comments

Comments

@hackzilla

hackzilla commented Aug 11, 2014

I've followed coreos vmware guide, and chosen stable release.

https://coreos.com/docs/running-coreos/platforms/vmware/

However once the system boots up, the login screen says: SSH host key: is a
and I can't ssh into the VM.

screen shot 2014-08-11 at 16 46 51

@crawford crawford added os labels Aug 11, 2014

@crawford

This comment has been minimized.

Member

crawford commented Aug 11, 2014

There seems to be a race condition with the key generation. To unblock you, try creating the VM again. If that doesn't work, stop the machine in the early boot prompt (hit any key within half a second) and enter boot_kernel coreos.autologin to disable the password check.

@hackzilla

This comment has been minimized.

hackzilla commented Aug 12, 2014

recreating didn't work. I'd created 3 machines in a row with exactly the same issue.

boot_kernel coreos.autologin worked, but no idea how to regenerate ssh host keys, or where they live.

@crawford

This comment has been minimized.

Member

crawford commented Aug 12, 2014

The keys are located in /etc/ssh/ and can be removed by running rm /etc/ssh/ssh_host_*. To regenerate them, reboot the machine or run systemctl restart sshd-keygen.service.

@hackzilla

This comment has been minimized.

hackzilla commented Aug 13, 2014

worked a treat. thanks for the explicit instructions. 👍

@crawford

This comment has been minimized.

Member

crawford commented Aug 13, 2014

Good! I'll have to root cause this one. Thanks for the report.

@jfoy

This comment has been minimized.

jfoy commented Mar 18, 2015

I just encountered this issue on CoreOS v618.0.0, VMware Fusion Pro v7.1.1, OS X v10.10.2. Thanks for the workaround: enabling autologin, then sudo rm /etc/ssh/ssh_host_* && sudo systemctl restart sshd-keygen.service resolved the problem for this host.

@sunadm

This comment has been minimized.

sunadm commented Apr 2, 2015

I just encountered this issue on CoreOS v607.0.0, VMware ESXi 5.5

First CoreOS instance have not this problem, when I created the other two vm on the same lan, the issue occured.

why?

@mischief

This comment has been minimized.

mischief commented Apr 2, 2015

@sunadm are the ssh keys ever printed on the console? is it simply delayed, or does it never work?

mischief added a commit to mischief/init that referenced this issue Apr 2, 2015

issuegen: add explicit dependency on sshd-keygen.service
if issuegen doesn't wait for sshd-keygen to complete, issuegen will
sometimes display empty ssh host keys.

fixes coreos/bugs#106
@lmcdasm

This comment has been minimized.

lmcdasm commented Feb 16, 2016

Hey there.

Hit the same issue today.
ISO installation from OVF of coreOS stable.
3 machines, all with the same cloud-config and .vmx setup (guestinfo.xyz)..

2 of the 3 machines came up with no ssh_host_xyz data populated in the files, 1 of the 3 hosts worked without issue.

I used the procedure of wiping out the files and regenerating them listed above

again
1 - sudo rm -rf /etc/ssh/ssh_host_*
2 - sudo systemctl restart sshd-keygen.service

doing an ls -l /etc/ssh/ shows that teh files are not populated with values (before they were all zero byte).

Thanks
Daniel

@mischief

This comment has been minimized.

mischief commented Feb 26, 2016

hm, does issuegen need Requires=sshd-keygen.service too?

@crawford

This comment has been minimized.

Member

crawford commented Feb 26, 2016

This has nothing to do with issuegen. The keys themselves aren't being generated properly.

@f0

This comment has been minimized.

f0 commented Jun 9, 2016

@crawford
hi, we are hitting this issue with 1010.5.0, reproduced 3 times with fresh installs is it possible to merge the workaround?

@mischief mischief removed their assignment Jun 20, 2016

@crawford crawford added this to the CoreOS 1096.0.0 milestone Jun 20, 2016

@crawford

This comment has been minimized.

Member

crawford commented Jun 20, 2016

@f0 we are going to try to figure out a root cause before the next Alpha. If we cannot figure out why it's failing, we'll pull in the workaround for now.

@crawford

This comment has been minimized.

Member

crawford commented Jun 22, 2016

I'm having trouble reproducing this locally in qemu. If you've seen this failure, can you provide the output of journalctl -u sshd-keygen --no-pager?

@mischief

This comment has been minimized.

mischief commented Jun 24, 2016

@f0 how can we reproduce this?

@f0

This comment has been minimized.

f0 commented Jun 26, 2016

@mischief
i installed a Bare Metal Server (DELL)

  • booted the coreos live iso
  • grab the coreos image
  • dd the image to disk
  • mount the partitions and copy the cloud config
  • adjust grub config to autologin
  • restart the system twice (to got the networking working)

@crawford ok i will do

@crawford crawford self-assigned this Jun 28, 2016

@mischief

This comment has been minimized.

mischief commented Jul 21, 2016

@f0 can you try CoreOS 1109.1.0?

@f0

This comment has been minimized.

f0 commented Jul 21, 2016

@mischief sure, maybe i can do this tomorrow

@f0

This comment has been minimized.

f0 commented Jul 22, 2016

@mischief testet 1109.1.0 the problem is gone

@f0

This comment has been minimized.

f0 commented Jul 22, 2016

@mischief hm wired, i also build a new system with the stable version 1068.8.0, works also without problems
And in found a difference between the systems with the problem, and them without...
The systems with the problem where installed with coreos before (the Disks where not "clean") , the systems without are clean systems without any installation before

@crawford crawford modified the milestones: CoreOS 1123.0.0, CoreOS 1137.0.0 Jul 26, 2016

@crawford

This comment has been minimized.

Member

crawford commented Aug 8, 2016

@f0 I don't think you are going to see any failures with the workaround, but you might see warnings in the logs. Can you check journalctl -u sshd-keygen.service --no-pager?

@f0

This comment has been minimized.

f0 commented Aug 10, 2016

@crawford installed stable from scratch, no warnings and the keys are correct

@crawford

This comment has been minimized.

Member

crawford commented Aug 23, 2016

@f0 have you noticed any failures? Can you try reproducing in the problematic environment for me?

@f0

This comment has been minimized.

f0 commented Aug 24, 2016

@crawford sure, i am on vacation atm and back in the office in 2 weeks

@crawford

This comment has been minimized.

Member

crawford commented Sep 12, 2016

@f0 have you had a chance to test this? I'd like to revert the workaround and understand what is actually failing.

@f0

This comment has been minimized.

f0 commented Sep 13, 2016

@crawford tested

cat /etc/os-release
NAME=CoreOS
ID=coreos
VERSION=1164.1.0
VERSION_ID=1164.1.0
BUILD_ID=2016-09-10-0834
PRETTY_NAME="CoreOS 1164.1.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues

problem is gone

journal

-- Logs begin at Tue 2016-09-13 08:52:29 CEST, end at Tue 2016-09-13 08:35:32 CEST. --
Sep 13 08:52:45 localhost systemd[1]: Starting Generate sshd host keys...
Sep 13 08:52:47 localhost sshd_keygen[1894]: ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519
Sep 13 08:52:47 localhost systemd[1]: Started Generate sshd host keys.
Sep 13 08:52:57 localhost systemd[1]: Stopped Generate sshd host keys.
-- Reboot --
Sep 13 08:54:40 localhost systemd[1]: Starting Generate sshd host keys...
Sep 13 08:54:42 localhost systemd[1]: Started Generate sshd host keys.
Sep 13 09:14:11 localhost systemd[1]: Stopped Generate sshd host keys.
-- Reboot --
Sep 13 09:16:09 localhost systemd[1]: Starting Generate sshd host keys...
Sep 13 09:16:11 localhost systemd[1]: Started Generate sshd host keys.

regards f0

@crawford

This comment has been minimized.

Member

crawford commented Sep 13, 2016

@f0 thank you for confirming. I'm going to revert that workaround.

@crawford

This comment has been minimized.

Member

crawford commented Sep 13, 2016

Actually, no. The workaround is still needed if the filesystem is uncleanly shutdown. I'm going to leave it in place.

@crawford crawford closed this Sep 13, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment