Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Login to CoreOS doesn't work #2617

Open
crackhd opened this issue Oct 1, 2019 · 14 comments

Comments

@crackhd
Copy link

commented Oct 1, 2019

Issue Report

I use Ignition config file that worked flawlessly previously.

With fresh install of coreos, i cannot login to my system after install anymore - both with password and SSH key.

Both using port 22 (publickey,password) connection (LAN/DHCP), and tty0 on physical server (password).

Here is ignition config:

{
  "ignition":{
    "version":"2.2.0"
  },
  "storage":{
    "disks": [
      {
        "device": "/dev/sdb",
        "partitions": [
          {
            "label": "docker_storage",
            "typeGuid": "85D5E45A-237C-11E1-B4B3-E89A8F7FC3A7",
            "start": 0
          }
        ],
        "wipeTable": true
      }
    ],
    "filesystems": [
    ],
    "files":[
      {
        "filesystem":"root",
        "path":"/etc/hostname",
        "mode":420,
        "contents":{
          "source":"data:,owl1"
        }
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/sysctl.d/nf-01.conf",
        "user": {},
        "contents": {
          "source": "data:,net.netfilter.nf_conntrack_max%3D131072",
          "verification": {}
        },
        "mode": 420
      }
    ]
  },
  "passwd":{
    "users":[
      {
        "name":"core",
        "passwordHash": "$2a$10$pmElHc4pbVt5PLCA84HUxO42PkDftgxBHKlIi612cHNCFY8qu0f/O",
        "sshAuthorizedKeys":[
          "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDLrklSzVSJYCRqS/SosIiTIASmPBaIlBXi42WtfZIseKxvGSX+q3om6hWPDofOQW9c6L8t+a3NJORUPzNi6PI8eyKaFEU+r6L6eriXjrlSGr9hlDWaVzCO9dMLoddLkFLIdI1yIRSS6gfuR7P5hAiYenKoypx11SmOzxZiPu1eL3Qk+tewCHJGE5JO5SuC9/Mycli3JA6lw9RnHJ842CWWezQkqMW2Pn4DPYCV7fM9TfXxeH76UKznu13pDC3p6QgZq/U3jltRty437t1Cl6Ie5MjecIEZzdRGWbkRYighyzNBPlIpvsekzKnTdSRd7JJy4QFr8y8zsPoaTteVQxbzMkAu7dkSFnEzI+woUDIA14T5Wai9mxUZRWkvKUtq0WiLgu1yZXPUdf+STBG7ZVTVPgZfS9WFk9tNgbxv1nEJS8WhhwEhLopncOwtFXcNNZA5w2CTMpcrJ9RT7A/fAcKPDCfU1MDYb6EkJmcfynWcPWsop8qkIO1QOx1Nf5D2XgQ9V4VfPaSWcTDyjVBHDOGVTM/eWNxVeuNqADZBhnBoyUlyZh4GzXwWe6sdaxvH79K22wlk8cF4fgMK+pUz/4dQ7N4Kg3y+x/gDNW+yibg0EJ4XArSyMyQoJfZP54r+jiZjaFhS/lGAYRbzYOkGwgmE3+vzHfX5LfJohxt+XpGpRQ=="
        ],
        "groups": [
          "sudo",
          "docker"
        ],
        "uid": 1000,
        "shell": "/bin/bash"
      }
    ]
  },
  "networkd":{
    "units":[
      {
        "name":"00-eth.network",
        "contents":"[Match]\nName=eth*\n\n[Network]\nBond=bond0"
      },
      {
        "name":"01-enp.network",
        "contents":"[Match]\nName=enp*\n\n[Network]\nBond=bond0"
      },
      {
        "name":"10-bond0.netdev",
        "contents":"[NetDev]\nName=bond0\nKind=bond"
      },
      {
        "name":"20-bond0.network",
        "contents":"[Match]\nName=bond0\n\n[Network]\nDHCP=true"
      }
    ]
  },
  "systemd":{
    "units":[
      {
        "dropins": [
          {
            "contents": "[Unit]\nAfter=var-lib-docker.mount\nRequires=var-lib-docker.mount",
            "name": "10-wait-docker.conf"
          }
        ],
        "name": "docker.service"
      },      {
        "name":"docker.service",
        "enabled":true,
        "dropins":[
          {
            "name":"20-aditional_options.conf",
            "contents":"[Service]\nEnvironment=DOCKER_OPTS='--storage-driver=overlay2 --log-driver=journald'"
          }
        ]
      },
      {
        "contents": "[Unit]\nDescription=Docker Socket for the API\n\n[Socket]\nListenStream=2375\nBindIPv6Only=both\nService=docker.service\n\n[Install]\nWantedBy=sockets.target",
        "enable": true,
        "name": "docker-tcp.socket"
      }
    ]
  }
}

Bug

I cannot see auth_log on coreos because i cannot login to any shell (it says login incorrect). Here is ssh -vvv output from my Mac:

$ ssh -vvvi /Users/username/.ssh/owl1 core@192.168.2.2
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/username/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: /etc/ssh/ssh_config line 52: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.2.2 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.2.2 [192.168.2.2] port 22.
debug1: Connection established.
debug1: identity file /Users/username/.ssh/owl1 type -1
debug1: identity file /Users/username/.ssh/owl1-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9
debug1: match: OpenSSH_7.9 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 192.168.2.2:22 as 'core'
debug3: hostkeys_foreach: reading file "/Users/username/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/username/.ssh/known_hosts:21
debug3: load_hostkeys: loaded 1 keys from 192.168.2.2
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:egFVhb19YamWGao51Ylj1rbWBKkbNB+tJMW/3HOkX3I
debug3: hostkeys_foreach: reading file "/Users/username/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/username/.ssh/known_hosts:21
debug3: load_hostkeys: loaded 1 keys from 192.168.2.2
debug1: Host '192.168.2.2' is known and matches the ECDSA host key.
debug1: Found key in /Users/username/.ssh/known_hosts:21
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /Users/username/.ssh/owl1  explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: start over, passed a different list publickey,password,keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/username/.ssh/owl1
debug3: sign_and_send_pubkey: RSA SHA256:i1qbC407EguqJvPYj6zmkNw3kpKCtH+j8l9+JvsR4os
debug3: sign_and_send_pubkey: signing using rsa-sha2-512
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password: // ignores valid password attempts

Container Linux Version

Production version 2191.5.0 (current)

Environment

Bare metal. I use small Asus Eee PC from 2014, that works flawlessly with Arch linux, Ubuntu.

Expected Behavior

I should be able to login to my server via password or via SSH immediatelly after boot.

Actual Behavior

System boots fine and hooks up dynamic IP address. I can ping it from outside, and port 22 is working. But i cannot login with publickey or password via port 22, and i cannot login by typing login and password on keyboard physicially. It hangs for few seconds and says "login is incorrect".

Reproduction Steps

  1. Create bootable USB stick with coreos-iso
  2. Install CoreOS on "bare metal"
  3. Try to login using publickey or password specified in Ignition config

Other Information

My coreOS install instructions are:

$ sudo dhcpcd
$ wget http://192.168.2.1/ign.json
$ sudo coreos-install -d /dev/mmcblk1 -i ign.json
$ sudo reboot

Removing uid and shell from passwd.users has no effect.

SSH key is generated with ssh-keygen -t rsa -b 4096

Password hash is generated using coreos/bcrypt-tool

@crackhd crackhd changed the title Login into CoreOS doesn't work Login to CoreOS doesn't work Oct 1, 2019
@ajeddeloh

This comment has been minimized.

Copy link

commented Oct 1, 2019

Does it only affect new installs or do machines that update also exhibit this behavior?

@crackhd

This comment has been minimized.

Copy link
Author

commented Oct 1, 2019

I do not have any other cloud CoreOS or another hardware at the moment unfortunately nor do I have ability to try in VirtualBox.

I do not know how to boot into chroot mode (say with CoreOS ISO), to see sshd log for possible problem(s). If anyone can provide any instructions, this will be very useful. Don't know what to mount for /var/logs.

@ajeddeloh

This comment has been minimized.

Copy link

commented Oct 1, 2019

You can set the coreos.autologin kernel command line option to get a shell on a serial console.

@crackhd

This comment has been minimized.

Copy link
Author

commented Oct 1, 2019

I should add this to the right of the command line accessible with GRUB edit GUI?

@ajeddeloh

This comment has been minimized.

Copy link

commented Oct 1, 2019

Yup

@crackhd

This comment has been minimized.

Copy link
Author

commented Oct 1, 2019

Oh, thanks. I will get back to this in a few days and i am sure i will find out what's up

@crackhd

This comment has been minimized.

Copy link
Author

commented Oct 3, 2019

I use sudo /usr/sbin/sshd -d -p 999 to see sshd log while connecting.

There is a message: cannot open file /home/core/.ssh/authorized_keys2 causing publickey method to fail. Folder /home/core/.ssh is empty.

Opening /etc/shadow, i see that core has no password causing keyboard-interactive method to fail.

This means, that ignition ignores passwd.users section in config file and i cannot login to my CoreOS box. But i do not see any errors or warnings during coreos-install.

Same for current alpha ISO
My workaround is to autologin to make sudo passwd core and manually install key with ssh-copy-id.

@ajeddeloh

This comment has been minimized.

Copy link

commented Oct 3, 2019

Can you run sudo journalctl -t ignition and post the results please?

@crackhd

This comment has been minimized.

Copy link
Author

commented Oct 3, 2019

Oct 03 18:45:10 localhost ignition[353]: Ignition v0.33.0-dirty
Oct 03 18:45:10 localhost ignition[353]: reading system config file "/usr/lib/ignition/base.ign"
Oct 03 18:45:10 localhost ignition[353]: no config at "/usr/lib/ignition/base.ign"
Oct 03 18:45:10 localhost ignition[353]: parsed url from cmdline: ""
Oct 03 18:45:10 localhost ignition[353]: no config URL provided
Oct 03 18:45:10 localhost ignition[353]: reading system config file "/usr/lib/ignition/user.ign"
Oct 03 18:45:10 localhost ignition[353]: no config at "/usr/lib/ignition/user.ign"
Oct 03 18:45:10 localhost ignition[353]: noop provider fetching empty config
Oct 03 18:45:10 localhost ignition[353]: failed to fetch config: not a config (empty)
Oct 03 18:45:10 localhost ignition[353]: not a config (empty): ignoring user-provided config
Oct 03 18:45:10 localhost ignition[353]: reading system config file "/usr/lib/ignition/default.ign"
Oct 03 18:45:10 localhost ignition[353]: no config at "/usr/lib/ignition/default.ign"
Oct 03 18:45:10 localhost ignition[353]: disks: disks passed
Oct 03 18:45:10 localhost ignition[353]: Ignition finished successfully
Oct 03 18:45:11 localhost ignition[395]: Ignition v0.33.0-dirty
Oct 03 18:45:11 localhost ignition[395]: reading system config file "/usr/lib/ignition/base.ign"
Oct 03 18:45:11 localhost ignition[395]: no config at "/usr/lib/ignition/base.ign"
Oct 03 18:45:11 localhost ignition[395]: parsed url from cmdline: ""
Oct 03 18:45:11 localhost ignition[395]: no config URL provided
Oct 03 18:45:11 localhost ignition[395]: reading system config file "/usr/lib/ignition/user.ign"
Oct 03 18:45:11 localhost ignition[395]: no config at "/usr/lib/ignition/user.ign"
Oct 03 18:45:11 localhost ignition[395]: noop provider fetching empty config
Oct 03 18:45:11 localhost ignition[395]: failed to fetch config: not a config (empty)
Oct 03 18:45:11 localhost ignition[395]: not a config (empty): ignoring user-provided config
Oct 03 18:45:11 localhost ignition[395]: reading system config file "/usr/lib/ignition/default.ign"
Oct 03 18:45:11 localhost ignition[395]: no config at "/usr/lib/ignition/default.ign"
Oct 03 18:45:11 localhost ignition[395]: files: compiled without relabeling support, skipping
Oct 03 18:45:11 localhost ignition[395]: files: files passed
Oct 03 18:45:11 localhost ignition[395]: Ignition finished successfully
```
@ajeddeloh

This comment has been minimized.

Copy link

commented Oct 3, 2019

Ignition isn't finding it's config at all. After install, but before booting can you confirm the config is actually in the OEM partition?

@crackhd

This comment has been minimized.

Copy link
Author

commented Oct 3, 2019

"OEM partition", where is it? If we pass argument -i filename.txt, verifying that cat filename.txt will work before doing install, isn't that enough?

@ajeddeloh

This comment has been minimized.

Copy link

commented Oct 3, 2019

After installing, the disk you installed to will have a partition with label OEM. Mount that somewhere and there should be config.ign there.

@crackhd

This comment has been minimized.

Copy link
Author

commented Oct 3, 2019

Yes, /dev/mmcblk1p6 has lost+found and config.ign, with contents of my config

@ajeddeloh

This comment has been minimized.

Copy link

commented Oct 3, 2019

Hrm. Can you add the kernel command line argument rd.break (which should drop you into a shell in the initramfs) and see if there is the config at /usr/lib/ignition/user.ign in the initramfs? Also try mounting the OEM partition there and see if it's still there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.