Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSH Key not work after some minutes #2553

Open
infnada opened this Issue Feb 18, 2019 · 6 comments

Comments

Projects
None yet
2 participants
@infnada
Copy link

infnada commented Feb 18, 2019

Issue Report

Bug

On node restart I'm able to login using SSH key but not after 2/3 minutes. I've got 4 SSH keys and non of them works with the error (putty):

login as: XXXX
Server refused our key

The key is in ~/.ssh/authorized_keys & ~/.ssh/authorized_keys.d/coreos-cloudinit

Container Linux Version

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1967.6.0
VERSION_ID=1967.6.0
BUILD_ID=2019-02-12-2138
PRETTY_NAME="Container Linux by CoreOS 1967.6.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

VMware ESXi, 6.5.0, 5310538
Default VM options.

Configured with:

coreos-install -d /dev/sda -c ci.yml -o vmware_raw

#cloud-config
coreos:
  update:
    reboot-strategy: "off"
  units:
    - name: systemd-timesyncd.service
      command: stop
      mask: true
    - name: ntpd.service
      command: start
      enable: true
    - name: systemd-networkd.service
      command: stop
    - name: down-interfaces.service
      command: start
      content: |
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/ip link set ens192 down
        ExecStart=/usr/bin/ip addr flush dev ens192
    ############################### CHANGE ###############################
    ############################### CHANGE ###############################
    ############################### CHANGE ###############################
    ############################### CHANGE ###############################
    - name: 00-ens192.network
      runtime: true
      content: |
        [Match]
        Name=ens192

        [Network]
        DNS= XXX
        Address= XXX
        Gateway=XXX
    - name: systemd-networkd.service
      command: restart
    - name: docker.service
      drop-ins:
        - name: 10-memlock.conf
          content: |
            [Service]
            LimitMEMLOCK=infinity
    - name: docker.service
      command: start
    - name: runsysctl.service
      command: start
      content: |
        [Unit]
        Description=Set Max Map Count

        [Service]
        Type=oneshot
        ExecStart=/usr/sbin/sysctl -w vm.max_map_count=16777216
    - name: runcmd.service
      command: start
      content: |
        [Unit]
        Description=Install vsphere plugin

        [Service]
        Type=oneshot
        ExecStart=/usr/bin/docker plugin ls | grep -q 'vsphere' && echo "matched" || /usr/bin/docker plugin install --grant-all-permissions --alias vsphere vmware/vsphere-storage-for-docker:latest "VDVS_SOCKET_GID=233"
        ExecStart=/usr/bin/docker plugin ls | grep -q 'vfile' && echo "matched" || /usr/bin/docker plugin install --grant-all-permissions --alias vfile vmware/vfile:latest VFILE_TIMEOUT_IN_SECOND=90 "VDVS_SOCKET_GID=233"
ssh_authorized_keys:
  - "XXXX"
############################### CHANGE ###############################
############################### CHANGE ###############################
############################### CHANGE ###############################
############################### CHANGE ###############################
hostname: "XXXXX"
users:
  - name: "core"
    passwd: "XXX"
    groups:
      - "sudo"
      - "docker"
  - name: "XXX"
    groups:
      - "sudo"
      - "docker"
    ssh-authorized-keys:
      - "XXXX"
write_files:
  - path: "/etc/resolv.conf"
    permissions: "0644"
    owner: "root"
    content: |
      nameserver XXXX
      nameserver XXXX
  - path: /etc/ntp.conf
    content: |
      server XXXXX

      # - Allow only time queries, at a limited rate.
      # - Allow all local queries (IPv4, IPv6)
      restrict default nomodify nopeer noquery limited kod
      restrict 127.0.0.1
      restrict [::1]
  - path: /etc/sysctl.conf
    content: |
      vm.max_map_count=262144
  - path: /etc/vfile.conf
    content: |
      {
        "MaxLogAgeDays": 28,
        "MaxLogFiles": 10,
        "MaxLogSizeMb": 10,
        "LogPath": "/var/log/vfile.log"
      }
  - path: /etc/vsphere-storage-for-docker.conf
    content: |
      {
        "MaxLogAgeDays": 28,
        "MaxLogFiles": 10,
        "MaxLogSizeMb": 10,
        "LogPath": "/var/log/vsphere-storage-for-docker.log"
      }
  - path: /etc/security/limits.conf
    content: |
      *            hard   memlock           unlimited
      *            soft   memlock           unlimited
  - path: /etc/sysctl.conf
    content: |
      vm.max_map_count=16777216

Expected Behavior

Be able to login with SSH using some of cloud-config provided keys

Actual Behavior

Can't login after 2/3 minutes of node restart.

Reproduction Steps

  1. Restart

  2. Instant login -> OK

  3. Wait 2/3 minutes -> Login fail

  4. Restart

  5. Wait 2/3 minutes -> Login fail

@bgilbert

This comment has been minimized.

Copy link
Member

bgilbert commented Feb 18, 2019

Is this for the core user or the other one? Is there a previous version of Container Linux where this was working properly?

@infnada

This comment has been minimized.

Copy link
Author

infnada commented Feb 19, 2019

This is for every user. Have 4 new users + core user and none of them can login. I'm new to CoreOs never tried it before.

@bgilbert

This comment has been minimized.

Copy link
Member

bgilbert commented Feb 19, 2019

If you're just starting with Container Linux, please use a Container Linux Config rather than a cloud-config. coreos-cloudinit is long deprecated and has unfixable race conditions. It seems likely that you're seeing one of them here.

@infnada

This comment has been minimized.

Copy link
Author

infnada commented Feb 21, 2019

Can i just delete /var/lib/coreos-install/user_data, copy the new ignition.json somewhere and reboot the VM?

@bgilbert

This comment has been minimized.

Copy link
Member

bgilbert commented Feb 21, 2019

At this point you should start with a new VM. Ignition only runs on the first boot, and anyway coreos-cloudinit has already made a bunch of changes to your system.

@infnada

This comment has been minimized.

Copy link
Author

infnada commented Feb 23, 2019

Exact same issue:

{
  "ignition": {
    "config": {},
    "timeouts": {},
    "version": "2.1.0"
  },
  "networkd": {
    "units": [
      {
        "contents": "[Match]\nName=ens192\n\n[Network]\nAddress=xxxxxx/24\nGateway=xxxxx\nDNS=xxxxx",
        "name": "static.network"
      }
    ]
  },
  "passwd": {
    "users": [
      {
        "groups": [
          "sudo",
          "docker"
        ],
        "name": "core",
        "passwordHash": "xxxxx",
        "sshAuthorizedKeys": [
          "ssh-rsa xxxxxxx"
        ]
      },
      {
        "groups": [
          "sudo",
          "docker"
        ],
        "name": "xxxx",
        "sshAuthorizedKeys": [
          "ssh-rsa xxxxx"
        ]
      },
      {
        "groups": [
          "sudo",
          "docker"
        ],
        "name": "xxx",
        "sshAuthorizedKeys": [
          "ssh-rsa xxxxx"
        ]
      },
      {
        "groups": [
          "sudo",
          "docker"
        ],
        "name": "xxxx",
        "sshAuthorizedKeys": [
          "ssh-rsa xxxx"
        ]
      },
      {
        "groups": [
          "sudo",
          "docker"
        ],
        "name": "xxxx",
        "sshAuthorizedKeys": [
          "ssh-rsa xxxxxx"
        ]
      }
    ]
  },
  "storage": {
    "files": [
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/coreos/update.conf",
        "user": {},
        "contents": {
          "source": "data:,GROUP%3Dstable%0AREBOOT_STRATEGY%3D%22off%22%0ASERVER%3Dhttps%3A%2F%2Fpublic.update.core-os.net%2Fv1%2Fupdate%2F%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/hostname",
        "user": {},
        "contents": {
          "source": "data:,xxxxx%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/resolv.conf",
        "user": {},
        "contents": {
          "source": "data:,nameserver%09xxxxx%0Anameserver%09xxxx%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/sysctl.conf",
        "user": {},
        "contents": {
          "source": "data:,vm.max_map_count=262144%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/vfile.conf",
        "user": {},
        "contents": {
          "source": "data:,%7B%22MaxLogAgeDays%22%3A28%2C%22MaxLogFiles%22%3A10%2C%22MaxLogSizeMb%22%3A10%2C%22LogPath%22%3A%22%2Fvar%2Flog%2Fvfile%2Elog%22%7D%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/vsphere-storage-for-docker.conf",
        "user": {},
        "contents": {
          "source": "data:,%7B%22MaxLogAgeDays%22%3A28%2C%22MaxLogFiles%22%3A10%2C%22MaxLogSizeMb%22%3A10%2C%22LogPath%22%3A%22%2Fvar%2Flog%2Fvsphere%2Dstorage%2Dfor%2Ddocker%2Elog%22%7D%0A",
          "verification": {}
        },
        "mode": 420
      },
      {
        "filesystem": "root",
        "group": {},
        "path": "/etc/security/limits.conf",
        "user": {},
        "contents": {
          "source": "data:,%2A%09hard%09memlock%09unlimited%0A%2A%09soft%09memlock%09unlimited%0A",
          "verification": {}
        },
        "mode": 420
      }
    ]
  },
  "systemd": {
    "units": [
      {
        "enable": true,
        "dropins": [
          {
            "contents": "[Service]\nLimitMEMLOCK=infinity",
            "name": "10-memlock.conf"
          }
        ],
        "name": "docker.service"
      },
      {
        "contents": "[Unit]\nDescription=Set Max Map Count\n\n[Service]\nType=oneshot\nExecStart=/usr/sbin/sysctl -w vm.max_map_count=16777216\n\n[Install]\nWantedBy=multi-user.target",
        "enable": true,
        "name": "runsysctl.service"
      },
      {
        "contents": "[Unit]\nDescription=Install vsphere plugin\nAfter=docker.service\nRequires=docker.service\n\n[Service]\nType=oneshot\nExecStart=/usr/bin/docker plugin ls | grep -q 'vsphere' && echo \"matched\" || /usr/bin/docker plugin install --alias vsphere vmware/vsphere-storage-for-docker:latest --grant-all-permissions \"VDVS_SOCKET_GID=233\"\nExecStart=/usr/bin/docker plugin ls | grep -q 'vfile' && echo \"matched\" || /usr/bin/docker plugin install --alias vfile vmware/vfile:latest VFILE_TIMEOUT_IN_SECOND=90 \"VDVS_SOCKET_GID=233\" --grant-all-permissions\n\n[Install]\nWantedBy=multi-user.target",
        "enable": true,
        "name": "runcmd.service"
      }
    ]
  }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.