"rke up" got "ssh: rejected: administratively prohibited (open failed)"

[nevermosby]

rke version: rke version v0.0.7-dev

problem:
when to run "./rke -d up" with the file "cluster.yml" in the same folder, I got:

INFO[0000] [certificates] Generating kubernetes certificates
INFO[0000] [certificates] Generating CA kubernetes certificates
...
INFO[0003][certificates] Deploying kubernetes certificates to Cluster nodes
DEBU[0003] [certificates] Pulling Certificate downloader Image on host [node1]
FATA[0008] Can't pull Docker image rancher/rke-cert-deployer:0.1.0 for host [node1]: error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/create?fromImage=rancher%2Frke-cert-deployer&tag=0.1.0: Error connecting to Docker socket on host [node1]: ssh: rejected: administratively prohibited (open failed)

Tried to fix it by enabling the ssh config,AllowTcpForwarding yes, but failed.

Any suggestions?

[superseb]

Is this on CentOS/RHEL by any chance?

[galal-hussein]

Can you also make sure that the ssh user you are using has access to docker engine

[iyacontrol]

i also meet the same problem。i even try root user to ssh ，but the rke still failed。

[iyacontrol]

@nevermosby have you solved the problem?

[superseb]

@iyacontrol Please share used OS (cat /etc/os-release)

[iyacontrol]

@superseb
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[iyacontrol]

@superseb Linux 3.10.0-693.el7.x86_64

[nevermosby]

@superseb yeah, it is a centos system, details are listed beblow:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

@galal-hussein I used "root" to run the command so it can be sure I have the access to docker engine.

@iyacontrol I still have this issue with centos.

New progress here:
I tried ubuntu system instead and it did work.
So some different OS configuration can be the root cause.

[galal-hussein]

@nevermosby I will try to reproduce with CentOS 7 to see if the problem is related to specific OS

[galal-hussein]

@nevermosby @iyacontrol @superseb I was able to reproduce the issue, however i am not sure what is the root cause, CentOS seems to reject ssh local forwarding using the root:

- Unit session-33.scope has begun starting up.
Dec 04 23:32:46 hussein-centos-01 sshd[11192]: refused streamlocal port forward: originator  port 0, target /var/run/docker.sock
Dec 04 23:32:46 hussein-centos-01 sshd[11192]: pam_unix(sshd:session): session closed for user root

➜  ssh -nNT -L /tmp/docker.sock:/var/run/docker.sock  root@x.x.x.x
channel 1: open failed: administratively prohibited: open failed
channel 1: open failed: administratively prohibited: open failed

however it works for another users other than root, for a work around you can specify other users than root for now

[nevermosby]

@galal-hussein thx for feedback. On the same centos machine, I took ur advice and tried another user other than root but got same error.

[galal-hussein]

@nevermosby can you make sure that the ssh user has access to docker engine, you can add the user to docker group and use it in cluster.yml

[nevermosby]

@galal-hussein yes I am sure I have added the user to docker group and tried "docker push" command, it works.fine.

[galal-hussein]

@nevermosby Can you post the version of openssh currently installed on the centos server

[nevermosby]

@galal-hussein sorry for late reply. The version installed on my centos machine is :

ssh -V

OpenSSH_6.6.1p1,OpenSSL 1.0.1e-fips 11 Feb 2013

[niusmallnan]

@galal-hussein
It seems that pam_unix causes this issue.
Try this: https://serverfault.com/questions/416441/pam-module-causes-flurry-of-ssh-sessions

[galal-hussein]

@niusmallnan No i don't think this is the issue, i just tested with disabling PAM and same error, also i think not using PAM is not recommended in RHEL

[moelsayed]

@nevermosby You need to upgrade your openssh server to 7.x to get support for local stream forwarding.

[nevermosby]

Hi @moelsayed ，thx for remind and I just upgraded my openssh version to 7.4 and enabled "AllowTcpForwarding" for yes:

ssh -V
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017

However, I still got the same error. Maybe the forwarding configuration is not right. Could u give some advise for this?

[ghost]

if you run rke in centos7, you should not use the root user to open the ssh tunnel.you can try the following step to run rke
in all nodes:

1. update openssh to 7.4,and docker version v1.12.6
2. set "AllowTcpForwarding yes" "PermitTunnel yes" to /etc/ssh/sshd_config,and then restart sshd service
3. the host which run rke can ssh to all nodes without password
4. run: "groupadd docker" to create docker group,while docker group is not exist.
5. run: "useradd -g docker yourusername" to create yourusername user and set it's group to docker
6. set the docker.service's MountFlags=shared (vi /xxx/xxx/docker.service)
7. run:"su yourusername" to change current user,and then restart the docker service. so in the user yourusername session the docker.sock will be created in the path /var/run/docker.sock
8. in cluster.yml set the ssh user to yourusername(in setup hosts)

  nodes:
  - address: x.x.x.x
     ...
    user:  yourusername
  - address: x.x.x.x
     ...
    user:  yourusername

9. in cluster.yml set the kubelet to use the systemd cgroup(in setup hosts)

  services:
    kubelet:
      image: rancher/k8s:v1.8.3-rancher2
      extra_args: {"cgroup-driver":"systemd","fail-swap-on":"false"}

now you can run "rke -d up" to setup your k8s cluster.
if you meet "Failed to Save Kubernetes certificates: Timeout waiting for K8s to be ready" when running rke,your can see something here #121

[subhransusekhar]

Hi @luoshiqian thanks your solutions worked for me on centos 7.4

[galal-hussein]

@luoshiqian Thanks for the steps, OpenSSH doesn't allow ssh tunneling for root users for some reason, closing this issue and will track the root user bug in this issue #136

[ibre5041]

Tried to fix it by enabling the ssh config,AllowTcpForwarding yes, but failed.

Any suggestions?

I had same problem, found you that problem was on mine side. In authorizedkeys in key usage restrictions.