Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"rke up" got "ssh: rejected: administratively prohibited (open failed)" #93

Closed
nevermosby opened this Issue Dec 3, 2017 · 22 comments

Comments

Projects
None yet
7 participants
@nevermosby
Copy link

nevermosby commented Dec 3, 2017

rke version: rke version v0.0.7-dev

problem:
when to run "./rke -d up" with the file "cluster.yml" in the same folder, I got:

INFO[0000] [certificates] Generating kubernetes certificates
INFO[0000] [certificates] Generating CA kubernetes certificates
...
INFO[0003][certificates] Deploying kubernetes certificates to Cluster nodes
DEBU[0003] [certificates] Pulling Certificate downloader Image on host [node1]
FATA[0008] Can't pull Docker image rancher/rke-cert-deployer:0.1.0 for host [node1]: error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/create?fromImage=rancher%2Frke-cert-deployer&tag=0.1.0: Error connecting to Docker socket on host [node1]: ssh: rejected: administratively prohibited (open failed)

Tried to fix it by enabling the ssh config,AllowTcpForwarding yes, but failed.

Any suggestions?

@superseb

This comment has been minimized.

Copy link
Member

superseb commented Dec 3, 2017

Is this on CentOS/RHEL by any chance?

@galal-hussein

This comment has been minimized.

Copy link
Collaborator

galal-hussein commented Dec 3, 2017

Can you also make sure that the ssh user you are using has access to docker engine

@iyacontrol

This comment has been minimized.

Copy link

iyacontrol commented Dec 4, 2017

i also meet the same problem。i even try root user to ssh ,but the rke still failed。

@iyacontrol

This comment has been minimized.

Copy link

iyacontrol commented Dec 4, 2017

@nevermosby have you solved the problem?

@superseb

This comment has been minimized.

Copy link
Member

superseb commented Dec 4, 2017

@iyacontrol Please share used OS (cat /etc/os-release)

@iyacontrol

This comment has been minimized.

Copy link

iyacontrol commented Dec 4, 2017

@superseb
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

@iyacontrol

This comment has been minimized.

Copy link

iyacontrol commented Dec 4, 2017

@superseb Linux 3.10.0-693.el7.x86_64

@nevermosby

This comment has been minimized.

Copy link
Author

nevermosby commented Dec 4, 2017

@superseb yeah, it is a centos system, details are listed beblow:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

@galal-hussein I used "root" to run the command so it can be sure I have the access to docker engine.

@iyacontrol I still have this issue with centos.

New progress here:
I tried ubuntu system instead and it did work.
So some different OS configuration can be the root cause.

@galal-hussein

This comment has been minimized.

Copy link
Collaborator

galal-hussein commented Dec 4, 2017

@nevermosby I will try to reproduce with CentOS 7 to see if the problem is related to specific OS

@galal-hussein

This comment has been minimized.

Copy link
Collaborator

galal-hussein commented Dec 5, 2017

@nevermosby @iyacontrol @superseb I was able to reproduce the issue, however i am not sure what is the root cause, CentOS seems to reject ssh local forwarding using the root:

- Unit session-33.scope has begun starting up.
Dec 04 23:32:46 hussein-centos-01 sshd[11192]: refused streamlocal port forward: originator  port 0, target /var/run/docker.sock
Dec 04 23:32:46 hussein-centos-01 sshd[11192]: pam_unix(sshd:session): session closed for user root
➜  ssh -nNT -L /tmp/docker.sock:/var/run/docker.sock  root@x.x.x.x
channel 1: open failed: administratively prohibited: open failed
channel 1: open failed: administratively prohibited: open failed

however it works for another users other than root, for a work around you can specify other users than root for now

@nevermosby

This comment has been minimized.

Copy link
Author

nevermosby commented Dec 5, 2017

@galal-hussein thx for feedback. On the same centos machine, I took ur advice and tried another user other than root but got same error.

@galal-hussein

This comment has been minimized.

Copy link
Collaborator

galal-hussein commented Dec 5, 2017

@nevermosby can you make sure that the ssh user has access to docker engine, you can add the user to docker group and use it in cluster.yml

@nevermosby

This comment has been minimized.

Copy link
Author

nevermosby commented Dec 5, 2017

@galal-hussein yes I am sure I have added the user to docker group and tried "docker push" command, it works.fine.

@galal-hussein

This comment has been minimized.

Copy link
Collaborator

galal-hussein commented Dec 5, 2017

@nevermosby Can you post the version of openssh currently installed on the centos server

@nevermosby

This comment has been minimized.

Copy link
Author

nevermosby commented Dec 8, 2017

@galal-hussein sorry for late reply. The version installed on my centos machine is :

ssh -V

OpenSSH_6.6.1p1,OpenSSL 1.0.1e-fips 11 Feb 2013

@niusmallnan

This comment has been minimized.

Copy link
Member

niusmallnan commented Dec 8, 2017

@galal-hussein

This comment has been minimized.

Copy link
Collaborator

galal-hussein commented Dec 9, 2017

@niusmallnan No i don't think this is the issue, i just tested with disabling PAM and same error, also i think not using PAM is not recommended in RHEL

@moelsayed

This comment has been minimized.

Copy link
Contributor

moelsayed commented Dec 9, 2017

@nevermosby You need to upgrade your openssh server to 7.x to get support for local stream forwarding.

@nevermosby

This comment has been minimized.

Copy link
Author

nevermosby commented Dec 11, 2017

Hi @moelsayed ,thx for remind and I just upgraded my openssh version to 7.4 and enabled "AllowTcpForwarding" for yes:

ssh -V
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017

However, I still got the same error. Maybe the forwarding configuration is not right. Could u give some advise for this?

@ghost

This comment has been minimized.

Copy link

ghost commented Dec 11, 2017

if you run rke in centos7, you should not use the root user to open the ssh tunnel.you can try the following step to run rke
in all nodes:

  1. update openssh to 7.4,and docker version v1.12.6
  2. set "AllowTcpForwarding yes" "PermitTunnel yes" to /etc/ssh/sshd_config,and then restart sshd service
  3. the host which run rke can ssh to all nodes without password
  4. run: "groupadd docker" to create docker group,while docker group is not exist.
  5. run: "useradd -g docker yourusername" to create yourusername user and set it's group to docker
  6. set the docker.service's MountFlags=shared (vi /xxx/xxx/docker.service)
  7. run:"su yourusername" to change current user,and then restart the docker service. so in the user yourusername session the docker.sock will be created in the path /var/run/docker.sock
  8. in cluster.yml set the ssh user to yourusername(in setup hosts)
  nodes:
  - address: x.x.x.x
     ...
    user:  yourusername
  - address: x.x.x.x
     ...
    user:  yourusername
  1. in cluster.yml set the kubelet to use the systemd cgroup(in setup hosts)
  services:
    kubelet:
      image: rancher/k8s:v1.8.3-rancher2
      extra_args: {"cgroup-driver":"systemd","fail-swap-on":"false"}

now you can run "rke -d up" to setup your k8s cluster.
if you meet "Failed to Save Kubernetes certificates: Timeout waiting for K8s to be ready" when running rke,your can see something here #121

@subhransusekhar

This comment has been minimized.

Copy link

subhransusekhar commented Dec 12, 2017

Hi @luoshiqian thanks your solutions worked for me on centos 7.4

@galal-hussein

This comment has been minimized.

Copy link
Collaborator

galal-hussein commented Dec 12, 2017

@luoshiqian Thanks for the steps, OpenSSH doesn't allow ssh tunneling for root users for some reason, closing this issue and will track the root user bug in this issue #136

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.