Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to upgrade cluster on CentOS7 hosts to Pharos 2.4.* versions #1516

Closed
edita-timo opened this issue Dec 11, 2019 · 1 comment
Closed

Unable to upgrade cluster on CentOS7 hosts to Pharos 2.4.* versions #1516

edita-timo opened this issue Dec 11, 2019 · 1 comment

Comments

@edita-timo
Copy link

What happened:

Upgrading our pharos cluster gets stuck when using versions 2.4.*+oss.
Debug info seems to indicate some OpenSSL issue:

I, [2019-12-11T11:59:55.459891 #30421]  INFO -- K8s::Transport: Using config with server=https://localhost:65534
    [nmprimakube01] Populating client cache
    [nmprimakube01] Error: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3/TLS write client hello (OpenSSL::SSL::SSLError)
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/excon-0.66.0/lib/excon/ssl_socket.rb:125:in `connect_nonblock'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/excon-0.66.0/lib/excon/ssl_socket.rb:125:in `initialize'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/excon-0.66.0/lib/excon/connection.rb:455:in `new'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/excon-0.66.0/lib/excon/connection.rb:455:in `socket'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/excon-0.66.0/lib/excon/connection.rb:116:in `request_call'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/excon-0.66.0/lib/excon/middlewares/mock.rb:57:in `request_call'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/excon-0.66.0/lib/excon/middlewares/instrumentor.rb:34:in `request_call'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/excon-0.66.0/lib/excon/middlewares/idempotent.rb:19:in `request_call'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/excon-0.66.0/lib/excon/middlewares/base.rb:22:in `request_call'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/excon-0.66.0/lib/excon/middlewares/base.rb:22:in `request_call'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/excon-0.66.0/lib/excon/middlewares/redirect_follower.rb:15:in `request_call'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/excon-0.66.0/lib/excon/connection.rb:270:in `request'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.10.4/lib/k8s/transport.rb:284:in `request'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.10.4/lib/k8s/transport.rb:363:in `get'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.10.4/lib/k8s/client.rb:134:in `block in api_groups!'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.10.4/lib/k8s/client.rb:133:in `api_groups!'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.10.4/lib/k8s/client.rb:148:in `api_groups'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.10.4/lib/k8s/client.rb:156:in `apis'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/phases/configure_client.rb:44:in `client_prefetch'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/phases/configure_client.rb:20:in `call'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/phase_manager.rb:98:in `block in apply'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/retry.rb:20:in `perform'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/phase_manager.rb:71:in `block in run_serial'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/phase_manager.rb:70:in `map'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/phase_manager.rb:70:in `run_serial'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/phase_manager.rb:82:in `run'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/phase_manager.rb:95:in `apply'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/cluster_manager.rb:157:in `apply_phase'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/cluster_manager.rb:61:in `gather_facts'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/cluster_manager.rb:69:in `validate'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/command_options/load_config.rb:27:in `block in cluster_manager'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/command_options/load_config.rb:23:in `tap'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/command_options/load_config.rb:23:in `cluster_manager'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/up_command.rb:38:in `configure'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/up_command.rb:29:in `block in execute'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/up_command.rb:28:in `chdir'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/up_command.rb:28:in `execute'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/clamp-1.2.1/lib/clamp/command.rb:63:in `run'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/command.rb:25:in `run'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/clamp-1.2.1/lib/clamp/subcommand/execution.rb:11:in `execute'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/clamp-1.2.1/lib/clamp/command.rb:63:in `run'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/lib/pharos/command.rb:25:in `run'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/clamp-1.2.1/lib/clamp/command.rb:132:in `run'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.4.10/bin/pharos:12:in `<top (required)>'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/bin/pharos:23:in `load'
    [nmprimakube01]     /__enclose_io_memfs__/lib/ruby/gems/2.5.0/bin/pharos:23:in `<main>'

Tested with Debian 10 client and with Mac OSX and both gave same error so I'm guessing the issue on server side.

What you expected to happen:
Expected successful cluster upgrade.

How to reproduce it (as minimally and precisely as possible):
Install Centos7 hosts, install kube cluster with pharos to 2.3.9+oss. Try to upgrade from there to newer release.

Anything else we need to know?:

Environment:

- Pharos version (use `pharos --version`): 
Kontena Pharos:
  - pharos version 2.4.10+oss
Common:
  - calico-cni 3.6.2 (Apache License 2.0)
  - calico-kube-controllers 3.6.2 (Apache License 2.0)
  - calico-node 3.6.2 (Apache License 2.0)
  - coredns 1.3.1 (Apache License 2.0)
  - dns-node-cache 1.15.1 (Apache License 2.0)
  - etcd 3.3.10 (Apache License 2.0)
  - kubelet-rubber-stamp 0.1.0 (Apache License 2.0)
  - kubernetes 1.14.9 (Apache License 2.0)
  - kubernetes-cni 0.7.5 (Apache License 2.0)
  - metrics-server 0.3.2 (Apache License 2.0)
  - pharos-kubelet-proxy 0.3.7 (Apache License 2.0)
  - weave-flying-shuttle 0.3.1 (Apache License 2.0)
  - weave-net 2.5.2 (Apache License 2.0)
Debian 9:
  - cfssl 1.2 (MIT)
  - cri-o 1.14.11 (Apache License 2.0)
  - docker-ce 18.06.2 (Apache License 2.0)
Centos 7:
  - cfssl 1.2 (MIT)
  - cri-o 1.14.11 (Apache License 2.0)
  - docker 1.13.1 (Apache License 2.0)
Rhel 7.4:
  - cfssl 1.2 (MIT)
  - cri-o 1.14.11 (Apache License 2.0)
  - docker 1.13.1 (Apache License 2.0)
Rhel 7.5:
  - cfssl 1.2 (MIT)
  - cri-o 1.14.11 (Apache License 2.0)
  - docker 1.13.1 (Apache License 2.0)
Rhel 7.6:
  - cfssl 1.2 (MIT)
  - cri-o 1.14.11 (Apache License 2.0)
  - docker 1.13.1 (Apache License 2.0)
Rhel 7.7:
  - cfssl 1.2 (MIT)
  - cri-o 1.14.11 (Apache License 2.0)
  - docker 1.13.1 (Apache License 2.0)
Ubuntu 18.04:
  - cfssl 1.2 (MIT)
  - cri-o 1.14.11 (Apache License 2.0)
  - docker 18.09 (Apache License 2.0)
Ubuntu 16.04:
  - cfssl 1.2 (MIT)
  - cri-o 1.14.11 (Apache License 2.0)
  - docker 18.09 (Apache License 2.0)
Add-ons:
  - cert-manager 0.8.1 (Apache License 2.0)
  - helm 2.13.1 (Apache License 2.0)
  - host-upgrades 0.3.1 (Apache License 2.0)
  - ingress-nginx 0.25.1 (Apache License 2.0)

  • Cloud provider or hardware configuration:
    VMWare virtual hosts

  • OS (e.g. from /etc/os-release; both for the client and for the used nodes):
    client:

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Hosts:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="Red Hat"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Others:

cluster.yml:

hosts:
  - address: "nmprimakube01.example.com"
    user: foo
    role: master
    container_runtime: cri-o
    private_address: xxx.xxx.xxx.xxx

  - address: "nmprimakube02.example.com"
    user: foo
    role: worker
    private_address: xxx.xxx.xxx.xxx
    container_runtime: cri-o

  - address: "nmprimakube03.example.com"
    user: foo
    role: worker
    private_address: xxx.xxx.xxx.xxx
    container_runtime: cri-o

network:
  provider: weave
  dns_replicas: 2
  service_cidr: 172.100.0.0/16
  pod_network_cidr: 172.200.0.0/16
  weave:
    trusted_subnets:
      - 10.xxx.xxx.0/25
      - 10.xxx.xxx.128/25

addons:
  cert-manager:
    enabled: true
    issuer:
      name: letsencrypt-staging
      server: https://acme-staging-v02.api.letsencrypt.org/directory
      email: admin@example.com
@edita-timo
Copy link
Author

Got this figured out, found following errors on kube master journald:
"refused local port forward: originator 127.0.0.1 port 65531, target localhost port 6443"

After enabling tcp port forwarding on ssh server the pharos got past this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@edita-timo