Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

call to 'certificates_to_synchronize' causes openshift_master_certificates to abend with 'ERROR! |failed expects hostvars is a dict' #7596

Closed
AllanCano opened this issue Mar 21, 2018 · 30 comments

Comments

@AllanCano
Copy link

AllanCano commented Mar 21, 2018

Description

Provide a brief description of your issue here. For example:

On a multi master install deploy_cluster.yml terminates after task
"openshift_master_certificates : Generate the loopback master client config"
with the error message
"ERROR! |failed expects hostvars is a dict"

Version
  • oc verions
# oc version
oc v3.7.1+c2ce2c0-1
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO
  • Ansible verions
# ansible --version
ansible 2.6.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
  • Ansible Openshift Playbooks
# cd /etc/ansible/
# mkdir playbooks
# cd playbooks
# git clone https://github.com/openshift/openshift-ansible
  • CentOS version
# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
  • Openshift version
# openshift version
openshift v3.7.1+c2ce2c0-1
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8
Inventory Hosts Files
#---- Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=root
debug_level='2'
containerized='false'
openshift_deployment_type=origin
os_firewall_use_firewalld='true'

#-- Configure networking
#openshift_master_default_subdomain=fossil.com
openshift_use_openshift_sdn=true
#openshift_use_flannel=true
#flannel_interface=bond0

# Uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider.
#openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

# Native high availbility cluster method with optional load balancer.
# If no lb group is defined installer assumes that a load balancer has
# been preconfigured. For installation the value of
# openshift_master_cluster_hostname must resolve to the load balancer
# or to one or all of the masters defined in the inventory if no load
# balancer is present.
#-- When we get to production use an external LB
openshift_master_cluster_method=native
#openshift_master_cluster_hostname=uusrcdkub00.fossil.com
#openshift_master_cluster_public_hostname=uusrcdkub00.fossil.com

# apply updated node defaults
openshift_node_kubelet_args={'pods-per-core': ['10'], 'max-pods': ['250'], 'image-gc-high-threshold': ['90'], 'image-gc-low-threshold': ['80']}

# enable ntp on masters to ensure proper failover
openshift_clock_enabled=true

# Setup rolling updates at the service level
openshift_rolling_restart_mode='services'

# Change all the certificates to last for 10 years.
openshift_hosted_registry_cert_expire_days='720'
openshift_ca_cert_expire_days='720'
openshift_node_cert_expire_days='720'
openshift_master_cert_expire_days='720'
etcd_ca_default_days='720'


#---- host group for masters
[masters]
uusrcdkub01.fossil.com
uusrcdkub02.fossil.com
uusrcdkub03.fossil.com

#---- host group for etcd
[etcd]
uusrcdkub01.fossil.com
uusrcdkub02.fossil.com
uusrcdkub03.fossil.com

#---- Specify load balancer host
#[lb]
#uusrcdkub00.fossil.com

#---- host group for nodes, includes region info
[nodes]
uusrcdkub01.fossil.com openshift_schedulable=true       openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
uusrcdkub02.fossil.com openshift_schedulable=true       openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
uusrcdkub03.fossil.com openshift_schedulable=true       openshift_node_labels="{'region': 'infra', 'zone': 'default'}"


# Create an OSEv3 group that contains the master, nodes, etcd, and lb groups.
# The lb group lets Ansible configure HAProxy as the load balancing solution.
# Comment lb out if your load balancer is pre-configured.
[OSEv3:children]
masters
nodes
etcd
#lb
Steps To Reproduce
  • After base CentOS 7.3 install
  1. Install pre-reqs and update the OS
yum -y install wget git net-tools bind-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct curl docker etcd atomic pyOpenSSL.x86_64 python-cryptography python-lxml
yum update
systemctl reboot
  1. Increase the root volume group to use the complete disk
fdisk /dev/sda
d
2
n
primary
2
default
default

systemctl reboot
pvresize /dev/sda2
  1. Update the docker config to use a LV for the docker storage
systemctl stop docker
rm -rf /var/lib/docker
vi /etc/sysconfig/docker-storage-setup
# File contents below
/bin/docker-storage-setup
systemctl start docker
systemctl enbale docker
docker info
lvs

File: /etc/sysconfig/docker-storage-setup

# Edit this file to override any configuration options specified in
# /usr/lib/docker-storage-setup/docker-storage-setup.
#
# For more details refer to "man docker-storage-setup"
#STORAGE_DRIVER
#DEVS=/dev/sda
VG=centos
GROWPART=enable
AUTO_EXTEND_POOL=enable
MIN_DATA_SIZE=8G
POOL_AUTOEXTEND_THRESHOLD=60
POOL_AUTOEXTEND_PERCENT=10
  • Install Ansible on the first master
    We need at least Ansible 2.4.3. At the time of writing this doc the latest published was 2.4.2. So, we need to role our own RPM. To build Ansible we need:
	yum -y install asciidoc rpm-build python2-devel
  • Build and install the latest Ansible
	mkdir Development
	git clone https://github.com/ansible/ansible.git
	cd ansible
	make rpm
	cd /root/Development/ansible/rpm-build
	rpm --install ansible-2.6.0-100.git201803142047.0cf2ecb.devel.el7.centos.noarch.rpm
  • Pull the OpenShift Ansible Source
	cd /etc/ansible/
	mkdir playbooks
	cd playbooks
	git clone https://github.com/openshift/openshift-ansible
  • Create the Ansible Host File
  vi /etc/ansbile/hosts

File contents listed above.

  • Run the pre-reqs and deploy
  cd /etc/ansible/playbooks/openshift-ansible/playbooks/
  ansible-playbook -i /etc/ansible/hosts $PWD/prerequisites.yml
	
  cd /etc/ansible/playbooks/openshift-ansible/playbooks/openshift-checks
  vi roles/openshift_health_checker/openshift_checks/package_version.py

I updated the allowed docker versions for (3,7) to include 1.13 and 1.13.1

  ansible-playbook -i /etc/ansible/hosts $PWD/pre-install.yml
    
  cd /etc/ansible/playbooks/openshift-ansible/playbooks/
  ansible-playbook -vvv -i /etc/ansible/hosts $PWD/deploy_cluster.yml
Expected Results

Describe what you expected to happen.

I expected the install and configuration to complete
Observed Results

Everything is chugging along find. The verbose output before the abend is listed below. I don't see any other errors in the output until this.

TASK [openshift_master_certificates : Generate the loopback master client config] ***************************************************************************************
task path: /etc/ansible/playbooks/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml:63
Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '/root\n', '')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1521658222.4-93760209595884 `" && echo ansible-tmp-1521658222.4-93760209595884="` echo /root/.ansible/tmp/ansible-tmp-1521658222.4-93760209595884 `" ) && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, 'ansible-tmp-1521658222.4-93760209595884=/root/.ansible/tmp/ansible-tmp-1521658222.4-93760209595884\n', '')
<uusrcdkub01.fossil.com> PUT /root/.ansible/tmp/ansible-local-20837_E4d2Q/tmpbS7CiZ TO /root/.ansible/tmp/ansible-tmp-1521658222.4-93760209595884/command.py
<uusrcdkub01.fossil.com> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 '[uusrcdkub01.fossil.com]'
...
Skip a lot
...
Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '/root\n', '')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726 `" && echo ansible-tmp-1521656686.1-214052715328726="` echo /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726 `" ) && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, 'ansible-tmp-1521656686.1-214052715328726=/root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726\n', '')
<uusrcdkub01.fossil.com> PUT /root/.ansible/tmp/ansible-local-115241x1n7s/tmp4R_seg TO /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/command.py
<uusrcdkub01.fossil.com> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 '[uusrcdkub01.fossil.com]'
<uusrcdkub01.fossil.com> (0, 'sftp> put /root/.ansible/tmp/ansible-local-115241x1n7s/tmp4R_seg /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/command.py\n', '')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/ /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/command.py && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '', '')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 -tt uusrcdkub01.fossil.com '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/command.py && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '\r\n{"invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "oc adm create-api-client-config\\n --certificate-authority=/etc/origin/master/ca.crt\\n  --client-dir=/etc/origin/generated-configs/master-uusrcdkub03.fossil.com\\n --groups=system:masters,system:openshift-master\\n --master=https://uusrcdkub03.fossil.com:8443\\n --public-master=https://uusrcdkub03.fossil.com:8443\\n --signer-cert=/etc/origin/master/ca.crt\\n --signer-key=/etc/origin/master/ca.key\\n --signer-serial=/etc/origin/master/ca.serial.txt\\n --user=system:openshift-master\\n --basename=openshift-master\\n --expire-days=720", "removes": null, "creates": "/etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig", "chdir": null, "stdin": null}}, "cmd": "oc adm create-api-client-config\\n --certificate-authority=/etc/origin/master/ca.crt\\n  --client-dir=/etc/origin/generated-configs/master-uusrcdkub03.fossil.com\\n --groups=system:masters,system:openshift-master\\n --master=https://uusrcdkub03.fossil.com:8443\\n --public-master=https://uusrcdkub03.fossil.com:8443\\n --signer-cert=/etc/origin/master/ca.crt\\n --signer-key=/etc/origin/master/ca.key\\n --signer-serial=/etc/origin/master/ca.serial.txt\\n --user=system:openshift-master\\n --basename=openshift-master\\n --expire-days=720", "stdout": "skipped, since /etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig exists", "rc": 0, "changed": false}\r\n', 'Shared connection to uusrcdkub01.fossil.com closed.\r\n')
<uusrcdkub01.fossil.com> ESTABLISH SSH CONNECTION FOR USER: root
<uusrcdkub01.fossil.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/21d2154973 uusrcdkub01.fossil.com '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1521656686.1-214052715328726/ > /dev/null 2>&1 && sleep 0'"'"''
<uusrcdkub01.fossil.com> (0, '', '')
ok: [uusrcdkub01.fossil.com -> uusrcdkub01.fossil.com] => (item=uusrcdkub03.fossil.com) => {
    "changed": false,
    "cmd": "oc adm create-api-client-config\n --certificate-authority=/etc/origin/master/ca.crt\n  --client-dir=/etc/origin/generated-configs/master-uusrcdkub03.fossil.com\n --groups=system:masters,system:openshift-master\n --master=https://uusrcdkub03.fossil.com:8443\n --public-master=https://uusrcdkub03.fossil.com:8443\n --signer-cert=/etc/origin/master/ca.crt\n --signer-key=/etc/origin/master/ca.key\n --signer-serial=/etc/origin/master/ca.serial.txt\n --user=system:openshift-master\n --basename=openshift-master\n --expire-days=720",
    "invocation": {
        "module_args": {
            "_raw_params": "oc adm create-api-client-config\n --certificate-authority=/etc/origin/master/ca.crt\n  --client-dir=/etc/origin/generated-configs/master-uusrcdkub03.fossil.com\n --groups=system:masters,system:openshift-master\n --master=https://uusrcdkub03.fossil.com:8443\n --public-master=https://uusrcdkub03.fossil.com:8443\n --signer-cert=/etc/origin/master/ca.crt\n --signer-key=/etc/origin/master/ca.key\n --signer-serial=/etc/origin/master/ca.serial.txt\n --user=system:openshift-master\n --basename=openshift-master\n --expire-days=720",
            "_uses_shell": false,
            "chdir": null,
            "creates": "/etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig",
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": "uusrcdkub03.fossil.com",
    "rc": 0,
    "stdout": "skipped, since /etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig exists",
    "stdout_lines": [
        "skipped, since /etc/origin/generated-configs/master-uusrcdkub03.fossil.com/openshift-master.kubeconfig exists"
    ]
}
ERROR! |failed expects hostvars is a dict
@AllanCano
Copy link
Author

Ansible -vvv output

@AllanCano AllanCano reopened this Mar 21, 2018
@AllanCano
Copy link
Author

Non-verbose output

@AllanCano
Copy link
Author

Found that the error is coming from:

/etc/ansible/playbooks/openshift-ansible/roles/lib_utils/filter_plugins/openshift_master.py line 487.

hostvars[inventory_hostname] is being passed in to the custom filter certificates_to_synchronize

I did a little googling and found that the type for hostvars[inventory_hostname] is returned as a class instead of a dict. I'm not a python scripter so this is taking longer than I hoped to work through anyone with an idea here on what to do?

debug code via altering line 487:

raise errors.AnsibleFilterError("|failed expects hostvars is a dict, found %s" % type(hostvars) )

@AllanCano
Copy link
Author

OK. I think I understand what's happening and how I got around the problem, though, I'm at least a week away from knowing enough python to prove/disprove anything.

In /etc/ansible/playbooks/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml around line 63 I simply changed the with_items clause from
- "{{ hostvars[inventory_hostname] | certificates_to_synchronize }}"
to
- "{{ hostvars[inventory_hostname]['ansible_facts'] | certificates_to_synchronize }}"
and everything progressed.

I think since the hostvars[inventory] is magic it doesn't exist as a dictionary until accessed. So, accessing a dictionary variable ansible_facts solves 2 problems of instantiating the variable and passing an actual dictionary to the customer filter.

Don't know how to raise this as a bug (possible), but if someone in the know ever reads this please let me know if I guessed right as to why the work-a-round worked and how to place this as a bug report.

@AllanCano AllanCano changed the title Deploy Cluster abends 'ERROR! |failed expects hostvars is a dict' after task openshift_master_certificates call to 'certificates_to_synchronize' causes openshift_master_certificates to abend with 'ERROR! |failed expects hostvars is a dict' Mar 26, 2018
@Pectojin
Copy link
Contributor

Pectojin commented Mar 29, 2018

@AllanCano, thanks for the detective work.

I got the same error in openshift_master.py (although it's on line 516 in my case).
Making your change to openshift_master_certificates/tasks/main.yml (now on line 108) also helped me past the failure.

For the sake of completeness, here's my openshift version settings:

openshift_deployment_type=origin
openshift_release=v3.7
openshift_pkg_version=-3.7.1
openshift_image_tag=v3.7.1

And running ansible 2.6.0.

@Pectojin
Copy link
Contributor

Just FYI

After fixing this issue manually I continued and ran into #4985, where it was suggested to work under ansible 2.4.2, so I downgraded.

After downgrading the above fix did not work, so I had to un-do it to continue installation under ansible 2.4.2.

abn added a commit to abn/openshift-ansible that referenced this issue Apr 15, 2018
Ensure that the openshift_master.py filter plugin handles ansible 2.4
and 2.5 hostvars gracefully.

This is required due to the changes introduced in ansible via changes in
ansible/ansible#35913.

Resolves: openshift#7596
abn added a commit to section6nz/openshift-ansible that referenced this issue Apr 15, 2018
Ensure that the openshift_master.py filter plugin handles ansible 2.4
and 2.5 hostvars gracefully.

This is required due to the changes introduced in ansible via changes in
ansible/ansible#35913.

Resolves: openshift#7596
abn added a commit to abn/openshift-ansible that referenced this issue Jun 4, 2018
Ensure that the openshift_master.py filter plugin handles ansible 2.4
and 2.5 hostvars gracefully.

This is required due to the changes introduced in ansible via changes in
ansible/ansible#35913.

Resolves: openshift#7596
@onknows
Copy link

onknows commented Jul 3, 2018

This fix works for me for deploy_cluster but not for
playbooks/openshift-master/redeploy-certificates.yml

This will again fail with this message.

To successfully run redeploy this fix needs to be reversed I think. So two separate 3.9 branches are necessary to successful run 3.9 branch. This looks like a mess.

@djfoley01
Copy link

@AllanCano Not sure if anyone confirmed this, but your resolution of modifying /etc/ansible/playbooks/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml has fixed our issue. This was line 95 for us.

@damiandragowski
Copy link

damiandragowski commented Jul 7, 2018

Hi @ALL,

I have the same issue with deploying openshift, but after apply this patch above everythink work fine.
Patch was made in origin/release-3.9 branch. I hope this will solve yours problems :).

Greatings,
Damian

diff --git a/playbooks/init/validate_hostnames.yml b/playbooks/init/validate_hostnames.yml
index b49f7dd..88b275f 100644
--- a/playbooks/init/validate_hostnames.yml
+++ b/playbooks/init/validate_hostnames.yml
@@ -40,4 +40,4 @@
     when:
     - openshift_ip is defined
     - openshift_ip not in ansible_all_ipv4_addresses
-    - openshift_ip_check | default(true) | bool
+    - openshift_ip_check | default(false) | bool
diff --git a/roles/container_runtime/defaults/main.yml b/roles/container_runtime/defaults/main.yml
index 4e27541..e54ed1f 100644
--- a/roles/container_runtime/defaults/main.yml
+++ b/roles/container_runtime/defaults/main.yml
@@ -87,7 +87,7 @@ openshift_use_crio_only: False
 l_openshift_image_tag_default: "{{ 'v' ~ openshift_release if openshift_release is defined else 'latest' }}"
 l_openshift_image_tag: "{{ openshift_image_tag | default(l_openshift_image_tag_default) | string}}"
 
-l_required_docker_version: '1.12'
+l_required_docker_version: '1.13'
 
diff --git a/roles/lib_utils/filter_plugins/openshift_master.py b/roles/lib_utils/filter_plugins/openshift_master.py
index e67b19c..062b8bf 100644
--- a/roles/lib_utils/filter_plugins/openshift_master.py
+++ b/roles/lib_utils/filter_plugins/openshift_master.py
@@ -483,8 +483,6 @@ class FilterModule(object):
     @staticmethod
     def certificates_to_synchronize(hostvars, include_keys=True, include_ca=True):
         ''' Return certificates to synchronize based on facts. '''
-        if not issubclass(type(hostvars), dict):
-            raise errors.AnsibleFilterError("|failed expects hostvars is a dict")
         certs = ['admin.crt',
                  'admin.key',
                  'admin.kubeconfig',

@s-u-b-h-a-k-a-r
Copy link

I am also facing the same error while installing openshift 3.9.0. What is the actual fix that needs to be made to fix this issue 'ERROR! |failed expects hostvars '

@IRobL
Copy link

IRobL commented Jul 7, 2018

Things have been going no where for me for days :(

@Tomelin
Copy link

Tomelin commented Jul 7, 2018

I am also facing the same error while installing openshift 3.9.0.

@Tomelin
Copy link

Tomelin commented Jul 7, 2018

My error is:

ok: [master.rafaeltomelin.co -> master.rafaeltomelin.co] => {
"changed": false,
"examined": 0,
"files": [],
"invocation": {
"module_args": {
"age": null,
"age_stamp": "mtime",
"contains": null,
"depth": null,
"excludes": null,
"file_type": "file",
"follow": false,
"get_checksum": false,
"hidden": false,
"paths": [
"/etc/origin/master/legacy-ca/"
],
"patterns": [
".*-ca.crt"
],
"recurse": false,
"size": null,
"use_regex": true
}
},
"matched": 0,
"msg": "/etc/origin/master/legacy-ca/ was skipped as it does not seem to be a valid directory or it cannot be accessed\n"
}

TASK [openshift_master_certificates : Create the master server certificate] *********************************************************************************
task path: /root/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml:39

TASK [openshift_master_certificates : Generate the loopback master client config] ***************************************************************************
task path: /root/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml:63
ERROR! |failed expects hostvars is a dict

@Tomelin
Copy link

Tomelin commented Jul 7, 2018

I change line 95 in openshift-ansible/roles/openshift_master_certificates/tasks/main.yml and success:

Line 95:

certificates_to_synchronize is a custom filter in lib_utils

#- "{{ hostvars[inventory_hostname] | certificates_to_synchronize }}"
to

  • "{{ hostvars[inventory_hostname]['ansible_facts'] | certificates_to_synchronize }}"

@plaimbock
Copy link

For release-3.9 and Ansible 2.6.0.0 this solved it for me: abn@b20684b

@IRobL
Copy link

IRobL commented Jul 8, 2018

Thanks @plaimbock! Using the patch branch got passed the error!

But did you happen to hit an error on task

openshift_web_console : Verify that the console is running

wherein the /usr/local/bin/oc get deployment webconsole -o json -n openshift-web-console command fails after 60 retries?

Manually running said command yields the below output (note that the hostname centos7 routes to it's local VBox NIC IP):

$  /usr/local/bin/oc get deployment webconsole -o json -n openshift-web-console
The connection to the server centos7:8443 was refused - did you specify the right host or port

@s-u-b-h-a-k-a-r
Copy link

By changing the below lines it solved for me
In /etc/ansible/playbooks/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml around line 63 I simply changed the with_items clause from

  • "{{ hostvars[inventory_hostname] | certificates_to_synchronize }}"
    to
  • "{{ hostvars[inventory_hostname]['ansible_facts'] | certificates_to_synchronize }}"
    and everything progressed.

@vrutkovs
Copy link
Member

vrutkovs commented Jul 9, 2018

So no modifications are required for ansible 2.4, right? This seems to be caused by ansible 2.6 changes

@IRobL
Copy link

IRobL commented Jul 9, 2018

You ask a very good question, @vrutkovs. The readme indicates that 2.5.x is out of the question, and that anything lower than 2.4.3.0 won't work either. Here's what I have available in Centos7's yum packages:

$ yum --showduplicates list ansible
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.centos.org
 * extras: mirror.centos.org
 * updates: mirror.centos.org
Installed Packages
ansible.noarch                                                                             2.6.0-1.el7                                                                               @epel
Available Packages
ansible.noarch                                                                             2.3.0.0-3.el7                                                                             centos-openshift-origin
ansible.noarch                                                                             2.4.2.0-2.el7                                                                             extras

I have a feeling 2.3.0.0 wouldn't work since it's technically lower than 2.4.3.0. I'm testing with immutable infrastructure and would be happy to test out other version of Ansible. What's an ideal way to get ansible 2.4.3.0 installed? Also, can validations be written into the code base to ensure that a compatible version of Ansible is being used, I have a feeling that would spare many people the frustration I felt last week.

@vrutkovs
Copy link
Member

vrutkovs commented Jul 9, 2018

2.4.2 would work, although its known to leak memory. Not sure why would CentOS not update to 2.4.3 - but that's a bug in CentOS anyway.

@lbrigman124
Copy link

I'm running from the EPEL package on CentOS. I have version 2.5.5 installed. No building of packages required.
Anyway. I also ran into this error. If the error has been there since March, I'm not sure why I haven't seen it. We are typically doing one or two installs a week.

@lbrigmanArris
Copy link

BTW, I tested the patch. It worked for me.

@damiandragowski
Copy link

@lbrigman124 Funny thing is that I had installed openshift on centos 7 last thursday and it went fine, when I tried to install on other machine ( same OS ) at saturday, the issue came out :).

@lbrigman124
Copy link

@damiandragowski I had the exact same experience. I have been using a fixed commit(c264b57) since about the middle of April without any issues including multiple changes in the version of Ansible. Friday of last week(July 6), I started seeing the above failure. Something else in the environment changed to trigger this failure.

@xrow
Copy link

xrow commented Jul 10, 2018

For release-3.9 and Ansible 2.6.0.0 this solved it for me: abn/openshift-ansible@b20684b

Will one merge it?

vrutkovs pushed a commit to vrutkovs/openshift-ansible that referenced this issue Jul 10, 2018
Ensure that the openshift_master.py filter plugin handles ansible 2.4
and 2.5 hostvars gracefully.

This is required due to the changes introduced in ansible via changes in
ansible/ansible#35913.

Resolves: openshift#7596
vrutkovs pushed a commit to vrutkovs/openshift-ansible that referenced this issue Jul 10, 2018
Ensure that the openshift_master.py filter plugin handles ansible 2.4
and 2.5 hostvars gracefully.

This is required due to the changes introduced in ansible via changes in
ansible/ansible#35913.

Resolves: openshift#7596
@j00p34
Copy link
Contributor

j00p34 commented Jul 10, 2018

The openshift/origin-ansible container v3.9 has the same issue (3.9.33)
tag v3.9.28 does work.

@vrutkovs
Copy link
Member

3.9 now supports ansible 2.6, fixed by #9125 (thanks @abn). 3.10 should work fine as well. 3.11 would require 2.6.

Other versions would still need 2.4 as support for both 2.4 and 2.6 would be too complex.

@sailinnthu
Copy link

By changing the below lines it solved for me
In /etc/ansible/playbooks/openshift-ansible/roles/openshift_master_certificates/tasks/main.yml

"{{ hostvars[inventory_hostname] | certificates_to_synchronize }}"
to
"{{ hostvars[inventory_hostname]['ansible_facts'] | certificates_to_synchronize }}"
and everything progressed.

@nstrug
Copy link

nstrug commented Jul 27, 2018

This is https://bugzilla.redhat.com/show_bug.cgi?id=1605046
The fix is on QA and should be released soon.
In the meantime you can apply the patch from #9125

@karthik101
Copy link

karthik101 commented Oct 1, 2018

I have something similar issue while redeploying master certificates and ended up with errors below:

ASK [Remove generated certificates] **************************************************************************************************************************
fatal: [master1.com]: FAILED! => {
"failed": true
}

MSG:

Unexpected templating type error occurred on ({{ hostvars[inventory_hostname] | certificates_to_synchronize(include_keys=false, include_ca=false) }}): certificates_to_synchronize() got an unexpected keyword argument 'include_ca'

openshift-ansible version : openshift-ansible-3.4.90-1
oc version: oc v1.4.1
Ansible: ansible-2.3.2.0-2.el7.noarch
#10265

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests