Failed to find exact match for v1.Namespace by [kind, name, singularName, shortNames]

[rabin-io]

SUMMARY

When trying to create a namespace for a new operator with OpenShift 4.9.15, I get this error message, when I run the playbook from a RHEL 8.2 node, but from my local machine running Fedora 35, I don't have this issue.

TASK [olm-operator : Create Namespace for OLM operator] *************************************************
fatal: [localhost]: FAILED! => changed=false 
  msg: Failed to find exact match for v1.Namespace by [kind, name, singularName, shortNames]

ISSUE TYPE

• Bug Report

COMPONENT NAME

kubernetes.core.k8s

ANSIBLE VERSION

• From RHEL 8.2 node (with the problem)

ansible [core 2.12.1]
  config file = /root/workspace/deploy-cnv-4.9-on-ibmc-upi-ryasharz/cnv-qe-automation/ocp/bm-upi/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /root/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /root/.local/bin/ansible
  python version = 3.8.0 (default, Mar  9 2020, 18:02:46) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
  jinja version = 3.0.3
  libyaml = True

• From fedora 35 node

ansible [core 2.12.1]
  config file = ${HOME}/src/ansible-openshift-upi-install/ansible.cfg
  configured module search path = ['${HOME}/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = ${HOME}/.local/lib/python3.10/site-packages/ansible
  ansible collection location = ${HOME}/.ansible/collections:/usr/share/ansible/collections
  executable location = ${HOME}/.local/bin/ansible
  python version = 3.10.2 (main, Jan 17 2022, 00:00:00) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)]
  jinja version = 3.0.3
  libyaml = True

COLLECTION VERSION

# ${HOME}/.local/lib/python3.10/site-packages/ansible_collections
Collection      Version
--------------- -------
kubernetes.core 2.2.3

# ${HOME}/.ansible/collections/ansible_collections
Collection      Version
--------------- -------
kubernetes.core 2.2.2  

CONFIGURATION

CACHE_PLUGIN(${WORKDIR}/bm-upi/ansible.cfg) = jsonfile
CACHE_PLUGIN_CONNECTION(${WORKDIR}/bm-upi/ansible.cfg) = ./.ansible/facts_cache
CACHE_PLUGIN_TIMEOUT(${WORKDIR}/bm-upi/ansible.cfg) = 86400
DEFAULT_GATHERING(${WORKDIR}/bm-upi/ansible.cfg) = smart
DEFAULT_HASH_BEHAVIOUR(${WORKDIR}/bm-upi/ansible.cfg) = merge
DEFAULT_HOST_LIST(${WORKDIR}/bm-upi/ansible.cfg) = ['${WORKDIR}/bm-upi/inventory/beaker.yaml']
DEFAULT_LOAD_CALLBACK_PLUGINS(${WORKDIR}/bm-upi/ansible.cfg) = True
DEFAULT_MANAGED_STR(${WORKDIR}/bm-upi/ansible.cfg) = This file is managed by Ansible.%n
template: {file}
date    : %Y-%m-%d %H:%M:%S
by      : {uid}@{host}
DEFAULT_STDOUT_CALLBACK(${WORKDIR}/bm-upi/ansible.cfg) = yaml
RETRY_FILES_SAVE_PATH(${WORKDIR}/bm-upi/ansible.cfg) = ${WORKDIR}/bm-upi/.ansible/retry

OS / ENVIRONMENT

• The failing node is a RHEL 8.2

STEPS TO REPRODUCE

---
- hosts: localhost
  gather_facts: false
  vars:
  tasks:

    - name: Create Namespace for OLM operator
      kubernetes.core.k8s:
        definition:
          apiVersion: v1
          kind: Namespace
          metadata:
            name: local-storage-operator
            labels: "{{ ns_labels | default(omit) }}"
            annotations: "{{ ns_annotations | default(omit) }}"

EXPECTED RESULTS

Apply/Create the Namespace or just skip if the Namespace is already exists.

ACTUAL RESULTS

Fail with the message

TASK [Create Namespace for OLM operator] ******************************************************************************************************
fatal: [localhost]: FAILED! => changed=false 
  msg: Failed to find exact match for v1.Namespace by [kind, name, singularName, shortNames]

https://gist.github.com/rabin-io/ac9e6f81c377e037804096bb61647ac9

[Akasurde]

@rabin-io Thanks for reporting this issue. Could you please mention the Kubernetes library version like -

pip list | grep kuber
kubernetes                         12.0.1
kubernetes-validate                1.19.0

[rabin-io]

On both nodes

kubernetes               12.0.1

[rabin-io]

I tried to diff the zipped AnsiballZ_k8s.py files, to see if there is anything different in the generated output of the modules, but they seems to be the same.

grep 'ZIPDATA = ' /tmp/rhel.AnsiballZ_k8s.py | awk '{print($3)}' | tr -d '"' | base64 -d > /tmp/r/rhel.zip
grep 'ZIPDATA = ' /tmp/fedora.AnsiballZ_k8s.py | awk '{print($3)}' | tr -d '"' | base64 -d > /tmp/f/fedora.zip
# for folders r and f unzip each file
diff -y f r
Common subdirectories: f/ansible and r/ansible
Common subdirectories: f/ansible_collections and r/ansible_collections
Only in f: fedora.zip
Only in r: rhel.zip

[gravesm]

Per our debugging session, I'm going to close this as it seems to be working now. Feel free to reopen if needed.

[Akasurde]

@gravesm Could you please add the debugging steps and output here so that we record them for future use?

[gravesm]

The steps just involved me watching it get run. That was enough to make it work. My best guess is that there was probably something cached somewhere that was causing the problem and it eventually just expired. I don't have any other explanation for it. There's no reason I can see why it would have failed in the first place.

[rabin-io]

@gravesm I think you are right, I think it started working once I created a new user, and run the playbook from it. Running the playbook from the root user (yes I know, bad idea) still reproduce the error.

Another thing I like to add here, is the snippet code which can help to debug the problem, based on the location my playbook failed, and the stack trace.

The full traceback is:                                                                                                                                  
  File "/tmp/ansible_kubernetes.core.k8s_payload_78xg0aj9/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_uti
ls/common.py", line 243, in find_resource                                                                                                               
    return self.client.resources.get(api_version=api_version, short_names=[kind])                                                                       
  File "/tmp/ansible_kubernetes.core.k8s_payload_78xg0aj9/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_ut$
ls/client/discovery.py", line 140, in get                                                                                                               
    results = self.search(**kwargs)                                                                                                                     
  File "/root/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py", line 237, in search                                                  
    results = self.__search(self.__build_search(**kwargs), self.__resources, [])                                                                        
  File "/root/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py", line 283, in __search                                                
    matches.extend(self.__search([key] + parts[1:], resources, reqParams))
  File "/root/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py", line 269, in __search
    return self.__search(parts[1:], resourcePart, reqParams + [part] )
  File "/root/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py", line 283, in __search
    matches.extend(self.__search([key] + parts[1:], resources, reqParams))
  File "/root/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py", line 269, in __search
    return self.__search(parts[1:], resourcePart, reqParams + [part] )
  File "/root/.local/lib/python3.8/site-packages/kubernetes/dynamic/discovery.py", line 261, in __search
    raise ResourceNotFoundError

import kubernetes
from kubernetes.dynamic import DynamicClient

kubernetes.config.load_kube_config()
config = kubernetes.client.Configuration().get_default_copy()
client = DynamicClient(kubernetes.client.ApiClient(config))
resource = client.resources.get(api_version="v1", kind="Namespace")

[gravesm]

Could you try running the following in a playbook and post the output?

- kubernetes.core.k8s_cluster_info:
  register: output

- debug:
    msg: "{{ output.apis.v1 }}"

[Akasurde]

Thanks @gravesm @rabin-io

[rabin-io]

Hi @gravesm, see the output below,
what wired is that I re-deployed the cluster last night, and I didn't encounter this issue.
but today, I rerun the job, and it failed again with the above message

PLAY [localhost] ***************************************************************

TASK [kubernetes.core.k8s_cluster_info] ****************************************
ok: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => 
  msg:
    Binding:
      categories: []
      name: bindings
      namespaced: true
      preferred: true
      short_names: []
      singular_name: binding
    ComponentStatus:
      categories: []
      name: componentstatuses
      namespaced: false
      preferred: true
      short_names:
      - cs
      singular_name: componentstatuse
    ConfigMap:
      categories: []
      name: configmaps
      namespaced: true
      preferred: true
      short_names:
      - cm
      singular_name: configmap
    Endpoints:
      categories: []
      name: endpoints
      namespaced: true
      preferred: true
      short_names:
      - ep
      singular_name: endpoint
    Event:
      categories: []
      name: events
      namespaced: true
      preferred: true
      short_names:
      - ev
      singular_name: event
    LimitRange:
      categories: []
      name: limitranges
      namespaced: true
      preferred: true
      short_names:
      - limits
      singular_name: limitrange
    List:
      categories: []
      name: null
      namespaced: null
      preferred: null
      short_names: []
      singular_name: null
    Namespace:
      categories: []
      name: namespaces
      namespaced: false
      preferred: true
      short_names:
      - ns
      singular_name: namespace
    Node:
      categories: []
      name: nodes
      namespaced: false
      preferred: true
      short_names:
      - 'no'
      singular_name: node
    PersistentVolume:
      categories: []
      name: persistentvolumes
      namespaced: false
      preferred: true
      short_names:
      - pv
      singular_name: persistentvolume
    PersistentVolumeClaim:
      categories: []
      name: persistentvolumeclaims
      namespaced: true
      preferred: true
      short_names:
      - pvc
      singular_name: persistentvolumeclaim
    Pod:
      categories:
      - all
      name: pods
      namespaced: true
      preferred: true
      short_names:
      - po
      singular_name: pod
    PodTemplate:
      categories: []
      name: podtemplates
      namespaced: true
      preferred: true
      short_names: []
      singular_name: podtemplate
    ReplicationController:
      categories:
      - all
      name: replicationcontrollers
      namespaced: true
      preferred: true
      short_names:
      - rc
      singular_name: replicationcontroller
    ResourceQuota:
      categories: []
      name: resourcequotas
      namespaced: true
      preferred: true
      short_names:
      - quota
      singular_name: resourcequota
    Secret:
      categories: []
      name: secrets
      namespaced: true
      preferred: true
      short_names: []
      singular_name: secret
    Service:
      categories:
      - all
      name: services
      namespaced: true
      preferred: true
      short_names:
      - svc
      singular_name: service
    ServiceAccount:
      categories: []
      name: serviceaccounts
      namespaced: true
      preferred: true
      short_names:
      - sa
      singular_name: serviceaccount

TASK [Create Namespace for OLM operator] ***************************************
fatal: [localhost]: FAILED! => changed=false 
  msg: Failed to find exact match for v1.Namespace by [kind, name, singularName, shortNames]

[gravesm]

Reopening this one. We'll need to find a way to reliably reproduce this. There's no obvious reason why it's randomly failing. The cluster info shows that the namespace resource exists.

[rabin-io]

background info

• The playbook run on a Jenkins agent node, running RHEL (8.5)
• The job deploy OpenShift cluster every few hours for testing.
• The Jenkins agent node is dedicated for this job only.

Insights

One solution we found is related to the created CACHE file, under the /tmp folder prefiex with k8srcp-*, they remain on the node between deployment, so we have an old cache file, from an older deployment. Clearing them before running the job, seems to avoid the error.

sudo  find /tmp -maxdepth 1 -name "k8srcp*" -ls -delete

One more thing, not sure if it is related, running packet capture to monitor the API calls of the modules, show that the code is querying a api endpoint which return 404, and there is 4 attempts, and then the connection terminate. Can it be that the ResourceNotFound exception is thrown from HTTP status code ?

[gravesm]

Thanks @rabin-io for the wireshark capture. I think this explains things. I manually forced the kubernetes client to throw a 404 when trying to fetch resource information to write to the cache and am able to get the same failure behavior you are experiencing. I strongly suspect these 404 errors you are seeing in the wireshark capture are responsible.

I don't know why openshift is returning 404 for these ceph apis, though. In the k8s client, when generating the resource cache, there's an initial request to get the list of apis (you should see a GET request for /apis) and then a subsequent request for each of the api groups. These requests are only done once to generate the cache in /tmp. Further k8s client calls will use this resource cache and skip these initial calls. I would think if the ceph api does not exist then it shouldn't be in the initial list of apis (that first /apis request), but that does not seem to be the case. I don't know enough about openshift to know whether this is intentional behavior or a bug. My suspicion is that this is happening shortly after you spin up a new openshift cluster and the api has not fully populated, but I don't know.

If you wanted to try and further investigate, it might be worth seeing if this reliably happens right after the cluster gets spun up. That would be further evidence of a possible race condition. You could try and reproduce this at the openshift level by just directly making calls against the openshift api. First, a GET request to /apis to confirm that ceph.rook.io is in that list and then another GET to /apis/ceph.rook.io/v1 to see if you get a 404.

[rabin-io]

Can it be that the module is trying to "refresh" cache based on the cache last values ?

As part of the deployment of openshift I do install odf/ceph, so it possible the cache file is updated, and when I reset the cluster and redeploy it again, the cache file contain the old reference to the ceph api which it try to query.

[gravesm]

@rabin-io After talking with @fabianvf I think we may have a workaround for this in #364. If you'd like, you can try using the fix and see if that addresses the failure.

[rabin-io]

@gravesm I will, but at the moment I have a problem in my infrastructure which cussing problem with the nodes boot process, so I can't test this right know. But I will the moment I resolve my boot issue.

Thanks for the quick respond.

[rabin-io]

Hi @gravesm , I can confirm that your branch did work, I run 3 deploys, with/o my workaround and none of them failed.

Thanks again.

[houshym]

Hi
I have the same error. I did not get a resolution for this issue. when I ran it from my macOS it works but for my colleague's macOS. just I have ansible [core 2.12.1] and another workstation is 2.12.2
ansible [core 2.12.2]

config file = None

configured module search path = ['/Users/myuser/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']

ansible python module location = /usr/local/Cellar/ansible/5.3.0/libexec/lib/python3.10/site-packages/ansible

ansible collection location = /Users/myuser/.ansible/collections:/usr/share/ansible/collections

executable location = /usr/local/bin/ansible

python version = 3.10.2 (main, Feb 2 2022, 06:19:27) [Clang 13.0.0 (clang-1300.0.29.3)]

jinja version = 3.0.3

libyaml = True

[rabin-io]

@houshym did you try removing the temp/cache files and try again ?

sudo  find /tmp -maxdepth 1 -name "k8srcp*" -ls -delete