Multiple CR Versions in watches.yaml results in duplicate Ansible reconciliations

[jessebye]

Bug Report

What did you do?
Created a CRD with two versions:

...
  versions:
    - name: v1
      served: true
      storage: false
    - name: v2
      served: true
      storage: true

Created a watches.yaml with:

- version: v1
  group: aws.rsk.io
  kind: IAMRole
  reconcilePeriod: 24h
  role: /opt/ansible/roles/iam-role/v1
- version: v2
  group: aws.rsk.io
  kind: IAMRole
  reconcilePeriod: 24h
  role: /opt/ansible/roles/iam-role/v2

Created a CR on v2:

apiVersion: aws.rsk.io/v2
kind: IAMRole
metadata:
  name: test
spec:
  dependentQueues:
    - name: test

What did you expect to see?
Only the v2 role should have been invoked.

What did you see instead? Under which circumstances?
Both the v1 and v2 roles are invoked simultaneously. Here were the operator logs:

operator TASK [v2 : Fetch AWS config values from k8s ***********************************
operator task path: /opt/ansible/roles/iam-role/v2/tasks/main.yml:1
operator -------------------------------------------------------------------------------
operator {"level":"info","ts":1588290111.9663565,"logger":"logging_event_handler","msg":"[playbook task","name":"pronode","namespace":"defensive-yards","gvk":"aws.rsk.io/v2, Kind=IAMRole","event_type":"playbook_on_task_start","job":"6032467244848876436","EventData.Name":"v2 : Fetch AWS config values from k8s"}
operator {"level":"info","ts":1588290111.9859517,"logger":"logging_event_handler","msg":"[playbook task","name":"pronode","namespace":"defensive-yards","gvk":"aws.rsk.io/v1, Kind=IAMRole","event_type":"playbook_on_task_start","job":"907430288210826867","EventData.Name":"v1 : Fetch AWS config values from k8s"}
operator --------------------------- Ansible Task StdOut -------------------------------
operator
operator TASK [v1 : Fetch AWS config values from k8s ***********************************
operator task path: /opt/ansible/roles/iam-role/v1/tasks/main.yml:1
operator -------------------------------------------------------------------------------
operator {"level":"info","ts":1588290112.0637093,"logger":"logging_event_handler","msg":"[playbook task","name":"pronode","namespace":"defensive-yards","gvk":"aws.rsk.io/v2, Kind=IAMRole","event_type":"playbook_on_task_start","job":"6032467244848876436","EventData.Name":"v2 : Fetch AWS configuration from Kubernetes"}
operator --------------------------- Ansible Task StdOut -------------------------------
operator
operator TASK [v2 : Fetch AWS configuration from Kubernetes ****************************
operator task path: /opt/ansible/roles/common/aws-config.yml:2
operator -------------------------------------------------------------------------------
operator {"level":"info","ts":1588290112.0763645,"logger":"logging_event_handler","msg":"[playbook task","name":"pronode","namespace":"defensive-yards","gvk":"aws.rsk.io/v1, Kind=IAMRole","event_type":"playbook_on_task_start","job":"907430288210826867","EventData.Name":"v1 : Fetch AWS configuration from Kubernetes"}
operator --------------------------- Ansible Task StdOut -------------------------------

Environment

• operator-sdk version:

v0.17.0

• go version:

n/a

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T23:35:15Z", GoVersion:"go1.14.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-af3caf", GitCommit:"af3caf6136cd355f467083651cc1010a499f59b1", GitTreeState:"clean", BuildDate:"2020-03-27T21:51:36Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

EKS

• Are you writing your operator in ansible, helm, or go?

Ansible

Possible Solution
n/a

Additional context
n/a

[camilamacedo86]

Hi @jessebye,

Thank you for contacting us. Could you please let us know if you have a CR for each version applied or not? Could you please check that and let us know?

[jessebye]

@camilamacedo86 I only have one CR applied (the v2 resource).

[camilamacedo86]

HI @jessebye,

Really great to know. So, may we have some bug in the ansible watch which is not checking the versions. So, IMO the next steps for we analyse it and then provide a fix if it is the case are:

• Create a POC based in the Ansible Memcache Sample with
• Then, after reproducing the issue in the POC and confirm the behaviour we can troubleshoot it.

Notes :

• here is the load of the watches and may it be related to the scenario of the watch based on the annotations
• Also, just to share in the case of you wish collab with. See the doc which describe how to test the changes locally.

So, please feel free to work on these steps and contribute. Otherwise, we will try o check it as soon as possible.

@fabianvf wdyt? Do you agree with the steps? Would you like to supplement it?

[joelanford]

This may be related to this comment: #2895 (comment)

I don't think multi-version CRDs are supported with Ansible-based operators.

At the very least, we may need to document this limitation.

[fabianvf]

Yeah, I think long-term the proper way to handle multi-version CRDs is with a conversion webhook, which we've not yet tackled with Ansible-based Operators. As a workaround for now, I'd recommend creating a playbook that will conditionally import either the v1 or v2 role based on the API version found in the _aws_rsk_io_iamrole.apiVersion variable (_aws_rsk_io_iamrole is the unmodified resource that triggered the event)

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[camilamacedo86]

This issue was answered already. See the latest comment.

[simararneja]

@camilamacedo86 @fabianvf At the very least can we have something added to the documentation please ? There are still grey areas to support multi versions in our operators ?

1. What goes into watches.yaml
2. Do they still point to same playbook, or an abstract playbook for both
3. What goes into PROJECT

Thanks in advance.

[simararneja]

Even with creating separate roles as suggested in the comment above and conditionally including them is invoking the ansible with both v1 and v2

 TASK [debug crdVersion] ********************************
ok: [localhost] => {
    "crdVersion": "v2"
}

-------------------------------------------------------------------------------
{"level":"info","ts":1632745980.4025545,"logger":"logging_event_handler","msg":"[playbook debug]","name":"openpagesinstance3","namespace":"zen","gvk":"openpages.cpd.com/v2, Kind=OpenPagesInstance","event_type":"runner_on_ok","job":"6426100070888298971","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug crdVersion] ********************************
ok: [localhost] => {
    "crdVersion": "v1"
}

I can confirm that CRD which is created is only v2.

Edit -

crdVersion is extracted from

_openpages_cpd_com_openpagesinstance.apiVersion

  - name: community.kubernetes
    version: "1.2.1"
  - name: operator_sdk.util
    version: "0.2.0"

[lucas4p]

Referring to Multiple CR Versions in watches.yaml results in duplicate Ansible reconciliations, do you know if there is already a solution?