Ansible: how to prevent CR deletion on finalizer playbook failure

[tomsucho]

Type of question

how to implement a specific feature

Question

What did you do?
When deleting a Custom Resource(CR) object the relevant playbook is executed (as per my finalizer dict config in watches.yml) however the CR is deleted from k8s no matter what happens in the playbook i.e. even if it fails. This causes inconsistent state, in which CR is no longer present in k8s, but the underlaying/related resources which live outside of the cluster are not really cleaned up properly. I was hoping that if the playbook fails, then the finalizer is not going to be removed, which will keep the resource object in k8s. If that would be the case, then I could use k8_status from the playbook to flag to end user that deletion and related cleanup actions didn't succeed and the user would be able to manually clean up things and then edit the resource and remove the finalizer manually for the delete process to complete. Could you please advise how to achieve this?
BTW> I'm using "manageStatus: False"

What did you expect to see?
CR not deleted from k8s in case finalizer linked playbook fails

What did you see instead? Under which circumstances?
CR is deleted from k8s no matter that finalizer linked playbook failed to complete all its actions

Environment

• operator-sdk version:
  quay.io/operator-framework/ansible-operator:v0.15.1

  insert release or Git SHA here

• Kubernetes version information:

  insert output of kubectl version here

• Kubernetes cluster kind:

Additional context
Add any other context about the question here.

[tomsucho]

Testing this further it looks like behavior may depend on the type of a problem during Ansible playbook run. The behavior I described above was when there was Ansible playbook error with syntax, however when there's no error but just a task failure, then it works as expected.

[djzager]

So you have established that when the Ansible runs successfully but with an error, it behaves as you expect. But the fact that you had a syntax error is interesting. While a syntax error is the most likely culprit, I doubt it is the only way that it could fail.

There are two scenarios where ansible-runner (ansible-playbook under the covers) fails that should be considered:

1. Normal Reconcile, ie. when the playbook is being run in response to generic event.
2. Pending Finalizers, ie. when the playbook is being run and the resource has been deleted.

What concerns me is that it appears that the run is being treated as successful when the actual command fails.

• In the reconciler https://github.com/operator-framework/operator-sdk/blob/master/pkg/ansible/controller/reconcile.go#L158-L166
• In the runner you can see only an error is logged

  operator-sdk/pkg/ansible/runner/runner.go

  Lines 221 to 226 in 0eac8a1

  	output, err := dc.CombinedOutput()
  	if err != nil {
  	logger.Error(err, string(output))
  	} else {
  	logger.Info("Ansible-runner exited successfully")
  	}

The only question that remains for me is what should the expected behavior be when the ansible command fails? My guess is that the ansible controller should:

• Get notified of the error by the runner (ie. runner needs to be updated to bubble up the failure)
• Put something on the status of the resource because it failed
• Get requeued return reconcileResult, err

[tomsucho]

In general one should thoroughly test the playbook, so that playbook fatal errors don't happen.. but on the other hand you never know, and there could always be this one corner case like something in the environment changed, something you didn't thought about, which would cause Ansible to error out... It would be lot safer if the framework could catch any type of errors and flag it to the user through the resource status, without deleting it silently.

[camilamacedo86]

Hi @tomsucho,

Could you please advise how to achieve this?
BTW> I'm using "manageStatus: False"

Are you using manageStatus: false? Am I right? Then, it means that the operator will ignore any event error faced. See:

Ref: https://github.com/operator-framework/operator-sdk/blob/master/doc/ansible/user-guide.md

However, if you face the same whiteout this configuration, could you please provide a simple POC like the Memcached example for we can see and reproduce your scenario? Or just the steps with snippets of the code of the setup that you are doing. And then, let us know what you are facing and what would you like to see instead of?

[joelanford]

@camilamacedo86 I think @djzager summed it up nicely. It looks like we may need some extra handling of errors (other than just logging) when the ansible-runner process exits with a non-zero exist status.

/cc @fabianvf

[camilamacedo86]

HI @joelanford and @djzager,

• Regards the issue scenario:

In the first comment on this issue, we have the affirmation that they are using "manageStatus: False".Then, it will make the operator ignore any event which came from the tasks that face failures and will finalize as he describes. See:

https://github.com/operator-framework/operator-sdk/blob/v0.15.x/pkg/ansible/controller/reconcile.go#L213-L231

• Regards the @djzager comment.

The only question that remains for me is what should the expected behavior be when the ansible command fails? My guess is that the ansible controller should:

Get notified of the error by the runner (ie. runner needs to be updated to bubble up the failure)
Put something on the status of the resource because it failed
Get requeued return reconcileResult, err

Then, if the ansible command fails I understand that it means that no event.Status was returned. Am I right? So, see follows that it will be marked with Failed to get ansible-runner stdout and retrigger the reconcile.

https://github.com/operator-framework/operator-sdk/blob/v0.15.x/pkg/ansible/controller/reconcile.go#L189-L198

In this way, if I misunderstand something here, could you please provide the steps to reproduce the scenario where the "manageStatus: False" is NOT used and the playbooks fails whiteout re-trigger the reconcile?

[tomsucho]

@camilamacedo86 I do have "manageStatus: False". However from that I can see in my testing when there is something wrong and task fails, then the resource is not deleted and reconciliation loop still goes on. I think this principle works also when creating resources, no matter the manageStatus setting and basically if something fails, then the reconciliation will keep going on retrying (even if there is no reconcilePeriod set explicitly). This is great to have as then I can simply catch the failure in my playbook and use k8_status module to update resource Status Conditions accordingly(since it is not auto-managed), to hint the user that deletion didn't work fine. However when I made indentation issue, which wasn't flagged by my editor which resulted in ansible ERROR like follows:

`task path: /opt/ansible/my-playbook.yml:6
ERROR! 'include_tasks' is not a valid attribute for a Block

The error appears to be in '/opt/ansible/roles/my-role/tasks/main.yml': line 13, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- name: Do something here
  ^ here

PLAY RECAP *********************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0`

then the resource was deleted from k8s despite my resource removal playbook didn't even start as per above output..
I wanted to flag this since while this type of errors I caused myself, can be easily catched during dev cycle, maybe there can be some other error situations out there that will happen only during runtime and then not catching these errors and silently deleting resources without really executing the playbook will lead to inconsistencies. And I suspect same situation when creating resources, if same type of Ansible ERROR happened. It would be good to cover for this even if manageStatus: false I think.

[camilamacedo86]

Hi @tomsucho,
Really tks for your info and I will try to create a POC for we reproduce the scenario and check it as well.
c/c @joelanford @djzager @fabianvf

[camilamacedo86]

Hi @tomsucho,

I did a POC with a syntax error in the Ansible task, and I could check that:

• The reconcile still be retrigged
• The error Failed to get ansible-runner stdout will be faced as described in Ansible: how to prevent CR deletion on finalizer playbook failure #2546 (comment)
• Also, the same behaviour will occur no matter the manageStatus: False in this scenario.

You can see all the details below.

Then, could you please input here the content of your build/Dockerfile?

PS.: I am asking that because in the 0.14.0 version of SDK we did a fix related to the issue of the reconcile not be re-trigged in failure scenarios. See in the CHANGELOG. You described that you are using the operator-sdk version 0.15.1. However, was the project scaffolded with this version and/or upgrade to use the 0.15.1 ansible image?

Following my POC.

1. I added the main task which has a syntax error as follows: (roles/testcr/tasks/main.yml)

---
# tasks file for testcr

- msg: invalid sintax
    include_tasks: include_tasks:

2. I added a watch to call this task:

---
- version: v1alpha1
  group: glothriel.com
  kind: TestCR
  role: testcr

3. And then, see that the reconcile will be retrigged as I described above.

Then, if the ansible command fails I understand that it means that no event.Status was returned. Am I right? So, see follows that it will be marked with Failed to get ansible-runner stdout and retrigger the reconcile.

https://github.com/operator-framework/operator-sdk/blob/v0.15.x/pkg/ansible/controller/reconcile.go#L189-L198

Following the logs (kubectl logs deployment.apps/testoperator -c operator -n default)

 $ kubectl logs deployment.apps/testoperator -c operator -n default
{"level":"info","ts":1581591309.5070565,"logger":"cmd","msg":"Go Version: go1.13.3"}
{"level":"info","ts":1581591309.5071228,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1581591309.5071518,"logger":"cmd","msg":"Version of operator-sdk: v0.15.0+git"}
{"level":"info","ts":1581591309.507189,"logger":"cmd","msg":"Watching namespace.","Namespace":"default"}
{"level":"info","ts":1581591309.828779,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1581591309.8295484,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"WORKER_TESTCR_GLOTHRIEL_COM","default":1}
{"level":"info","ts":1581591309.8299675,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_TESTCR_GLOTHRIEL_COM","default":2}
{"level":"info","ts":1581591309.8303409,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"glothriel.com","Options.Version":"v1alpha1","Options.Kind":"TestCR"}
{"level":"info","ts":1581591309.8307326,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1581591310.1490538,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1581591310.1549094,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1581591310.7842658,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"testoperator-metrics","Service.Namespace":"default"}
{"level":"info","ts":1581591311.0901892,"logger":"cmd","msg":"Could not create ServiceMonitor object","Namespace":"default","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1581591311.0908258,"logger":"cmd","msg":"Install prometheus-operator in your cluster to create ServiceMonitor objects","Namespace":"default","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1581591311.0921226,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
{"level":"info","ts":1581591311.092865,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"testcr-controller","source":"kind source: glothriel.com/v1alpha1, Kind=TestCR"}
{"level":"info","ts":1581591311.09327,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"testcr-controller"}
{"level":"info","ts":1581591311.094092,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1581591311.1940362,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"testcr-controller","worker count":1}
{"level":"error","ts":1581591313.8124533,"logger":"runner","msg":"ansible-playbook 2.9.4\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nERROR! Syntax Error while loading YAML.\r\n  mapping values are not allowed here\r\n\r\nThe error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 5, column 18, but may\r\nbe elsewhere in the file depending on the exact syntax problem.\r\n\r\nThe offending line appears to be:\r\n\r\n- msg: invalid sintax\r\n    include_tasks: include_tasks:\r\n                 ^ here\r\n","job":"6129484611666145821","name":"example-testcr","namespace":"default","error":"exit status 4","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:223"}
{"level":"error","ts":1581591313.8128512,"logger":"reconciler","msg":"ansible-playbook 2.9.4\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nERROR! Syntax Error while loading YAML.\r\n  mapping values are not allowed here\r\n\r\nThe error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 5, column 18, but may\r\nbe elsewhere in the file depending on the exact syntax problem.\r\n\r\nThe offending line appears to be:\r\n\r\n- msg: invalid sintax\r\n    include_tasks: include_tasks:\r\n                 ^ here\r\n","job":"6129484611666145821","name":"example-testcr","namespace":"default","error":"did not receive playbook_on_stats event","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/controller.(*AnsibleOperatorReconciler).Reconcile\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/controller/reconcile.go:197\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1581591313.8130338,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"testcr-controller","request":"default/example-testcr","error":"did not receive playbook_on_stats event","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1581591317.3889155,"logger":"runner","msg":"ansible-playbook 2.9.4\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nERROR! Syntax Error while loading YAML.\r\n  mapping values are not allowed here\r\n\r\nThe error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 5, column 18, but may\r\nbe elsewhere in the file depending on the exact syntax problem.\r\n\r\nThe offending line appears to be:\r\n\r\n- msg: invalid sintax\r\n    include_tasks: include_tasks:\r\n                 ^ here\r\n","job":"4037200794235010051","name":"example-testcr","namespace":"default","error":"exit status 4","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:223"}
{"level":"error","ts":1581591317.3891814,"logger":"reconciler","msg":"ansible-playbook 2.9.4\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nERROR! Syntax Error while loading YAML.\r\n  mapping values are not allowed here\r\n\r\nThe error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 5, column 18, but may\r\nbe elsewhere in the file depending on the exact syntax problem.\r\n\r\nThe offending line appears to be:\r\n\r\n- msg: invalid sintax\r\n    include_tasks: include_tasks:\r\n                 ^ here\r\n","job":"4037200794235010051","name":"example-testcr","namespace":"default","error":"did not receive playbook_on_stats event","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/controller.(*AnsibleOperatorReconciler).Reconcile\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/controller/reconcile.go:197\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1581591317.3893642,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"testcr-controller","request":"default/example-testcr","error":"did not receive playbook_on_stats event","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:88"}

Look that the logs has:

: nERROR! Syntax Error while loading YAML
And
,"error":"did not receive playbook_on_stats event","stacktrace":"github.com/go-logr/zapr.

And then, see that the reconcile still been retrigged forever by checking the Ansible logs too.
( kubectl logs deployment.apps/testoperator -c ansible -n default )

The offending line appears to be:

- msg: invalid sintax
    include_tasks: include_tasks:
                 ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/3510942875414458836//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! Syntax Error while loading YAML.
  mapping values are not allowed here

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 5, column 18, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- msg: invalid sintax
    include_tasks: include_tasks:
                 ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/2933568871211445515//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! Syntax Error while loading YAML.
  mapping values are not allowed here

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 5, column 18, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- msg: invalid sintax
    include_tasks: include_tasks:
                 ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/4324745483838182873//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! Syntax Error while loading YAML.
  mapping values are not allowed here

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 5, column 18, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- msg: invalid sintax
    include_tasks: include_tasks:
                 ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/2610529275472644968//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! Syntax Error while loading YAML.
  mapping values are not allowed here

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 5, column 18, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- msg: invalid sintax
    include_tasks: include_tasks:
                 ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/2703387474910584091//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! Syntax Error while loading YAML.
  mapping values are not allowed here

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 5, column 18, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- msg: invalid sintax
    include_tasks: include_tasks:
                 ^ here

4. And then, I also checked it with ManageStatus=False as well. (watches.yaml)

---
- version: v1alpha1
  group: glothriel.com
  kind: TestCR
  role: testcr
  manageStatus: False

c/c @joelanford @djzager

[tomsucho]

@camilamacedo86 what did the last Ansible line in PLAY RECAP looked like in your case? Was it same as mine?
PLAY RECAP ********************************************************************* localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
as you can see it looks like everything went fine, though it did not of course. When I tried to do a syntax error in other places it worked as well, like it would be treated just as a task fail not really general Ansible error. Maybe to be on same page you can try a block like I did:

    - name: Do something
      block:
      include_tasks: other-tasks.yml

Regarding:

You described that you are using the operator-sdk version 0.15.1. However, was the project scaffolded with this version and/or upgrade to use the 0.15.1 ansible image?

project was scaffolded with v0.13.0 I think and then I just bumped up the image tag to 0.15.1 on the operator. I guess that was not the right thing to do? Was going to ask about that in another question, but forgot in the end occupied with testing other stuff. How would I upgrade existing project then?

[camilamacedo86]

Hi @tomsucho,

I did the test with your snippet code and it also worked as my POC which means that the reconcile was re-trigged. Then, all lead me to believe that the root cause here is the issue solved at 0.14.0 and the fact that you did not upgrade your project.

So, could you please add here the content of your build/Dockerfile?

Following the logs with your snippet code

• From the container:

{"level":"info","ts":1581601032.9016247,"logger":"cmd","msg":"Install prometheus-operator in your cluster to create ServiceMonitor objects","Namespace":"default","error":"no ServiceMonitor registered with the API"}
{"level":"info","ts":1581601032.903348,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"testcr-controller","source":"kind source: glothriel.com/v1alpha1, Kind=TestCR"}
{"level":"info","ts":1581601032.9043787,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
{"level":"info","ts":1581601032.9045417,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1581601033.0064297,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"testcr-controller"}
{"level":"info","ts":1581601033.0066123,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"testcr-controller","worker count":1}
{"level":"error","ts":1581601035.620032,"logger":"runner","msg":"ansible-playbook 2.9.4\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nERROR! 'include_tasks' is not a valid attribute for a Block\r\n\r\nThe error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 4, column 3, but may\r\nbe elsewhere in the file depending on the exact syntax problem.\r\n\r\nThe offending line appears to be:\r\n\r\n\r\n- name: Do something\r\n  ^ here\r\n","job":"6129484611666145821","name":"example-testcr","namespace":"default","error":"exit status 4","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:223"}
{"level":"error","ts":1581601035.6203415,"logger":"reconciler","msg":"ansible-playbook 2.9.4\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nERROR! 'include_tasks' is not a valid attribute for a Block\r\n\r\nThe error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 4, column 3, but may\r\nbe elsewhere in the file depending on the exact syntax problem.\r\n\r\nThe offending line appears to be:\r\n\r\n\r\n- name: Do something\r\n  ^ here\r\n","job":"6129484611666145821","name":"example-testcr","namespace":"default","error":"did not receive playbook_on_stats event","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/controller.(*AnsibleOperatorReconciler).Reconcile\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/controller/reconcile.go:197\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1581601035.620502,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"testcr-controller","request":"default/example-testcr","error":"did not receive playbook_on_stats event","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tpkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1581601039.4677804,"logger":"runner","msg":"ansible-playbook 2.9.4\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nERROR! 'include_tasks' is not a valid attribute for a Block\r\n\r\nThe error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 4, column 3, but may\r\nbe elsewhere in the file depending on the exact syntax problem.\r\n\r\nThe offending line appears to be:\r\n\r\n\r\n- name: Do something\r\n  ^ here\r\n","job":"4037200794235010051","name":"example-testcr","namespace":"default","error":"exit status 4","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:223"}
{"level":"error","ts":1581601039.4680433,"logger":"reconciler","msg":"ansible-playbook 2.9.4\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]\r\nUsing /etc/ansible/ansible.cfg as config file\r\nERROR! 'include_tasks' is not a valid attribute for a Block\r\n\r\nThe error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 4, column 3, but may\r\nbe elsewhere in the file depending on the exact syntax problem.\r\n\r\nThe offending line appears to be:\r\n\r\n\r\n- name: Do something\r\n  ^ here\r\n","job":"4037200794235010051","name":"example-testcr","namespace":"default","error":"did not receive playbook_on_stats event","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/controller.(*AnsibleOperatorReconciler).Reconcile\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/controller/reconcile.go:197\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachiner

• From Ansible:

The offending line appears to be:

- name: Do something
  ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/6334824724549167320//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! 'include_tasks' is not a valid attribute for a Block

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 4, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Do something
  ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/605394647632969758//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! 'include_tasks' is not a valid attribute for a Block

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 4, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Do something
  ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/1443635317331776148//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! 'include_tasks' is not a valid attribute for a Block

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 4, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Do something
  ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/894385949183117216//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! 'include_tasks' is not a valid attribute for a Block

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 4, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Do something
  ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/2775422040480279449//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! 'include_tasks' is not a valid attribute for a Block

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 4, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Do something
  ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/4751997750760398084//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! 'include_tasks' is not a valid attribute for a Block

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 4, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Do something
  ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/7504504064263669287//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! 'include_tasks' is not a valid attribute for a Block

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 4, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Do something
  ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/1976235410884491574//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! 'include_tasks' is not a valid attribute for a Block

The error appears to be in '/opt/ansible/roles/testcr/tasks/main.yml': line 4, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: Do something
  ^ here
/tmp/ansible-operator/runner/glothriel.com/v1alpha1/TestCR/default/example-testcr/artifacts/3510942875414458836//stdout
ansible-playbook 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 15:04:54) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
ERROR! 'include_tasks' is not a valid attribute for a Block

• Regards your question: "How would I upgrade the existing project then?"

a) You will always need to upgrade the build/Dockerfile with the image of the release
b) You may need or would like to also upgrade scaffold files since they can be changed. Note that, sometimes they can indeed have some breaking change.

• Then, I would usually recommend you check the migration-guide and Changelog.

IMPORTANT: We had a daily to add the steps in the migration guide for the 0.14 and 0.15 release. However, you can check the open pr's for that. See: #2516 and #2519.

IHMO: The easy and more accurate path is to scaffold a new Ansible project and then, just compare the files to see what changed and be able to get the new changes.