CR for fio_distributed can't start FIO servers when deployed through Ripsaw benchmark operator.

[abdasgupta]

I am trying to install Ripsaw benchmark operator https://github.com/cloud-bulldozer/ripsaw on Power 9 server. When I create the benchmark operator following this guide https://github.com/cloud-bulldozer/ripsaw/blob/master/docs/installation.md, the benchmark operator runs fine. As I am running the setup on Power 9, I had to build docker images for ansible operator https://github.com/operator-framework/operator-sdk and ripsaw benchmark operator https://github.com/cloud-bulldozer/ripsaw/blob/master/build/Dockerfile . But when I tried to deploy a CR using https://github.com/cloud-bulldozer/ripsaw/blob/master/docs/fio_distributed.md , I get the following error in ansible container of benchmark operator:

$ oc logs -n my-ripsaw benchmark-operator-584d477c4f-xnbdl -c ansible

WARN: This script is deprecated and will soon be removed
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/tmp/ansible-operator/runner/ripsaw.cloudbulldozer.io/v1alpha1/Benchmark/my-ripsaw/fio-benchmark/artifacts/6129484611666145821//stdout
ansible-playbook 2.9.6
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Oct 11 2019, 10:45:08) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /tmp/ansible-operator/runner/ripsaw.cloudbulldozer.io/v1alpha1/Benchmark/my-ripsaw/fio-benchmark/inventory/hosts as it did not pass its verify_file() method
script declined parsing /tmp/ansible-operator/runner/ripsaw.cloudbulldozer.io/v1alpha1/Benchmark/my-ripsaw/fio-benchmark/inventory/hosts as it did not pass its verify_file() method
auto declined parsing /tmp/ansible-operator/runner/ripsaw.cloudbulldozer.io/v1alpha1/Benchmark/my-ripsaw/fio-benchmark/inventory/hosts as it did not pass its verify_file() method
Set default localhost to localhost
Parsed /tmp/ansible-operator/runner/ripsaw.cloudbulldozer.io/v1alpha1/Benchmark/my-ripsaw/fio-benchmark/inventory/hosts inventory source with ini plugin
ERROR! couldn't resolve module/action 'k8s_status'. This often indicates a misspelling, missing collection, or incorrect module path.

The error appears to be in '/opt/ansible/playbook.yml': line 21, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


    - name: Setting the uuid for the benchmark
      ^ here
/tmp/ansible-operator/runner/ripsaw.cloudbulldozer.io/v1alpha1/Benchmark/my-ripsaw/fio-benchmark/artifacts/4037200794235010051//stdout

It seems k8s_status in ripsaw benchmark operator can't be parsed.

My environment:

$cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.1 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.1"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.1
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.1"

$lscpu
Architecture:        ppc64le
Byte Order:          Little Endian
CPU(s):              128
On-line CPU(s) list: 0-127
Thread(s) per core:  4
Core(s) per socket:  16
Socket(s):           2
NUMA node(s):        2
Model:               2.2 (pvr 004e 1202)
Model name:          POWER9, altivec supported
CPU max MHz:         3800.0000
CPU min MHz:         2166.0000
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            10240K
NUMA node0 CPU(s):   0-63
NUMA node8 CPU(s):   64-127

[camilamacedo86]

HI @abdasgupta,

Sorry, but it is not very clear for me what are you trying to achieve and/or the steps performed. However, shows that you are trying to creating Dockerfile images with the lastest released version of SDK, 0.16, when this https://github.com/cloud-bulldozer/ripsaw is using a very old version 0.8.1. See: https://github.com/cloud-bulldozer/ripsaw/blob/master/build/Dockerfile#L1

Then, the dependencies, and requirements as the default layout of the dockerfile are no longer the same as the features/libs provided by the image quay.io/operator-framework/ansible-operator regards the tag 0.8.1 and 0.16.

Is it make sense?

[abdasgupta]

Hi @camilamacedo86 Yeah, that's true.

See, my objective is to run Ripsaw benchmark operator (https://github.com/cloud-bulldozer/ripsaw) on openshift 4.3 cluster at first. Then I have to run fio-distributed workload (https://github.com/cloud-bulldozer/ripsaw/blob/master/docs/fio_distributed.md ) with the benchmark operator.

Benchmark operator docker image depends on ansible-operator docker image.

But there is no available docker image of ansible-operator and benchmark-operator on ppc64le architecture. So, I built both of the docker images. Though in the Dockerfile (https://github.com/cloud-bulldozer/ripsaw/blob/master/build/Dockerfile#L1), benchmark operator was using older version of ansible operator, but as I had to build the ansible operator for ppc64le architecture, I used the latest (0.16.0) one. So I was using latest released ansible operator image. My initial findings were based on the latest versions.

After reading your comment, I built the 0.8.0 version of ansible operator image and built the benchmark-operator image based on that. But, still I find the same error.

I hope, you understood the problem statement now.

[camilamacedo86]

HI @abdasgupta,

Sorry, but still hard to follow your explanation. May is because I do not know these operator projects and I have not its context. However, following some comments inline that I hope that helps you with.

Then I have to run fio-distributed workload (https://github.com/cloud-bulldozer/ripsaw/blob/master/docs/fio_distributed.md ) with the benchmark operator.

What do you mean with?

Also, you say:

So, I built both of the docker images. Though in the Dockerfile (https://github.com/cloud-bulldozer/ripsaw/blob/master/build/Dockerfile#L1), benchmark operator was using older version of ansible operator, but as I had to build the ansible operator for ppc64le architecture, I used the latest (0.16.0) one. So I was using latest released ansible operator image. My initial findings were based on the latest versions.

How you did this build? Did you clone SDK project, then check out the tag v0.8.1 and built an ansible SDK image for ppc64le and then, update the Dockerfile of this operator that is using the old one to use your SDK image pushed for your own registry?

The benchmark operator is using the version 0.8.1 it means that the structure of the files, what is supported or not and etc do not work as the version 0.16.0. From 0.8.1 to 0.16.0 has many differences and indeed may the playbooks/roles need to be changed to work with 0.16.

See the error : ERROR! couldn't resolve module/action 'k8s_status'.

Note that in 0.14.0 release: The Ansible module k8s_status was extracted and is now provided by the operator_sdk.util Ansible collection. See developer_guide for new usage. See the migration guide over this braking change

See, my objective is to run Ripsaw benchmark operator (https://github.com/cloud-bulldozer/ripsaw) on openshift 4.3 cluster at first.

Are you able to just test as standalone the operator https://github.com/cloud-bulldozer/ripsaw to ensure that is working as it is? If I understood properly, what do you think about first at all working to do all changes required to upgrade this project to use the latest version of SDK (0.16) and then, after you ensure that all is working with the latest (0.16) try to aggregate it to your project which is using the latest as you wish? Would that make sense?

[abdasgupta]

Hi @camilamacedo86 I did all of that as you mentioned
I did exactly this:

How you did this build? Did you clone SDK project, then check out the tag v0.8.1 and built an ansible SDK image for ppc64le and then, update the Dockerfile of this operator that is using the old one to use your SDK image pushed for your own registry?

And I tried with both 0.8.1 version and 0.16.0 version. My standalone operator was running fine. But as soon as I deployed the CR, it shoot error

[camilamacedo86]

If you are running the operator as standalone perfect well it would mean that when you apply the CR it also would work well. Then, it shows not be the case according to your last comment.

So, the error ERROR! couldn't resolve module/action 'k8s_status' lead to me think that it can be faced because you are using a structure of a project that is not compatible with the SDK version used/defined in the Dockerfile and/or has an error in the playbook/role which is called when the CR is applied.

I hope that this info can help you with.

[asmacdo]

In 0.14.0, the k8s_status module was moved from file generated by the sdk to an Ansible Collection.
https://github.com/operator-framework/operator-sdk/blob/master/doc/migration/version-upgrade-guide.md#add-operator_sdkutil-ansible-collection

Adding that collection to your roles should do the trick. Since the collection is not released as a part of the sdk, this should work with previous versions as well, providing you have an ansible minimum version of 2.9

[asmacdo]

I'm going to go ahead and close this issue, assuming that the above suggestion will work. Please reopen if not.