ansible collection devenv and tests not passing

[rvalle]

Hi,

I am trying to setup my dev environment before contributing here.
One question outstanding is how to test these modules now that they are not part of ansible.
There is a relevant discussion here: ansible/ansible#60215

But I got tests to start working by setting my repo in a special directory, cloning at:

$HOME/src/ansible_collections/ngine_io/cloudstack

It seems to be required to move/rename the repository so the hierarchy from ansible_collections... is there.
I also create a virtual environent at /src/venv and install ansible (2.9) and cs.

And then tests will start running with:

ansible-test units --docker default

both unit/sanity tests are reporting a number of issues, and I wonder whether this is development environment issue or the code needs fixing.

For example:

when running unit tests:

WARNING: Rename "tests/units/" to "tests/unit/" to run unit tests.
WARNING: All targets skipped.

Or when running sanity tests:

ERROR: Found 24 validate-modules issue(s) which need to be resolved:
ERROR: tests/sanity/ignore-2.9.txt:1:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_disk_offering.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:2:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_firewall.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:3:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_host.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:4:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_instance.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:5:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_ip_address.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:6:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_iso.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:7:1: A100: Ignoring 'doc-required-mismatch' on 'plugins/modules/cs_loadbalancer_rule.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:8:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_loadbalancer_rule.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:9:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_loadbalancer_rule_member.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:10:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_network.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:11:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_network_acl_rule.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:12:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_network_offering.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:13:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_physical_network.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:14:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_portforward.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:15:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_project.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:16:1: A100: Ignoring 'invalid-ansiblemodule-schema' on 'plugins/modules/cs_resourcelimit.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:17:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_service_offering.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:18:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_storage_pool.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:19:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_template.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:20:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_vmsnapshot.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:21:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_volume.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:22:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_vpc.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:23:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_vpc_offering.py' is unnecessary
ERROR: tests/sanity/ignore-2.9.txt:24:1: A100: Ignoring 'parameter-list-no-elements' on 'plugins/modules/cs_vpn_customer_gateway.py' is unnecessary

I find it hard to believe that there are so many outstanding issues as Ansible's release process is very demanding and this collection was release as part as ansible:

I wonder if:

• Is this the right way to run the tests?
• Is ansible-2.9.xx good to launch the tests?
• Is "master" the latest code released?
• shall we start fixing these issues?

[felixfontein]

@rvalle the common reason for such problems is that ignore-2.9.txt is out of date since 2.9's ansible-test isn't used for sanity tests. @resmo are you running tests only with ansible-test from devel, or also from stable-2.10 and/or stable-2.9? (Since ignore-2.10.txt looks very similarly to ignore-2.9.txt, I guess it also hasn't been updated. Since ignore-2.11.txt isn't there I guess all actual problems have been resolved.)

[rvalle]

I am using ansible-base instead of ansible 2.9.
Sanity and Unit tests have some failures I guess related to the migration deadline #14
Integration tests seem to work as expected, with only some (ignored) failures, which I assume are WIP or similar.

[resmo]

this is fixed by #19