Where to report issues with newer pytest (?) versions breaking the tests?

[johanneskastl]

Hi all,

I am helping out with the molecule package for openSUSE. I noticed the package failing to pass the tests recently. There are multiple errors due to Duplicate test name and has an id that does not match our safe pattern.

My current guess is that this is because of some new version of pytest or on of its plugins.

I wanted to report this, in case this needs to be fixed upstream. I had a look at the bug report template, but most of the questions do not make sense in this case, hence my question here, on how to report this?

[   13s] ============================= test session starts ==============================
[   13s] platform linux -- Python 3.11.5, pytest-7.4.3, pluggy-1.3.0 -- /usr/bin/python3.11
[   13s] cachedir: .pytest_cache
[   13s] rootdir: /home/abuild/rpmbuild/BUILD/molecule-6.0.2
[   13s] configfile: pyproject.toml
[   13s] testpaths: src/molecule/test/
[   13s] plugins: plus-0.0.0, mock-3.12.0, xdist-3.3.1
[   14s] collecting ... ERROR: Failed run due to following issues being identified:
[   14s] Duplicate test name 'test_init_calls_setup', found at src/molecule/test/a_unit/command/test_base.py:93 and src/molecule/test/a_unit/test_scenario.py:81
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/test_cleanup.py:42 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/test_converge.py:35 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/test_create.py:36 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/test_dependency.py:29 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/test_destroy.py:41 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/test_idempotence.py:44 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/test_list.py:25 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/test_login.py:35 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/test_prepare.py:37 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute_skips_when_playbook_not_configured', found at src/molecule/test/a_unit/command/test_prepare.py:73 and src/molecule/test/a_unit/command/test_cleanup.py:65
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/test_side_effect.py:47 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute_skips_when_playbook_not_configured', found at src/molecule/test/a_unit/command/test_side_effect.py:71 and src/molecule/test/a_unit/command/test_cleanup.py:65
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/test_syntax.py:35 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/test_verify.py:29 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/command/init/test_scenario.py:43 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_config_private_member', found at src/molecule/test/a_unit/dependency/test_shell.py:51 and src/molecule/test/a_unit/command/test_base.py:89
[   14s] Duplicate test name 'test_name_property', found at src/molecule/test/a_unit/dependency/test_shell.py:68 and src/molecule/test/a_unit/test_scenario.py:85
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/dependency/test_shell.py:97 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_config_private_member', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:67 and src/molecule/test/a_unit/command/test_base.py:89
[   14s] Duplicate test name 'test_default_options_property', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:71 and src/molecule/test/a_unit/dependency/test_shell.py:55
[   14s] Duplicate test name 'test_default_env_property', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:77 and src/molecule/test/a_unit/dependency/test_shell.py:61
[   14s] Duplicate test name 'test_name_property', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:86 and src/molecule/test/a_unit/test_scenario.py:85
[   14s] Duplicate test name 'test_enabled_property', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:90 and src/molecule/test/a_unit/dependency/test_shell.py:73
[   14s] Duplicate test name 'test_options_property', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:94 and src/molecule/test/a_unit/dependency/test_shell.py:77
[   14s] Duplicate test name 'test_options_property_handles_cli_args', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:106 and src/molecule/test/a_unit/dependency/test_shell.py:84
[   14s] Duplicate test name 'test_env_property', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:119 and src/molecule/test/a_unit/dependency/test_shell.py:92
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:140 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute_does_not_execute_when_disabled', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:159 and src/molecule/test/a_unit/dependency/test_shell.py:111
[   14s] Duplicate test name 'test_execute_bakes', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:188 and src/molecule/test/a_unit/dependency/test_shell.py:125
[   14s] Duplicate test name 'test_setup', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:211 and src/molecule/test/a_unit/command/test_base.py:97
[   14s] Duplicate test name 'test_config_private_member', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:66 and src/molecule/test/a_unit/command/test_base.py:89
[   14s] Duplicate test name 'test_default_options_property', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:70 and src/molecule/test/a_unit/dependency/test_shell.py:55
[   14s] Duplicate test name 'test_default_env_property', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:76 and src/molecule/test/a_unit/dependency/test_shell.py:61
[   14s] Duplicate test name 'test_name_property', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:85 and src/molecule/test/a_unit/test_scenario.py:85
[   14s] Duplicate test name 'test_enabled_property', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:89 and src/molecule/test/a_unit/dependency/test_shell.py:73
[   14s] Duplicate test name 'test_options_property', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:93 and src/molecule/test/a_unit/dependency/test_shell.py:77
[   14s] Duplicate test name 'test_options_property_handles_cli_args', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:105 and src/molecule/test/a_unit/dependency/test_shell.py:84
[   14s] Duplicate test name 'test_env_property', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:118 and src/molecule/test/a_unit/dependency/test_shell.py:92
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:138 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute_does_not_execute_when_disabled', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:157 and src/molecule/test/a_unit/dependency/test_shell.py:111
[   14s] Duplicate test name 'test_execute_does_not_execute_when_no_requirements_file', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:171 and src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:173
[   14s] Duplicate test name 'test_execute_bakes', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:186 and src/molecule/test/a_unit/dependency/test_shell.py:125
[   14s] Duplicate test name 'test_role_file', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:210 and src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:215
[   14s] Duplicate test name 'test_has_requirements_file', found at src/molecule/test/a_unit/dependency/ansible_galaxy/test_roles.py:214 and src/molecule/test/a_unit/dependency/ansible_galaxy/test_collections.py:219
[   14s] Duplicate test name 'test_config_private_member', found at src/molecule/test/a_unit/driver/test_delegated.py:59 and src/molecule/test/a_unit/command/test_base.py:89
[   14s] Duplicate test name 'test_name_property', found at src/molecule/test/a_unit/driver/test_delegated.py:70 and src/molecule/test/a_unit/test_scenario.py:85
[   14s] Duplicate test name 'test_options_property', found at src/molecule/test/a_unit/driver/test_delegated.py:74 and src/molecule/test/a_unit/dependency/test_shell.py:77
[   14s] Duplicate test name 'test_created', found at src/molecule/test/a_unit/driver/test_delegated.py:333 and src/molecule/test/a_unit/test_state.py:42
[   14s] Duplicate test name 'test_driver', found at src/molecule/test/a_unit/model/v2/test_driver_section.py:38 and src/molecule/test/a_unit/test_state.py:46
[   14s] Duplicate test name 'test_config_private_member', found at src/molecule/test/a_unit/provisioner/test_ansible.py:94 and src/molecule/test/a_unit/command/test_base.py:89
[   14s] Duplicate test name 'test_default_options_property', found at src/molecule/test/a_unit/provisioner/test_ansible.py:119 and src/molecule/test/a_unit/dependency/test_shell.py:55
[   14s] Duplicate test name 'test_default_env_property', found at src/molecule/test/a_unit/provisioner/test_ansible.py:123 and src/molecule/test/a_unit/dependency/test_shell.py:61
[   14s] Duplicate test name 'test_name_property', found at src/molecule/test/a_unit/provisioner/test_ansible.py:137 and src/molecule/test/a_unit/test_scenario.py:85
[   14s] Duplicate test name 'test_options_property', found at src/molecule/test/a_unit/provisioner/test_ansible.py:167 and src/molecule/test/a_unit/dependency/test_shell.py:77
[   14s] Duplicate test name 'test_options_property_handles_cli_args', found at src/molecule/test/a_unit/provisioner/test_ansible.py:185 and src/molecule/test/a_unit/dependency/test_shell.py:84
[   14s] Duplicate test name 'test_env_property', found at src/molecule/test/a_unit/provisioner/test_ansible.py:197 and src/molecule/test/a_unit/dependency/test_shell.py:92
[   14s] Duplicate test name 'test_inventory_directory_property', found at src/molecule/test/a_unit/provisioner/test_ansible.py:284 and src/molecule/test/a_unit/test_scenario.py:97
[   14s] Duplicate test name 'test_directory_property', found at src/molecule/test/a_unit/provisioner/test_ansible.py:308 and src/molecule/test/a_unit/test_scenario.py:89
[   14s] Duplicate test name 'test_write_config', found at src/molecule/test/a_unit/provisioner/test_ansible.py:459 and src/molecule/test/a_unit/test_config.py:355
[   14s] Duplicate test name 'test_config_private_member', found at src/molecule/test/a_unit/provisioner/test_ansible_playbook.py:95 and src/molecule/test/a_unit/command/test_base.py:89
[   14s] Duplicate test name 'test_execute_bakes', found at src/molecule/test/a_unit/provisioner/test_ansible_playbook.py:196 and src/molecule/test/a_unit/dependency/test_shell.py:125
[   14s] Duplicate test name 'test_config_private_member', found at src/molecule/test/a_unit/verifier/test_ansible.py:33 and src/molecule/test/a_unit/command/test_base.py:89
[   14s] Duplicate test name 'test_default_options_property', found at src/molecule/test/a_unit/verifier/test_ansible.py:37 and src/molecule/test/a_unit/dependency/test_shell.py:55
[   14s] Duplicate test name 'test_default_env_property', found at src/molecule/test/a_unit/verifier/test_ansible.py:41 and src/molecule/test/a_unit/dependency/test_shell.py:61
[   14s] Duplicate test name 'test_env_property', found at src/molecule/test/a_unit/verifier/test_ansible.py:48 and src/molecule/test/a_unit/dependency/test_shell.py:92
[   14s] Duplicate test name 'test_name_property', found at src/molecule/test/a_unit/verifier/test_ansible.py:53 and src/molecule/test/a_unit/test_scenario.py:85
[   14s] Duplicate test name 'test_enabled_property', found at src/molecule/test/a_unit/verifier/test_ansible.py:57 and src/molecule/test/a_unit/dependency/test_shell.py:73
[   14s] Duplicate test name 'test_directory_property', found at src/molecule/test/a_unit/verifier/test_ansible.py:61 and src/molecule/test/a_unit/test_scenario.py:89
[   14s] Duplicate test name 'test_options_property', found at src/molecule/test/a_unit/verifier/test_ansible.py:67 and src/molecule/test/a_unit/dependency/test_shell.py:77
[   14s] Duplicate test name 'test_options_property_handles_cli_args', found at src/molecule/test/a_unit/verifier/test_ansible.py:74 and src/molecule/test/a_unit/dependency/test_shell.py:84
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/verifier/test_ansible.py:82 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_config_private_member', found at src/molecule/test/a_unit/verifier/test_testinfra.py:66 and src/molecule/test/a_unit/command/test_base.py:89
[   14s] Duplicate test name 'test_default_options_property', found at src/molecule/test/a_unit/verifier/test_testinfra.py:70 and src/molecule/test/a_unit/dependency/test_shell.py:55
[   14s] Duplicate test name 'test_default_env_property', found at src/molecule/test/a_unit/verifier/test_testinfra.py:109 and src/molecule/test/a_unit/dependency/test_shell.py:61
[   14s] Duplicate test name 'test_env_property', found at src/molecule/test/a_unit/verifier/test_testinfra.py:135 and src/molecule/test/a_unit/dependency/test_shell.py:92
[   14s] Duplicate test name 'test_name_property', found at src/molecule/test/a_unit/verifier/test_testinfra.py:144 and src/molecule/test/a_unit/test_scenario.py:85
[   14s] Duplicate test name 'test_enabled_property', found at src/molecule/test/a_unit/verifier/test_testinfra.py:148 and src/molecule/test/a_unit/dependency/test_shell.py:73
[   14s] Duplicate test name 'test_directory_property', found at src/molecule/test/a_unit/verifier/test_testinfra.py:152 and src/molecule/test/a_unit/test_scenario.py:89
[   14s] Duplicate test name 'test_options_property', found at src/molecule/test/a_unit/verifier/test_testinfra.py:172 and src/molecule/test/a_unit/dependency/test_shell.py:77
[   14s] Duplicate test name 'test_options_property_handles_cli_args', found at src/molecule/test/a_unit/verifier/test_testinfra.py:186 and src/molecule/test/a_unit/dependency/test_shell.py:84
[   14s] Duplicate test name 'test_bake', found at src/molecule/test/a_unit/verifier/test_testinfra.py:202 and src/molecule/test/a_unit/provisioner/test_ansible_playbook.py:99
[   14s] Duplicate test name 'test_execute', found at src/molecule/test/a_unit/verifier/test_testinfra.py:224 and src/molecule/test/a_unit/command/test_check.py:35
[   14s] Duplicate test name 'test_execute_does_not_execute', found at src/molecule/test/a_unit/verifier/test_testinfra.py:241 and src/molecule/test/a_unit/verifier/test_ansible.py:94
[   14s] Duplicate test name 'test_execute_bakes', found at src/molecule/test/a_unit/verifier/test_testinfra.py:268 and src/molecule/test/a_unit/dependency/test_shell.py:125
[   14s] Test <Function test_command_check[driver/delegated-default-default]> has an id that does not match our safe pattern '^[\w_\-\.]+$' for use with a terminal.
[   14s] Test <Function test_command_cleanup[driver/delegated-default-default]> has an id that does not match our safe pattern '^[\w_\-\.]+$' for use with a terminal.
[   14s] Test <Function test_command_converge[driver/delegated-default-default]> has an id that does not match our safe pattern '^[\w_\-\.]+$' for use with a terminal.
[   14s] Test <Function test_command_create[driver/delegated-default-default]> has an id that does not match our safe pattern '^[\w_\-\.]+$' for use with a terminal.
[   14s] Test <Function test_command_destroy[driver/delegated-default-default]> has an id that does not match our safe pattern '^[\w_\-\.]+$' for use with a terminal.
[   14s] Test <Function test_command_idempotence[driver/delegated-default-default]> has an id that does not match our safe pattern '^[\w_\-\.]+$' for use with a terminal.
[   14s] <Function test_command_list_with_format_plain[driver/delegated-default-instance        default ansible default]> has an id that looks above 60 characters.
[   14s] Test <Function test_command_prepare[driver/delegated-default-default]> has an id that does not match our safe pattern '^[\w_\-\.]+$' for use with a terminal.
[   14s] Test <Function test_command_side_effect[driver/delegated-default-default]> has an id that does not match our safe pattern '^[\w_\-\.]+$' for use with a terminal.
[   14s] Test <Function test_command_syntax[driver/delegated-default-default]> has an id that does not match our safe pattern '^[\w_\-\.]+$' for use with a terminal.
[   14s] Test <Function test_command_test[driver/delegated-default-default]> has an id that does not match our safe pattern '^[\w_\-\.]+$' for use with a terminal.
[   14s] Test <Function test_command_test_with_platform_name[driver/delegated-default-default-instance]> has an id that does not match our safe pattern '^[\w_\-\.]+$' for use with a terminal.
[   14s] <Function test_command_test_with_role_name_check_equals_to_1[driver/delegated_invalid_role_name_with_role_name_check_equals_to_1-default-default]> has an id that looks above 60 characters.
[   14s] Test <Function test_command_verify[driver/delegated-default-default]> has an id that does not match our safe pattern '^[\w_\-\.]+$' for use with a terminal.
[   14s] 
[   14s] collected 458 items
[   14s] 
[   14s] ============================ no tests ran in 0.92s =============================

[johanneskastl]

For the record, the fix was found in commit 4ab4f1c, but the general question is still open...

[zhan9san]

@johanneskastl

The error is reported by this package, pytest-plus.

So you can report the issue in pytest-plus

Maybe it's a feature, not bug

[johanneskastl]

No, the error is caused by changes in pytest-plus, but the fix needs to be in molecule. As stated below, it has been fixed already.

My question is where to report such findings? Is a discussion like this one appropriate? Or does it need a full-fledged bug report (with lots of questions that do not fit the case at all)?

[zhan9san]

I had a look at the bug report template, but most of the questions do not make sense in this case, hence my question here, on how to report this?

Both are fine. But I think a bug report is more appropriate. For this specific case, it blocks the CI jobs, so it can be described in What happened

[johanneskastl]

Thanks! Is is fine if I omit answers to questions not applying (or rather, enter something like "not applicable" as I need to enter something...)?

[zhan9san]

Is is fine if I omit answers to questions not applying (or rather, enter something like "not applicable" as I need to enter something...)?

I cannot get your point. Could you please explain it in more detail?

[johanneskastl]

Sorry, my statement was unclear. I meant the questions which do not really make sense in case of automated builds on the openSUSE Build service. Like "I was able to reproduce the issue on a different machine" or similar.

That was the main reason not directly opening up a bug report for this in the first place....