Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testinfra conflicts with pytest-ansible #58

Open
ISF opened this issue Jan 27, 2016 · 8 comments
Open

testinfra conflicts with pytest-ansible #58

ISF opened this issue Jan 27, 2016 · 8 comments

Comments

@ISF
Copy link

ISF commented Jan 27, 2016

When both testinfra and pytest-ansible are installed, testinfra fails with the following output:

$ testinfra --connection=ansible --sudo --ansible-inventory=inventory test_proxy.py
Traceback (most recent call last):
  File "/home/ivan/cde/devops/devops-env/bin/testinfra", line 11, in <module>
    sys.exit(main())
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/testinfra/main.py", line 100, in main
    return pytest.main()
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 38, in main
    config = _prepareconfig(args, plugins)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 117, in _prepareconfig
    pluginmanager=pluginmanager, args=args)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 595, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 249, in _wrapped_call
    wrap_controller.send(call_outcome)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/helpconfig.py", line 28, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
    _reraise(*ex)  # noqa
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 264, in __init__
    self.result = func()
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596, in execute
    res = hook_impl.function(*args)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 852, in pytest_cmdline_parse
    self.parse(args)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 957, in parse
    self._preparse(args)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 922, in _preparse
    self.known_args_namespace = ns = self._parser.parse_known_args(args)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 490, in parse_known_args
    return self.parse_known_and_unknown_args(args)[0]
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 496, in parse_known_and_unknown_args
    optparser = self._getparser()
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 475, in _getparser
    arggroup.add_argument(*n, **a)
  File "/usr/lib64/python2.7/argparse.py", line 1308, in add_argument
    return self._add_action(action)
  File "/usr/lib64/python2.7/argparse.py", line 1509, in _add_action
    action = super(_ArgumentGroup, self)._add_action(action)
  File "/usr/lib64/python2.7/argparse.py", line 1322, in _add_action
    self._check_conflict(action)
  File "/usr/lib64/python2.7/argparse.py", line 1460, in _check_conflict
    conflict_handler(action, confl_optionals)
  File "/usr/lib64/python2.7/argparse.py", line 1467, in _handle_conflict_error
    raise ArgumentError(action, message % conflict_string)
argparse.ArgumentError: argument --ansible-inventory: conflicting option string(s): --ansible-inventory

Removing --ansible-inventory from the command line does not change the output. Uninstalling pytest-ansible and running testinfra fixes the issue, but leaves my tests which use it broken.

Contents of test_proxy.py:

# Proxy servers must have nginx installed, running and enabled on boot
def test_nginx(Package, Service):
    assert Package('nginx').is_installed
    assert Service('nginx').is_running
    assert Service('nginx').is_enabled

Environment:
Python 2.7.11
pip 7.1.2
ansible 1.9.4
configured module search path = None
virtualenv 13.1.2
testinfra 1.0.1
pytest-ansible 1.3.1
pytest 2.8.4

@philpep
Copy link
Contributor

philpep commented Jan 27, 2016

(testinfra command is an alias to py.test)

Both pytest-ansible and testinfra are pytest plugins that define the same --ansible-inventory option (pytest options scope is global). I'll look if there is a way to handle this in pytest.

BTW, I think you can easily rewrite your pytest-ansible tests with testinfra with the Ansible module: https://testinfra.readthedocs.org/en/latest/modules.html#ansible ;)

@ISF
Copy link
Author

ISF commented Jan 27, 2016

I'm using pytest-ansible to test custom modules and/or plugins, not the infrastructure itself (so it's not viable to just substitute it with testinfra). With testinfra I plan to rewrite all of my playbook's/role's unit tests (which are currently just special ansible playbooks) and expand infrastructure tests, if we can get it to work alongside pytest-ansible.

Let me know if I can help with anything.

@philpep
Copy link
Contributor

philpep commented Jan 27, 2016

As a workaround you can add -p no:pytest-ansible when running testinfra tests and -p no:testinfra when running pytest-ansible tests.

@decentral1se
Copy link
Collaborator

Is this still a valid issue? #58 (comment) seems like a fine solution. That pytest-ansible package hasn't seen a release for quite some time ... wondering if this is relevant anymore.

@ssbarnea
Copy link
Member

pytest-ansible had 5 releases in 2023.

@clickthisnick
Copy link

clickthisnick commented Oct 13, 2023

I'm having a similar issue for a different arg - any ideas what I need to do?

python3.11 -m venv venv
source venv/bin/activate
pip install pytest
pip install pytest-ansible
pip install pytest-testinra
pip install ansible
pip list

Package          Version
---------------- -------
ansible          8.4.0
ansible-core     2.15.4
cffi             1.16.0
coverage         7.3.2
cryptography     41.0.4
iniconfig        2.0.0
Jinja2           3.1.2
MarkupSafe       2.1.3
packaging        23.2
pip              23.2.1
pluggy           1.3.0
pycparser        2.21
pytest           7.4.2
pytest-ansible   4.1.0
pytest-testinfra 9.0.0
PyYAML           6.0.1
resolvelib       1.0.1
setuptools       68.1.2
snakeviz         2.2.0
tornado          6.3.3
pytest --collect-only
> argparse.ArgumentError: argument --connection: conflicting option string: --connection
 pytest --collect-only -p no:pytest-ansible
> works
pytest --collect-only -p no:testinfra -p no:pytest-testinfra
> argparse.ArgumentError: argument --connection: conflicting option string: --connection

EDIT: Solved this via this stackoverflow answer: https://stackoverflow.com/a/65356704

Running python -c "import pkg_resources; print(' '.join('-p no:' + ' '.join(dist.get_entry_map(group='pytest11').keys()) for dist in pkg_resources.working_set if dist.get_entry_map(group='pytest11')))" gives you all the pytest plugin you have that you can disable.

pytest --collect-only -p no:pytest11.testinfra works

@auburus
Copy link

auburus commented Dec 4, 2023

I've hit this issue too, and thanks to all the people that contributed to this thread I was able to achieve a solution, sharing it in case it is useful to someone else.

My problem

Using pytest-molecule for testing ansible roles in a monorepo, where each role is tested using molecule + testinfra.

This is an example layout

.
├── collections
│   └── ansible_collections
│       └── auburus
│           └── test_collection
│               ├── playbook.yml
│               └── roles
│                   └── test_role
│                       ├── molecule
│                       │   └── default
│                       │       ├── converge.yml
│                       │       ├── molecule.yml
│                       │       └── tests
│                       │           └── test_role.py
│                       └── tasks
│                           └── main.yml
├── poetry.lock
└── pyproject.toml

In September pytest-molecule was archived, and the path forward is pytest-ansible, so the goal is to move to that one. But we couldn't move since pytest-ansible and pytest-testinfra got the conflict described in this issue.

Solution

Get each molecule test to ignore pytest ansible.

# molecule.yaml
...
verifier:
  name: testinfra
  options:
    # pytest-ansible conflicts with --connection
    p: "no:pytest-ansible"

    # avoid loading global pyproject.toml options, so we don't
    # load the setting that says "disable testinfra"
    c: "."

At the top level, ignore pytest-testinfra

# pyproject.toml
[tool.pytest.ini_options]
norecursedirs="molecule"

# testinfra and pytest-ansible conflict as pytest
# plugins. We disable testinfra when running "all" tests,
# and disable `pytest-ansible` when running each molecule test
addopts = "-p no:pytest11.testinfra"

Finally follow this guide so all molecule tests are loaded when running pytest.

# tests/test_molecule.py
"""Tests for molecule scenarios."""
from __future__ import absolute_import, division, print_function

from pytest_ansible.molecule import MoleculeScenario


def test_integration(molecule_scenario: MoleculeScenario) -> None:
    """Run molecule for each scenario.

    :param molecule_scenario: The molecule scenario object
    """
    proc = molecule_scenario.test()
    assert proc.returncode == 0

Results

  • pytest can run all molecule tests from top level ✅
$ pytest --co
============================= test session starts ============================= 
platform linux -- Python 3.11.5, pytest-7.4.3, pluggy-1.3.0
ansible: 2.16.0
rootdir: /tmp/example-pytest-ansible-testinfra-conflict
configfile: pyproject.toml
plugins: ansible-4.1.1
collected 1 item

<Module tests/test_molecule.py>
  <Function test_integration[test-default]>

=============================  1 test collected in 0.04s ============================= 
  • each role can be tested with molecule test
$ molecule test
...
WARNING  Skipping, side effect playbook not configured.                                                                                                                                       
INFO     Running default > verify                                                                                                                                                             
INFO     Executing Testinfra tests found in /tmp/example-pytest-ansible-testinfra-conflict/collections/ansible_collections/auburus/test/roles/test/molecule/default/tests/...                 
============================= test session starts ==============================                                                                                                              
platform linux -- Python 3.11.5, pytest-7.4.3, pluggy-1.3.0                                                                                                                                   
rootdir: /tmp/example-pytest-ansible-testinfra-conflict/collections/ansible_collections/auburus/test/roles/test/molecule                                                                      
configfile: default                                                                                                                                                                           
plugins: testinfra-10.0.0                                                                                                                                                                     
collected 1 item                                                                                                                                                                              
                                                                                                                                                                                              
tests/test_role.py .                                                     [100%]                                                                                                               
                                                                                                                                                                                              
============================== 1 passed in 0.70s ===============================                                                                                                              
INFO     Verifier completed successfully. 
...

Edit: I've published this to a repo in case more details are needed

@davedittrich
Copy link

I ran into this over a year ago, but didn't have enough time to produce two documented PRs for the concurrent changes to both repos. I think pytest has a better way to deal with the conflict, which isn't at all easy to debug due to the way exception handling occurs when molecule calls pytest which dynamically loads plugins.

The TL;DR, both modules try to use the same command line options. My solution was to preface each option with a string specific to each module to deconflict. You can see the changes here:

ansible/pytest-ansible@main...davedittrich:pytest-ansible:develop

main...davedittrich:pytest-testinfra:develop

If someone else has time to resolve this, that would be great. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants