Undeclared depedency when using OpenStack provider

[gnuoy]

Commit 4cccbdc introduces a dependency on python-novaclient and python-keystoneclient but zaza itself does not depend on those packages. This breaks charms like grafana which are using zaza to test on openstack but do not include these OpenStack client modules in their dependencies.

Failure from grafana test run:

2021-06-30 08:51:18 [INFO] Using config /home/ubuntu/.local/share/juju/clouds.yaml                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                            
  File "/home/ubuntu/work/branches/grafana/.build/grafana/.tox/func/bin/functest-run-suite", line 8, in <module>                                                                                              
    sys.exit(main())                                                                                                                                                                                          
  File "/home/ubuntu/work/branches/grafana/.build/grafana/.tox/func/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 308, in main                                                  
    func_test_runner(                                                                                                                                                                                         
  File "/home/ubuntu/work/branches/grafana/.build/grafana/.tox/func/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 242, in func_test_runner                                      
    run_env_deployment(env_deployment, keep_model=preserve_model,                                                                                                                                             
  File "/home/ubuntu/work/branches/grafana/.build/grafana/.tox/func/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 166, in run_env_deployment                                    
    destroy.destroy(model_name)                                                                                                                                                                               
  File "/home/ubuntu/work/branches/grafana/.build/grafana/.tox/func/lib/python3.8/site-packages/zaza/charm_lifecycle/destroy.py", line 42, in destroy                                                         
    import zaza.utilities.openstack_provider as op                                                                                                                                                            
  File "/home/ubuntu/work/branches/grafana/.build/grafana/.tox/func/lib/python3.8/site-packages/zaza/utilities/openstack_provider.py", line 21, in <module>                                                   
    from keystoneauth1 import session                                                                                                                                                                         
ModuleNotFoundError: No module named 'keystoneauth1'    

[rgildein]

Could we solve this issue?

[ajkavanagh]

Could we solve this issue?

Yes, it can be solved; just need to make the module that imports openstack.py only do it on demand; essentially, make it a lazily loaded module.

[rgildein]

Oh, so python-novaclient and keystoneauth1 won't be added to zaza dependencies?

[ajkavanagh]

zaza only "needs" these dependencies when running on an OpenStack provider, as part of a clean-up, which it shouldn't really need to do. However, the world is a messy place. It would not be particularly good to make zaza have actual OpenStack dependencies as this would mean that these modules would have to be installed when testing on just K8s, or AWS or even just locally.

Ideally, the openstack provider and all associated functionality would be in a namespaced, separate, submodule that could be 'plugged in' as required. That module would have the OpenStack python modules as dependencies, so that when the OpenStack provider module is required for a particular test, its dependencies would already be available, and zaza would have no particular knowledge about OpenStack, MaaS, etc.

However, we are not there, yet. Thus, to solve this tactically, the best way would be to just make the imports lazy on demand so that if OpenStack is never used as a provider then the modules are never attempted to be imported and this error won't occur.

[rgildein]

I see your point here and I agree that those dependencies would not make sense for k8s or AWS, ... .

The 'plug in' should be nice, but also a bit complicated.

What do you think about using optional dependencies in setup.py? This approach will be much cleaner than adding these dependencies to tox.ini without commenting why they are there. Related bug.

setup.py

...
openstack_require = [
    "python-openstackclient"
]
...
    extras_require={
        'testing': tests_require,
        'openstack': openstack_require,
    },
...

tox.ini (in charm project)

...
[testenv:func]
deps =
  git+https://github.com/openstack-charmers/zaza.git#egg=zaza[openstack]
...