PR to include support for Skydive Node and Edge modules with Ansible

[justjais]

SUMMARY

Skydive is a network analyzer tool, which helps in analyzing SDNs and network and with this PR Ansible API and lookup modules are added to query Skydive objects using skydive python client.
Opened new PR closing the earlier skydive PR #50136

ISSUE TYPE

• New Module Pull Request

COMPONENT NAME

skydive

ADDITIONAL INFORMATION

[justjais]

cc: @safchain @pierrecregut please review the respective PR.

[ansibot]

@justjais this PR contains more than one new module.

Please submit only one new module per pull request. For a detailed explanation, please read the grouped modules documentation

click here for bot help

[ansibot]

The test ansible-test sanity --test import --python 2.6 [explain] failed with 3 errors:

lib/ansible/module_utils/network/skydive/api.py:86:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_node.py:94:0: ImportError: No module named q
test/runner/.tox/import/lib/ansible/module_utils/network/skydive/api.py:86:0: NameError: name 'WSClientDefaultProtocol' is not defined

The test ansible-test sanity --test import --python 2.7 [explain] failed with 5 errors:

lib/ansible/module_utils/network/skydive/api.py:86:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_capture.py:144:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_edge.py:119:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_node.py:94:0: ImportError: No module named q
test/runner/.tox/import/lib/ansible/module_utils/network/skydive/api.py:86:0: NameError: name 'WSClientDefaultProtocol' is not defined

The test ansible-test sanity --test import --python 3.5 [explain] failed with 5 errors:

lib/ansible/module_utils/network/skydive/api.py:86:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_capture.py:144:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_edge.py:119:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_node.py:94:0: ImportError: No module named 'q'
test/runner/.tox/import/lib/ansible/module_utils/network/skydive/api.py:86:0: NameError: name 'WSClientDefaultProtocol' is not defined

The test ansible-test sanity --test import --python 3.6 [explain] failed with 5 errors:

lib/ansible/module_utils/network/skydive/api.py:86:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_capture.py:144:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_edge.py:119:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_node.py:94:0: ModuleNotFoundError: No module named 'q'
test/runner/.tox/import/lib/ansible/module_utils/network/skydive/api.py:86:0: NameError: name 'WSClientDefaultProtocol' is not defined

The test ansible-test sanity --test import --python 3.7 [explain] failed with 5 errors:

lib/ansible/module_utils/network/skydive/api.py:86:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_capture.py:144:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_edge.py:119:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_node.py:94:0: ModuleNotFoundError: No module named 'q'
test/runner/.tox/import/lib/ansible/module_utils/network/skydive/api.py:86:0: NameError: name 'WSClientDefaultProtocol' is not defined

The test ansible-test sanity --test import --python 3.8 [explain] failed with 5 errors:

lib/ansible/module_utils/network/skydive/api.py:86:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_capture.py:144:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_edge.py:119:0: NameError: name 'WSClientDefaultProtocol' is not defined
lib/ansible/modules/network/skydive/skydive_node.py:94:0: ModuleNotFoundError: No module named 'q'
test/runner/.tox/import/lib/ansible/module_utils/network/skydive/api.py:86:0: NameError: name 'WSClientDefaultProtocol' is not defined

The test ansible-test sanity --test validate-modules [explain] failed with 4 errors:

lib/ansible/modules/network/skydive/skydive_capture.py:0:0: E321 Exception attempting to import module for argument_spec introspection, 'name 'WSClientDefaultProtocol' is not defined'
lib/ansible/modules/network/skydive/skydive_edge.py:0:0: E321 Exception attempting to import module for argument_spec introspection, 'name 'WSClientDefaultProtocol' is not defined'
lib/ansible/modules/network/skydive/skydive_node.py:0:0: E321 Exception attempting to import module for argument_spec introspection, 'No module named 'q''
lib/ansible/modules/network/skydive/skydive_node.py:94:0: E107 Imports should be directly below DOCUMENTATION/EXAMPLES/RETURN/ANSIBLE_METADATA.

click here for bot help

[justjais]

@safchain @pierrecregut ansible2.8 code freeze is scheduled in another week and I want the all modules with respect to Skydive to go in 2.8 release itself

[pierrecregut]

This is only a partial review. I will continue it when at least examples given in the documentation work.

[justjais]

@pierrecregut for me the documentation examples is working as expected, not sure why it's failing for you. Can we meet over BJ call if feasible, so that I can run it to you and also need to clarify a few changes that I did

[pierrecregut]

I test your code in a clean Ubuntu 18.04 VM. pip install skydive-client. cloned ansible. run hacking/env-setup
The test file is:

---
- name: Foo
  hosts: 127.0.0.1
  connection: local
  tasks:
    - name: create node
      skydive_node:
        name: TOR
        node_type: fabric
        seed: xxxx
        metadata:
          Brand: Cisco
        state: present
        provider:
          endpoint: 192.168.122.23:8082

I have added an epdb.serve() before the connect so that I can set a breakpoint:

Breakpoint 1 at /usr/local/lib/python2.7/dist-packages/trollius/selector_events.py:643
(Epdb) cont
> /usr/local/lib/python2.7/dist-packages/trollius/selector_events.py(643)__init__()
-> super(_SelectorSocketTransport, self).__init__(loop, sock, protocol, extra, server)
(Epdb) where
  /home/ubuntu/.ansible/tmp/ansible-tmp-1551868347.88-263470014915166/AnsiballZ_skydive_node.py(114)<module>()
-> _ansiballz_main()
  /home/ubuntu/.ansible/tmp/ansible-tmp-1551868347.88-263470014915166/AnsiballZ_skydive_node.py(106)_ansiballz_main()
-> invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  /home/ubuntu/.ansible/tmp/ansible-tmp-1551868347.88-263470014915166/AnsiballZ_skydive_node.py(49)invoke_module()
-> imp.load_module('__main__', mod, module, MOD_DESC)
  /tmp/ansible_skydive_node_payload_nFGiEC/__main__.py(126)<module>()
-> main()
  /tmp/ansible_skydive_node_payload_nFGiEC/__main__.py(121)main()
-> result = skydive_obj.run()
  /tmp/ansible_skydive_node_payload_nFGiEC/ansible_skydive_node_payload.zip/ansible/module_utils/network/skydive/api.py(300)run()
-> self.wsclient_object.connect()
  /usr/local/lib/python2.7/dist-packages/skydive/websocket/client.py(230)connect()
-> (transport, protocol) = self.loop.run_until_complete(coro)
  /usr/local/lib/python2.7/dist-packages/trollius/base_events.py(338)run_until_complete()
-> self.run_forever()
  /usr/local/lib/python2.7/dist-packages/trollius/base_events.py(309)run_forever()
-> self._run_once()
  /usr/local/lib/python2.7/dist-packages/trollius/base_events.py(1217)_run_once()
-> handle._run()
  /usr/local/lib/python2.7/dist-packages/trollius/events.py(136)_run()
-> self._callback(*self._args)
  /usr/local/lib/python2.7/dist-packages/trollius/tasks.py(259)_step()
-> result = coro.send(value)
  /usr/local/lib/python2.7/dist-packages/trollius/base_events.py(697)_create_connection_transport()
-> transport = self._make_socket_transport(sock, protocol, waiter)
  /usr/local/lib/python2.7/dist-packages/trollius/selector_events.py(80)_make_socket_transport()
-> extra, server)
> /usr/local/lib/python2.7/dist-packages/trollius/selector_events.py(643)__init__()
-> super(_SelectorSocketTransport, self).__init__(loop, sock, protocol, extra, server)
(Epdb) print(protocol)
<ansible.module_utils.network.skydive.api.skydive_inject_protocol object at 0x7fa1ecf492d0>

So as expected, the protocol object handled by trolius is the protocol object given to WSClient, it does not implement the protocol api as it extends object not the WSClientDefaultProtocol class. I understand why it fails, not why it would work in your setting.
If needed, I can try to attend a BlueJeans call (I guess that is the meaning of BJ). I work at Orange not Redhat. (firstname.lastname at orange dot com).

[safchain]

@pierrecregut @justjais Sorry I'm currently on vacation, adding @lebauce @nplanel @masco-enovance who can help

[justjais]

@pierrecregut ah got my mistake, earlier I had inherited WSClientDefaultProtocol in skydive_inject_protocol class and my development setup still had the same code, thus the module use case was passing for me. I had made the respective change to fix the ansible shippable error(#53112 (comment)), I am still not sure how to resolve the particular error coz I had tried to fix the NameError exception earlier but was not able to, hence had made the change which in-turn was failing with connection error.

I'll revert the same as earlier and shall try to fix the ansible sanity check error. Also, I'll write unit tests for these particular modules, so that such a mistake can be avoided in the future. Thanks

[pierrecregut]

A fast way to correct is to create a class in the init of skydive_wsclient:

class skydive_full_inject_protocol(skydive_inject_protocol, WSClientDefaultProtocol) : 
    pass

and using this class as protocol argument. Still you need to make two other modifications : you must call
self.sendWSMessage and self.stop and not consider them as class function as in WSClientDefaultProtocol.sendWsmessage(msg) or pass self as first argument but this look weird.
You could rename skydive_inject_protocol in skydive_inject_protocol_mixin to insist on the fact it is not complete. With those changes, I am able to create nodes and WSClientDefaultProtocol will not be required at the module toplevel.

[pierrecregut]

or (probably more pythonic):

protocol=type('skydive_full_inject_protocol',(skydive_inject_protocol, WSClientDefaultProtocol), dict())

[justjais]

@pierrecregut I have updated the PR fixing the review comments, kindly check and lemme me know if it's ok to merge. Also, thanks for guiding me in fixing the shippable error.

I shall raise another PR covering the unit tests for all ansible skydive modules.