Add engine relaying libvirt events #46461

cbosdo · 2018-03-09T16:42:41Z

What does this PR do?

Libvirt API offers clients to register callbacks for various events.
libvirt_events engine will listen on a libvirt URI (local or remote)
for events and send them to the salt event bus.

What issues does this PR fix or reference?

Nothing that is in the issue tracker to my knowledge

Tests written?

~~No: those would require a running libvirt daemon~~
Yes

Commits signed with GPG?

Yes

cbosdo · 2018-03-12T08:18:01Z

I can't change the callback signature: codeclimate would need to be silenced for those functions that have more than 4 arguments.

I don't really know what to do with the docstrings of the callbacks: either pylint is complaining not having them or code climate complains because their are too similar. Honestly, considering how useless those would be, I'ld just let pylint complain.

Codeclimate complains that the file is too long... but is it possible at all to have the code of an engine split into several files? Ideally I would separate the callbacks from the rest of the engine, not sure if that is possible / wanted.

isbm

Can we also rename callbacks from dom_<something> to domain_...? Because this is confusing with DOM (document object model). The callbacks can be also "hidden", i.e. starts from the underscore: def _something.

isbm · 2018-03-12T11:16:48Z

salt/engines/libvirt_events.py

+    ('pmsuspended', ("memory",
+                     "disk")),
+    ('crashed', ("panicked"))
+)


I would refactor all this into a dictionary. The def nth is not needed anywhere, because the dictionary has .get(...) function.

Found a way to get rid of these lists by introspecting libvirt module.

isbm · 2018-03-12T11:20:39Z

@cbosdo Unit tests do not need to have libvirt installed, only integration tests. So you can just mock.MagicMock all the output from the libvirt part and unit-test everything, since we do not really need to test the libvirt itself.

isbm · 2018-03-14T09:14:04Z

tests/unit/engines/test_libvirt_events.py

+
+                libvirt_events.start(uri, prefix)
+
+                assert call.call_list() != []


I am not sure you're using this correctly: https://docs.python.org/3/library/unittest.mock.html#unittest.mock.call

cbosdo · 2018-03-14T15:57:28Z

unit tests added now

rallytime · 2018-03-15T16:29:17Z

Thanks @cbosdo! There's some lint failures here: https://jenkins.saltstack.com/job/PR/job/salt-pr-lint-n/20144/violations/

cbosdo · 2018-03-15T16:40:24Z

Fixed the pylint errors now

rallytime

@cbosdo I have a couple of comments here that need to be addressed. Thank you so much for writing the tests, too!

rallytime · 2018-03-15T18:45:10Z

salt/engines/libvirt_events.py

@@ -0,0 +1,617 @@
+# coding=utf-8


I know this is picky, but can you make this encoding line the standard version?

# -*- coding: utf-8 -*-

rallytime · 2018-03-15T18:45:32Z

salt/engines/libvirt_events.py

+# coding=utf-8
+
+'''
+An engine that listens for libvirt events and resend them to the salt event bus.


resend --> resends

Can you also add a version directive to this?

.. versionadded:: Fluorine

rallytime · 2018-03-15T18:48:20Z

salt/engines/libvirt_events.py

+
+# pylint: disable=invalid-name
+
+msg = ''


This type of msg handling and version checking should be done in the __virtual__() function.

@cbosdo actually @rallytime is right. So just move checks to the __virtual__ itself:

try: import libvirt except ImportError: libvirt = None def __virtual__(): ''' Only load if libvirt python binding is present ''' if libvirt is None: msg = 'libvirt module not found' elif libvirt.getVersion() < 1000000: msg = 'libvirt >= 1.0.0 required' else: msg = '' return bool(libvirt), msg

OK, basically copypaste this. 😉

This makes pylint complain on the if libvirt is None: line:
[used-before-assignment] Using variable 'libvirt' before assignment

And on the libvirt = None line that you didn't move:
[redefined-outer-name] Redefining name 'libvirt' from outer scope

If checks has to be in __virtual__, then just mute Lint in this case.

rallytime · 2018-03-15T18:50:01Z

salt/engines/libvirt_events.py

+    '''
+    Convenience function adding common data to the event and sending it
+    on the salt event bus.
+    '''


Can you document all of the arguments here? And provide an example of how they should be used?

rallytime · 2018-03-15T18:50:22Z

salt/engines/libvirt_events.py

+def salt_send_domain_event(opaque, conn, domain, event, event_data):
+    '''
+    Helper function send a salt event for a libvirt domain
+    '''


Same comment here about docs.

Rather than commenting in many places repeatedly, I'll just say this here: Any public facing function needs docs for all args/kwargs and some examples of usage. If you could update all of those below, that would be great.

All these helper functions aren't intended to be used by external code, I'ld rather hide them, but adding those comments can't harm

@cbosdo even hidden usually should at least document what it does. But yes, I would underscore them in front, since they are all internal.

cbosdo · 2018-03-28T06:09:37Z

any update on that PR?

rallytime · 2018-03-28T13:22:48Z

@cbosdo There's a lint violation here: https://jenkins.saltstack.com/job/PR/job/salt-pr-lint-n/20500/violations/file/salt/engines/libvirt_events.py/

@isbm

Libvirt API offers clients to register callbacks for various events. libvirt_events engine will listen on a libvirt URI (local or remote) for events and send them to the salt event bus. Special thanks to @isbm for the code cleanup help

cbosdo · 2018-03-28T13:39:49Z

@rallytime right! fixed all the pylint errors now.

rallytime · 2018-03-28T14:21:49Z

@cbosdo Awesome! Thanks! We're working on getting some testing issues resolved here, so we'll restart them when that's resolved.

In the mean time, @garethgreenaway can you swing by here and review this when you get a moment?

garethgreenaway · 2018-03-28T16:26:31Z

Looks good!

cbosdo · 2018-03-28T18:43:39Z

hum... looks like the branch would need a rebase before merging... should I do it now that it has been approved or you can still do it at merge time?

cbosdo · 2018-04-03T07:03:01Z

@rallytime is it me or the changes you requested (which have been implemented by the time) are blocking this PR?

rallytime · 2018-04-04T20:46:22Z

re-run py2

rallytime · 2018-04-04T20:46:52Z

@cbosdo Apologies for the delay. Some of the test results timed out so I am re-running some. Once those come back we can get this in. Thanks for the bump!

rallytime requested a review from garethgreenaway March 9, 2018 17:44

isbm suggested changes Mar 12, 2018

View reviewed changes

cbosdo force-pushed the develop branch 2 times, most recently from 7c1524c to 1b0daf9 Compare March 13, 2018 12:30

isbm reviewed Mar 14, 2018

View reviewed changes

cbosdo force-pushed the develop branch from b115071 to 9e44461 Compare March 14, 2018 15:56

cbosdo force-pushed the develop branch 2 times, most recently from ba50b0e to 7ea9446 Compare March 14, 2018 15:59

cbosdo force-pushed the develop branch from 7ea9446 to 5af636f Compare March 15, 2018 16:39

rallytime suggested changes Mar 15, 2018

View reviewed changes

cbosdo force-pushed the develop branch 7 times, most recently from fb4ec0b to 53cb886 Compare March 22, 2018 08:11

cbosdo force-pushed the develop branch from 900e118 to 197eba7 Compare March 28, 2018 06:12

Add engine relaying libvirt events

7abcaf2

Libvirt API offers clients to register callbacks for various events. libvirt_events engine will listen on a libvirt URI (local or remote) for events and send them to the salt event bus. Special thanks to @isbm for the code cleanup help

cbosdo force-pushed the develop branch from 197eba7 to 7abcaf2 Compare March 28, 2018 13:39

garethgreenaway approved these changes Mar 28, 2018

View reviewed changes

Merge branch 'develop' into develop

a981acb

rallytime approved these changes Apr 4, 2018

View reviewed changes

Merge branch 'develop' into develop

d386daa

isbm approved these changes Apr 5, 2018

View reviewed changes

rallytime merged commit 9ab8c26 into saltstack:develop Apr 5, 2018

cbosdo mentioned this pull request Apr 27, 2018

Open suse 2018.3.0 openSUSE/salt#79

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add engine relaying libvirt events #46461

Add engine relaying libvirt events #46461

cbosdo commented Mar 9, 2018 •

edited

Loading

cbosdo commented Mar 12, 2018

isbm left a comment •

edited

Loading

isbm Mar 12, 2018

cbosdo Mar 13, 2018

isbm commented Mar 12, 2018 •

edited

Loading

isbm Mar 14, 2018

cbosdo commented Mar 14, 2018

rallytime commented Mar 15, 2018

cbosdo commented Mar 15, 2018

rallytime left a comment

rallytime Mar 15, 2018

rallytime Mar 15, 2018

rallytime Mar 15, 2018

rallytime Mar 15, 2018

isbm Mar 16, 2018

cbosdo Mar 16, 2018

isbm Mar 16, 2018

rallytime Mar 15, 2018

rallytime Mar 15, 2018

rallytime Mar 15, 2018

cbosdo Mar 16, 2018

isbm Mar 16, 2018

cbosdo commented Mar 28, 2018

rallytime commented Mar 28, 2018

cbosdo commented Mar 28, 2018

rallytime commented Mar 28, 2018

garethgreenaway commented Mar 28, 2018

cbosdo commented Mar 28, 2018

cbosdo commented Apr 3, 2018

rallytime commented Apr 4, 2018

rallytime commented Apr 4, 2018


		libvirt_events.start(uri, prefix)

		assert call.call_list() != []

Add engine relaying libvirt events #46461

Add engine relaying libvirt events #46461

Conversation

cbosdo commented Mar 9, 2018 • edited Loading

What does this PR do?

What issues does this PR fix or reference?

Tests written?

Commits signed with GPG?

cbosdo commented Mar 12, 2018

isbm left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

isbm commented Mar 12, 2018 • edited Loading

Choose a reason for hiding this comment

cbosdo commented Mar 14, 2018

rallytime commented Mar 15, 2018

cbosdo commented Mar 15, 2018

rallytime left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbosdo commented Mar 28, 2018

rallytime commented Mar 28, 2018

cbosdo commented Mar 28, 2018

rallytime commented Mar 28, 2018

garethgreenaway commented Mar 28, 2018

cbosdo commented Mar 28, 2018

cbosdo commented Apr 3, 2018

rallytime commented Apr 4, 2018

rallytime commented Apr 4, 2018

cbosdo commented Mar 9, 2018 •

edited

Loading

isbm left a comment •

edited

Loading

isbm commented Mar 12, 2018 •

edited

Loading