Add MQTT notification callback plugin #23769

mtreinish · 2017-04-19T20:40:38Z

This commit adds a new callback for emitting mqtt notifications for
tasks and playbooks.

SUMMARY

This commit adds a new callback module for emitting mqtt notifications for
tasks and playbooks.

ISSUE TYPE

New Module Pull Request

COMPONENT NAME

ANSIBLE VERSION

ansible 2.4.0 (detached HEAD c3cd60722f) last updated 2017/04/19 16:12:09 (GMT -400)

ADDITIONAL INFORMATION

dmsimard · 2017-04-21T14:41:05Z

lib/ansible/plugins/callback/mqtt.py

+        out_topic = self.base_topic + '/' + topic
+        publish.single(out_topic, msg, hostname=self.mqtt_hostname,
+                       port=self.port, auth=self.auth, tls=self.tls,
+                       client_id=self.client_id)


I think adding a timestamp field would be nice.

dmsimard · 2017-04-21T14:52:27Z

lib/ansible/plugins/callback/mqtt.py

+    CALLBACK_NAME = 'mqtt'
+    CALLBACK_NEEDS_WHITELIST = True
+
+    def _parse_config(self, config):


Just a thought, you could leverage get_config from constants.py to allow end users to use env variables or ansible.cfg as they prefer instead of relying on a different config file.

Example usage in ARA: https://github.com/openstack/ara/blob/master/ara/config.py

👍 on get_config for clarity.

dmsimard · 2017-05-02T14:38:01Z

Pinged you on IRC about it but for posterity...
I noticed you don't have item methods in the callback (ex: v2_runner_item_on_ok).

When dealing with a task that has items (ex: with_items), the result._result var of the task is actually pretty empty -- it doesn't contain the results of the items, just a summary message like "All items completed".

The results are kept in the items' result variables. Maybe you want to pick those up, I don't know.
Example implementation: https://review.openstack.org/#/c/461612/3/ara/plugins/callbacks/log_ara.py

Note that I just filed a bug about the inconsistent callback execution with items and task completion: #24207

mtreinish · 2017-05-28T14:14:54Z

@dmsimard The latest revision fixes mots of your comments. I still have to investigate the config switch, I haven't had a chance to dig into that yet though.

omgjlk

Some nits, but I think there's one actual problem to address.

omgjlk · 2017-10-07T20:02:16Z

lib/ansible/plugins/callback/mqtt.py

+
+    Optionally these values can be set via a yaml file in /etc/mqtt_client.yaml,
+    or $HOME/.mqtt_client.yaml (where each env variable lowercase is a key in
+    the yaml files)


Looks like some things could be in ENV, some could be in the user's dot file, and some could be in /etc/ and as long as they aren't overlapping, it'll gather all the parts from all the files. That's fine, but might want to mention it in the docs (could be as a follow up after the fact too). Additionally the ordering should be documented as well (ENV, user dir, /etc/ dir)

That's a good point, I say below that ENV takes first priority, but I should outline the config file priority too. I've got to respin this anyway, I'll add that bit on in the next rev

omgjlk · 2017-10-07T20:03:49Z

lib/ansible/plugins/callback/mqtt.py

+    def _publish(self, topic, msg):
+        out_topic = self.base_topic + '/' + topic
+        msg['timestamp'] = datetime.datetime.utcnow().isoformat()
+        publish.single(out_topic, json.dumps(msg),


Could this ever raise an error? Should it be trapped and logged correctly, or is there infra around callbacks that takes care of this already?

Oh, that's true, I don't know if there is any infra around callback plugins to do that or not. (I'd have to dig through the call path to find out, most of the interface isn't documented). It probably doesn't hurt to collect the errors either way though.

Actually, looking at it a bit more closely I'm not sure how to setup logging for something like this from the plugin context, I couldn't find any good examples of that to base it off of. If you have any suggestions I'll add that on in a future patch.

omgjlk · 2017-10-07T20:09:50Z

lib/ansible/plugins/callback/mqtt.py

+            'status': "FAILED",
+            'host': self.hostname,
+            'ansible_host': host,
+            'playbook_id': self.session,


Above playbook_id was set to self.uuid, but here it's being set to self.session which above is used for session. Maybe some copy/pasta got messed up?

oops, yep that's a good catch. I'll respin this now

mtreinish · 2017-10-16T17:56:29Z

The latest revision addresses the review comments from @j2sol

ansibot · 2017-10-16T18:56:19Z

The test ansible-test sanity --test pylint [?] failed with the following error:

lib/ansible/plugins/callback/mqtt.py:224:15: duplicate-key Duplicate key 'ansible_host' in dictionary

click here for bot help

This commit adds a new callback for emitting mqtt notifications for tasks and playbooks.

omgjlk

Looks good know to me. If I get info on handling tracebacks in a callback I'll respond.

omgjlk · 2017-10-17T22:50:27Z

shipit

jpmens · 2018-11-29T19:31:59Z

shipit

sivel · 2018-11-29T19:51:43Z

lib/ansible/plugins/callback/mqtt.py

+        MQTT_KEYFILE    (optional): The path pointing to the PEM encoded client
+                                    private key
+        MQTT_CLIENT_ID  (optional): MQTT client identifier, defaults to
+                                    hostname + pid


This callback needs to be updated to utilize "config". See other current callback plugins for their DOCUMENTATION and set_options method.

ssbarnea · 2018-11-29T19:52:02Z

@bcoca can you give us a hand with that?

Akasurde · 2020-08-19T08:53:10Z

Please submit new callback plugin requests to https://github.com/ansible-collections/community.general. Thanks for your contribution.

ansibot added affects_2.4 This issue/PR affects Ansible v2.4 c:plugins/callback needs_triage Needs a first human triage before being processed. new_plugin This PR includes a new plugin. labels Apr 19, 2017

mtreinish force-pushed the mqtt-callback branch from c3cd607 to 9d0ec4b Compare April 20, 2017 15:46

abadger removed the needs_triage Needs a first human triage before being processed. label Apr 20, 2017

dmsimard reviewed Apr 21, 2017

View reviewed changes

ansibot added the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label May 2, 2017

mtreinish force-pushed the mqtt-callback branch from 9d0ec4b to 63938c2 Compare May 28, 2017 14:14

ansibot removed the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label May 28, 2017

ansibot added stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. support:core This issue/PR relates to code supported by the Ansible Engineering Team. labels Jun 23, 2017

mtreinish force-pushed the mqtt-callback branch from 63938c2 to c0031e9 Compare July 6, 2017 14:30

ansibot removed the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Jul 6, 2017

ansibot added the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Jul 18, 2017

ansibot added the plugins/callback label Sep 9, 2017

omgjlk suggested changes Oct 7, 2017

View reviewed changes

ansibot added the needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. label Oct 7, 2017

mtreinish force-pushed the mqtt-callback branch from c0031e9 to 89b72b7 Compare October 16, 2017 17:56

ansibot removed the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Oct 16, 2017

Add MQTT notification callback plugin

8d7b239

This commit adds a new callback for emitting mqtt notifications for tasks and playbooks.

mtreinish force-pushed the mqtt-callback branch from 89b72b7 to 8d7b239 Compare October 16, 2017 19:09

omgjlk approved these changes Oct 17, 2017

View reviewed changes

ansibot removed the needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. label Oct 17, 2017

ansibot added the stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. label Oct 25, 2017

ansibot added support:community This issue/PR relates to code supported by the Ansible community. and removed support:core This issue/PR relates to code supported by the Ansible Engineering Team. labels Sep 19, 2018

ansibot added the community_review In order to be merged, this PR must follow the community review workflow. label Oct 25, 2018

ssbarnea approved these changes Nov 29, 2018

View reviewed changes

sivel requested changes Nov 29, 2018

View reviewed changes

ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed community_review In order to be merged, this PR must follow the community review workflow. labels Nov 29, 2018

ansibot added community_review In order to be merged, this PR must follow the community review workflow. and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels Mar 4, 2020

ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed community_review In order to be merged, this PR must follow the community review workflow. labels Mar 28, 2020

Akasurde closed this Aug 19, 2020

ansible locked and limited conversation to collaborators Sep 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add MQTT notification callback plugin #23769

Add MQTT notification callback plugin #23769

mtreinish commented Apr 19, 2017

dmsimard Apr 21, 2017

dmsimard Apr 21, 2017

evrardjp May 30, 2018

dmsimard commented May 2, 2017

mtreinish commented May 28, 2017

omgjlk left a comment

omgjlk Oct 7, 2017

mtreinish Oct 16, 2017

omgjlk Oct 7, 2017

mtreinish Oct 16, 2017

mtreinish Oct 16, 2017

omgjlk Oct 7, 2017

mtreinish Oct 16, 2017

mtreinish commented Oct 16, 2017

ansibot commented Oct 16, 2017

omgjlk left a comment

omgjlk commented Oct 17, 2017

jpmens commented Nov 29, 2018

sivel Nov 29, 2018

ssbarnea commented Nov 29, 2018

Akasurde commented Aug 19, 2020

Add MQTT notification callback plugin #23769

Add MQTT notification callback plugin #23769

Conversation

mtreinish commented Apr 19, 2017

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

ADDITIONAL INFORMATION

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmsimard commented May 2, 2017

mtreinish commented May 28, 2017

omgjlk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtreinish commented Oct 16, 2017

ansibot commented Oct 16, 2017

omgjlk left a comment

Choose a reason for hiding this comment

omgjlk commented Oct 17, 2017

jpmens commented Nov 29, 2018

Choose a reason for hiding this comment

ssbarnea commented Nov 29, 2018

Akasurde commented Aug 19, 2020