Webhook support with apprise

[papamoose]

Adds webhook support. This is a rethink on pull request #307, which I've now closed.

It uses the python module apprise to allow a user to specify any number of webhooks.

This commit compiles a dictionary with what actions u-u has taken and POST's the result as json to the webhooks specified.

Example of the resulting json POST'd.

{
  "hostname": "zeus.example.com",
  "reboot_flag": true,
  "hold_flag": false,
  "result": "SUCCESS",
  "packages": {
    "upgraded": [
      "alsa-ucm-conf",
      "cloud-init",
      "libdrm-common",
      "libdrm2",
      "libnetplan0",
      "libnss-systemd",
      "libpam-modules",
      "libpam-modules-bin",
      "libpam-runtime",
      "libpam-systemd",
      "libpam0g",
      "libprocps8",
      "libsystemd0",
      "libudev1",
      "libudisks2-0",
      "linux-firmware",
      "netplan.io",
      "open-vm-tools",
      "procps",
      "python-apt-common",
      "python3-apt",
      "python3-update-manager",
      "snapd",
      "systemd",
      "systemd-sysv",
      "systemd-timesyncd",
      "thermald",
      "udev",
      "udisks2",
      "update-manager-core"
    ]
  }
}

@simon-ourmachinery

[mvo5]

Thanks for this pull request. Let me think about this, I like the idea and I understand in todays world webhooks are much more important than sending a summary mail :) Here are some preliminary thoughts: One thing I'm unsure about is the use of apprise, AFAICT it's not packaged yet for Debian or Ubuntu so I would not be able to include it in the default packages (yet). I would like to make sure your use-case gets supported though. I wonder if you would benefit from having a plugin mechanism in unattended-upgrades, i.e. something like /usr/share/unattended-upgrades/plugins (and one version in /etc) that is imported by u-u and that will call pre/post upgrade hooks with essentially the same information as "send_summary_mail" has? Wdyt?

[papamoose]

I see how apprise not having a debian package would a roadblock. I thought it was nice as u-u wouldn't have to maintain any specifics on any particular webhook.

A plugin system like you describe sounds like a great idea! An ability to import send_summary_mail data to a plugin would definitely allow for the greatest flexibility and not require u-u to deal with any nuance of any webhook in particular, but instead stay focused on its core goal.

Additionally, it would allow for more customization by the user. Instead of only being able to send json to a webhook the plugin could customize the message to display in a more user friendly way based on the webhook used.

How do you imagine a user would specify the webhook url(s) and how would they be used in the context of the plugin?

[mvo5]

@papamoose Thanks for your thoughts on a plugin system. Python makes this fairly easy and it would allow some flexiblity, we could even ship some example plugins (e.g. one that uses apprise :)

I drafted some ideas in https://paste.ubuntu.com/p/jQ7fzvH253/ - obviously very incomplete but hopefully enough to get the idea. The user could install plugins into a location (TBH) like /etc/unattended-upgrades/plugins/ and the plugin can be any class that provides a "summary()" method with the right signature (also TBD, probably some dict for easier extensibility). WDYT?

[mvo5]

Fwiw, I pushed a slightly more elaborate version of the plugin idea into https://github.com/mvo5/unattended-upgrades/compare/plugins?expand=1 still needs some tweaks as it e.g. does not capture right now if a reboot is required (and some other things like hold pkg handling) or not but that should be pretty trivial to add.

[papamoose]

Thanks. What you posted on launchpad looks good. I'll take a look at your most recent update soon. I'm switching jobs and moving within the next month so I expect to have little time to dedicate to this. I'm not ignoring your on purpose. :) Your work on the plugin system is much appreciated.

[mvo5]

Thanks @papamoose - good luck with your job switch and the moving. I will keep working on this, probably a bit slower from next week on. Given that it's a public API that will need to be supported forever I don't want to rush things and be a bit careful about naming of e.g. the dict keys etc. I am also wondering if I should make it even more generic by just calling any executable in the plugin path and sending the data via json to stdin of the process. Makes the plugins themselfs slightly more complex (as they now need to read from stdin and de-serialize) but one can write in any language (hello /bin/sh) this way :)

[a-detiste]

Hi, this new plugin system is awesome. The python interface is fine. The "json / stdin" would be more limitatign in the future I guess; if the PluginManager do more stuff with the plugin objects.

[mabed-fr]

Hello

What method will be used?

GET? PUSH? PUT

Will it be possible to configure the headers to be sent?

Will there be more than one variable for the content of the message?

[mvo5]

Hello

What method will be used?

GET? PUSH? PUT

Will it be possible to configure the headers to be sent?

Will there be more than one variable for the content of the message?

Thanks for your interest. The current thinking is that unattended-upgrades will not by itself do the webhook call but instead provide a plugin interface that gives the plugin enough information to perform the action. So you have total flexibility for get/post/put. We will probably also push an example plugin (this apprise one looks very nice for example).

[mabed-fr]

Thank you for your explain.

I am missing a point to clarify. Who will be the trigger for this webhook / plugin?

Mabed

[papamoose]

As @mvo5 said. The goal has changed. u-u will only provide a plugin system which will probably pass a dictionary with the summary of changes to any and all plugins that is on the user to write.

As it stands now this commit only passes the summary data as text (in json format) to whatever webhooks apprise supports. I don't intend for it to do more than that at this time.

[mvo5]

Fwiw, I worked a bit more on this in https://github.com/mvo5/unattended-upgrades/compare/plugins?expand=1 and I'm reasonable happy with the result. The only thing I'm still mulling over is if the argument for the new postrun() function should be a simple python dict with json-like keys as it is right now (which will make the webhook thing very straightforward) or if instead I should use a python class/dataclass - the upside of this approach is that it's more python-ish and that mypy could ensure a certain level of type-safety when working with the data but it's more cumbersome in the plugin side if most plugins use this in a json(ish) context. Let me know if someone as opinions about this question please :)

And of course the next step is also to ship this nice apprise work as a advanced example or even as a default.

[a-detiste]

The objec-ified way is cleaner ...

[papamoose]

Finally had some time to take a look at this (https://github.com/mvo5/unattended-upgrades/compare/plugins?expand=1).

I was able to get a rough draft, minimal viable product, of the apprise json notifier plugin working.

1. Re: postrun python dict vs class/dataclass. I'll use either. Using the dict was easy and a low barrier to entry. As long as there is an example included on how to use I doubt it will make much difference to anyone writing a plugin. Meaning @mvo5 should choose what they prefer.
2. The plugin system seems to be working as expected!
3. I'm taking advantage of import apt_pkg so the user can set variables. Moving these variables into a dedicated file worked great. /etc/apt/apt.conf.d/51uu-apprise.
4. I also imported and copied the logging functions from u-u into my script so that it logs in the same way u-u does. It would be nice to get log_once() without needing to copy it into the plugin.

import logging
logged_msgs = set()  # type: AbstractSet[str]
def log_once(msg):
    # type: (str) -> None
    global logged_msgs
    if msg not in logged_msgs:
        logging.info(msg)
        logged_msgs.add(msg)  # type: ignore

I'm hoping to have a bit more time now so getting the plugin into a state that others can look at shouldn't be too much longer now.

[papamoose]

@mvo5 https://github.com/papamoose/unattended-upgrades/blob/plugins-apprise/examples/plugins/apprise.py

This is just your plugins branch with a WIP of my plugin.
I want to rename it and some of the functions inside it but it works as is now. I may end up refactoring it a bit too as it follows the pattern mail uses inside u-u which I no longer feel obligated to follow and could clean it up some.

I'd be happy to test a objectified version, as I did find I was passing result around more than I thought was necessary.

[stephen99scott]

@papamoose @mvo5 This is a very useful feature! I have a Raspbian Buster server natively running u-u@1.11.2, so I back-ported plugins support to this release (https://github.com/mvo5/unattended-upgrades/compare/1.11.2...stephen99scott:unattended-upgrades:plugins-buster?expand=1).

[a-detiste]

I have a Raspbian Buster server

This I will need this too for my 1300 brand new physicial Buster servers :-)

[papamoose]

Not sure where we left off with this... but I've been running this for the past year and have been very happy with it.

PR issued on the plugins branch: #349
Feel free to deny if this isn't how you want to proceed. I'm just getting the conversation started again around this. :)

[mvo5]

I finally managed to spent a bit of time on this again and I am quite happy with my current code in #355 - if someone could double check and test that would be super appreciated.

(and SORRY that this took me forever to get to!)

[JonnyBDev]

Hey guys,

just wanted to keep the conversation going. Currently in the process of using unattended-upgrades and we'd love to see a feature like this! We really need Webhooks for the management of our servers.

I can see that the issue #355 is in the status "Merged". Am I missing something here or is this already implemented? A bit confusing, that this PR is still open - even if it's for an older iteration of the system.

Would appreciate some feedback :) Thanks!

[papamoose]

@mvo5 has added plugin support. If you wish to add my apprise plugin you would need u-u v2.11+.

Then you can take a look my repo that contains notification plugin: https://github.com/papamoose/unattended-upgrades-plugin-webhook

I think this MR should be closed. What do you think @mvo5?

[mvo5]

@mvo5 has added plugin support. If you wish to add my apprise plugin you would need u-u v2.11+.

Then you can take a look my repo that contains notification plugin: https://github.com/papamoose/unattended-upgrades-plugin-webhook

I think this MR should be closed. What do you think @mvo5?

Yeah, I am fine closing it in it's current form. I would like to include a link to your plugin repo in the readme, I think that would be nice for people who need this functionaltiy. Or even include it in the examples/ dir (if it does not change very much) :)

[papamoose]

I would like to include a link to your plugin repo in the readme, I think that would be nice for people who need this functionality.

Sure!

Or even include it in the examples/ dir

I'm of two minds on this:

1. putting it in the examples directory will distribute the code out to the people that want to use it. Which is excellent!
2. It seems like a plugin should be a separate package (willing to be convinced otherwise). Though, I'm not familiar with the process in getting a package into Debian and I assume that would take a while to do.

Therefore, it seems like putting the plugin in the examples directory is a great solution for right now, with a longer term goal of getting the package added to Debian.

What do you think?

(if it does not change very much) :)

I don't imagine it would need to change much. Since I've written it I don't think I've touched the code outside the time you added plugin support. Worst case, you could kick it out if it becomes a problem and just point at my repo.

[papamoose]

I'm going to close this in favor of adding this as an example in #369.