Added custom notification messages using text/template

[MMauro94]

This should take care of #59. This is my first time ever with Golang so I'm sorry if I've done anything not in the correct way.

As per your suggestion, I've used the text/template package. The default template uses the exact same wording of current notifications.

A few things:

1. I've trimmed newlines on the template result, otherwise the templates has to look ugly
2. Currently supported params for templates: StartTime, LogOutput. BackupStopContainerLabel (and Error for failures). I would have liked to add more (like number of stopped containers, number of pruned backups, took time, and many more), but some things would have required too much refactor of the code to have this variables ready in the notifySuccess function. Any thoughts on this?
3. I've yet to update the README, waiting on your feedback.

[m90]

This is really good, thank you! I tested this and it works well for me.

I left some comments inline, let me know what you think.

[m90]

but some things would have required too much refactor of the code to have this variables ready in the notifySuccess function. Any thoughts on this?

My initial idea about approaching this would be the following: instead of having *script.start and *script.output we introduce a type like this:

type stats struct {
  Start time.Time
  Output *bytes.Buffer
  PrunedBackups int
  // whatever we want to add ...
}

type script struct {
// script holds all the stateful information required to orchestrate a
// single backup run.
type script struct {
	cli          *client.Client
	minioClient  *minio.Client
	webdavClient *gowebdav.Client
	logger       *logrus.Logger
	sender       *router.ServiceRouter
	template     *template.Template
	hooks        []hook
	hookLevel    hookLevel

	file   string
	stats *stats

	c *config
}

Any method on *script could now write to these stats whenever needed and the notify functions could pass it down to the templates.

[MMauro94]

I agree, that's the approach I was going to suggest.

One thing: basically all the propertis in the stats struct will have to be passed down to the template, so I think it would be a good idea to add it to the notification data, like this:

type NotificationData struct {
  Error error
  Config *config
  Stats *stats
}

But what happens to the *bytes.Buffer output when being passed down to a template? Is it simply ignored? (In this case we'll need to keep the LogOuptut string in NotificationData)

[m90]

But what happens to the *bytes.Buffer output when being passed down to a template?

That's a good question. The template package will try to somehow stringify anything it sees using reflection, and right now I'm not sure how it will represent a buffer (it could either be turned into an UTF-8 string or into a set of bytes, i.e. octal representation).

After trying it out, it seems it picks the human readable version though, so I think we could do this: https://go.dev/play/p/aw4XCMH4GjD

[MMauro94]

As you can see from the two commits I've implemented the discussed changes.

From my testing everything works fine. I'll also share my success template:

{{ define "title_success" -}}
✅ Successfully ran backup {{ .Config.BackupStopContainerLabel }}
{{- end }}


{{ define "body_success" -}}
▶️ Start time: {{ .Stats.StartTime | formatTime }}
⏹️ End time: {{ .Stats.EndTime | formatTime }}
⌛ Took time: {{ .Stats.TookTime }}
🛑 Stopped containers: {{ .Stats.Containers.Stopped }}/{{ .Stats.Containers.All }} ({{ .Stats.Containers.StopErrors }} errors)
⚖️ Backup size: {{ .Stats.BackupFile.SizeBin }}
🗑️ Pruned backups: {{ .Stats.Archives.Local.Pruned }}/{{ .Stats.Archives.Local.Total }} ({{ .Stats.Archives.Local.PruneErrors }} errors)
{{- end }}

And the result:

[m90]

This is going to be brilliant. Thanks for putting so much thought into this, it's really appreciated.

I left a few comments inline once more, nothing really major though.

We'll also need docs for this I guess. Do you think a new section in How to would be enough considering we'll have to document the entire NotificationData type?

[MMauro94]

We'll also need docs for this I guess. Do you think a new section in How to would be enough considering we'll have to document the entire NotificationData type?

I think the best way is to add a section in the How to that explains the feature, but leave the full documentation of NotificationData and the funcs in a separate file, that will be of course be linked from the how to section.

What do you think?

[m90]

I think the best way is to add a section in the How to that explains the feature, but leave the full documentation of NotificationData and the funcs in a separate file, that will be of course be linked from the how to section.

This probably depends on how much docs end up being written, but it definitely sounds like a good start. Once there is content, it can always be restructured.

[MMauro94]

I have took care of all the issues you mentioned. Also, I have drafted the initial documentation for the notification templates.

Let me know what do you think :)

EDIT: the tests have failed. Is it possible that the reason is that I've removed the sleep in the pruning, thus not leaving docker enough time to start up the containers before the check is made?

[m90]

Is it possible that the reason is that I've removed the sleep in the pruning, thus not leaving docker enough time to start up the containers before the check is made?

Considering docker ps prints the following for the failing test (notice the Status: Created):

CONTAINER ID        IMAGE                                      COMMAND                  CREATED             STATUS                     PORTS               NAMES
6f9cddc5ca40        postgres:12.2-alpine                       "docker-entrypoint.s…"   2 seconds ago       Up Less than a second      5432/tcp            test_stack_pg.1.v5xo597bkr5p87xzhndkw92lq
4d9334b0ecb0        offen/offen:latest                         "/sbin/tini -- offen"    2 seconds ago       Created                                        test_stack_offen.2.i9b6yneky00vz1it2ep0gxts3
15f16d539f49        postgres:12.2-alpine                       "docker-entrypoint.s…"   24 seconds ago      Exited (0) 4 seconds ago                       test_stack_pg.1.wk7mfmsoefwtiubaxfbb5dsew
20148f0030d4        offen/docker-volume-backup:canary          "/root/entrypoint.sh"    29 seconds ago      Up 25 seconds                                  test_stack_backup.1.bd4349hxlf0w3ap0tqxyr5gv0
9e586796d160        minio/minio:RELEASE.2020-08-04T23-10-51Z   "/bin/ash -c 'mkdir …"   30 seconds ago      Up 26 seconds              9000/tcp            test_stack_minio.1.y322im0htf6yf96rucucung8b
21e9a0de1407        offen/offen:latest                         "/sbin/tini -- offen"    30 seconds ago      Exited (0) 3 seconds ago                       test_stack_offen.2.79fst9vq9i0gmztm8r9vp4psd
50ef1d4b3876        offen/offen:latest                         "/sbin/tini -- offen"    30 seconds ago      Exited (0) 3 seconds ago                       test_stack_offen.1.r1ibf6j4s39yi0jgs815z8zqc

I think it's very likely this is a race condition created by removing the sleep. I think it'd be ok to add a short sleep before this check runs:

docker-volume-backup/test/swarm/run.sh

Lines 26 to 30 in 3bb99a7

	if [ "$(docker ps -q | wc -l)" != "5" ]; then
	echo "[TEST:FAIL] Expected all containers to be running post backup, instead seen:"
	docker ps -a
	exit 1
	fi

I'll have a close look at the PR itself later today.

[MMauro94]

I think it'd be ok to add a short sleep before this check runs

Agreed. Do you think 5 seconds is enough?

Looking at the test directory, I think the right place is just before these three lines:

• docker-volume-backup/test/cli/run.sh

  Line 53 in 3bb99a7

  if [ "$(docker ps -q | wc -l)" != "2" ]; then

• docker-volume-backup/test/compose/run.sh

  Line 32 in 3bb99a7

  if [ "$(docker-compose ps -q | wc -l)" != "4" ]; then

• docker-volume-backup/test/swarm/run.sh

  Line 26 in 3bb99a7

  if [ "$(docker ps -q | wc -l)" != "5" ]; then

Is it correct? Or just the line in swarm/run.sh is sufficient?

[m90]

I think we only need to add it for the swarm test:

• the cli test doesn't stop containers on purpose, so there's no restart to wait for
• considering the compose test does some gpg decryption before this gives us some implicit sleep that should be enough to cater for this setup

---

That being said: it could be argued that the compose test should check that containers are up before it looks at the artifacts, which would probably also require the sleep to be introduced.

Long story short: it would be great if you could add the sleep for the swarm test and I'll see if I can clean up and also document the tests a little better soon (I'd also like to add a test for the notifications feature at some point, but this will be a little tricky).

[m90]

This turned out really well. I left some minor suggestions, feel free to adjust or disregard these as you wish. If we can also get CI green again I would say this one is ready to go 🎉

[MMauro94]

It seems to be fixed with the sleep. I've also implemented the other changes.

[m90]

This is now released in v2.11.0. Thanks a lot for this top-notch contribution 🎩

[MMauro94]

Thanks to you for the project and your quick to the point assistance 🔥