Add support for prometheus export and push (#57)

* generation of prometheus file from backup summary

* add labels to metrics

* prometheus groups of profiles

* allow generic status by registering an interface

* docs: add prometheus in README

* display an error instead of a panic

* add user defined prometheus labels

Author: creativeprojects

.gitignore

@@ -29,3 +29,4 @@
 /*.log
 
 status.json
+*.prom

.goreleaser.yml

@@ -1,6 +1,6 @@
 project_name: resticprofile
 env:
-  - RESTIC_VERSION=0.12.0
+  - RESTIC_VERSION=0.12.1
 
 before:
   hooks:

Makefile

@@ -22,8 +22,6 @@ TESTS=./...
 COVERAGE_FILE=coverage.out
 
 BUILD=build/
-RESTIC_VERSION=0.12.0
-GO_VERSION=1.16
 
 BUILD_DATE=`date`
 BUILD_COMMIT=`git rev-parse HEAD`

README.md

@@ -29,10 +29,11 @@ With resticprofile:
 * You can generate cryptographically secure random keys to use as a restic key file
 * You can easily schedule backups, retentions and checks (works for *systemd*, *crond*, *launchd* and *windows task scheduler*)
 * You can generate a simple status file to send to some monitoring software and make sure your backups are running fine 
-* **[new for v0.10.0]** You can use a template syntax in your configuration file
-* **[new for v0.11.0]** You can generate scheduled tasks using *crond*
+* You can use a template syntax in your configuration file
+* You can generate scheduled tasks using *crond*
 * **[new for v0.12.0]** Get backup statistics in your status file
 * **[new for v0.14.0]** Automatically clear up [stale locks](#locks)
+* **[new for v0.15.0]** Export a **prometheus** file after a backup, or send the report to a push gateway automatically
 
 The configuration file accepts various formats:
 * [TOML](https://github.com/toml-lang/toml) : configuration file with extension _.toml_ and _.conf_ to keep compatibility with versions before 0.6.0
@@ -101,6 +102,7 @@ For the rest of the documentation, I'll be showing examples using different form
   * [Changing schedule\-permission from user to system, or system to user](#changing-schedule-permission-from-user-to-system-or-system-to-user)
 * [Status file for easy monitoring](#status-file-for-easy-monitoring)
   * [Extended status](#extended-status)
+* [Prometheus](#prometheus)
 * [Variable expansion in configuration file](#variable-expansion-in-configuration-file)
   * [Pre\-defined variables](#pre-defined-variables)
   * [Hand\-made variables](#hand-made-variables)
@@ -1395,6 +1397,8 @@ Here's an example of a generated file, where you can see that the last `check` f
 }
 ```
 
+## Extended status
+
 Note: In the backup section above you can see some fields like `files_new`, `files_total`, etc. This information is only available when resticprofile's output is either *not* sent to the terminal (e.g. redirected) or when you add the flag `extended-status` to your backup configuration.
 This is a technical limitation to ensure restic displays terminal output correctly. 
 
@@ -1405,8 +1409,6 @@ This is a technical limitation to ensure restic displays terminal output correct
 - stderr
 - duration
 
-## Extended status
-
 `extended-status` is **not set by default because it hides any output from restic**
 
 ```yaml
@@ -1421,6 +1423,89 @@ profile:
 
 ```
 
+# Prometheus
+
+resticprofile can generate a prometheus file, or send the report to a push gateway. For now, only a `backup` command will generate a report.
+Here's a configuration example with both options to generate a file and send to a push gateway:
+
+```yaml
+root:
+    inherit: default
+    prometheus-save-to-file: "root.prom"
+    prometheus-push: "http://localhost:9091/"
+    backup:
+        extended-status: true
+        no-error-on-warning: true
+        source:
+          - /
+```
+
+Please note you need to set `extended-status` to `true` if you want all the available metrics. See [Extended status](#extended-status) for more information.
+
+Here's an example of the generated prometheus file:
+
+```
+# HELP resticprofile_backup_added_bytes Total number of bytes added to the repository.
+# TYPE resticprofile_backup_added_bytes gauge
+resticprofile_backup_added_bytes{profile="root"} 35746
+# HELP resticprofile_backup_dir_changed Number of directories with changes.
+# TYPE resticprofile_backup_dir_changed gauge
+resticprofile_backup_dir_changed{profile="root"} 9
+# HELP resticprofile_backup_dir_new Number of new directories added to the backup.
+# TYPE resticprofile_backup_dir_new gauge
+resticprofile_backup_dir_new{profile="root"} 0
+# HELP resticprofile_backup_dir_unmodified Number of directories unmodified since last backup.
+# TYPE resticprofile_backup_dir_unmodified gauge
+resticprofile_backup_dir_unmodified{profile="root"} 314
+# HELP resticprofile_backup_duration_seconds The backup duration (in seconds).
+# TYPE resticprofile_backup_duration_seconds gauge
+resticprofile_backup_duration_seconds{profile="root"} 0.946567354
+# HELP resticprofile_backup_files_changed Number of files with changes.
+# TYPE resticprofile_backup_files_changed gauge
+resticprofile_backup_files_changed{profile="root"} 3
+# HELP resticprofile_backup_files_new Number of new files added to the backup.
+# TYPE resticprofile_backup_files_new gauge
+resticprofile_backup_files_new{profile="root"} 0
+# HELP resticprofile_backup_files_processed Total number of files scanned by the backup for changes.
+# TYPE resticprofile_backup_files_processed gauge
+resticprofile_backup_files_processed{profile="root"} 3925
+# HELP resticprofile_backup_files_unmodified Number of files unmodified since last backup.
+# TYPE resticprofile_backup_files_unmodified gauge
+resticprofile_backup_files_unmodified{profile="root"} 3922
+# HELP resticprofile_backup_processed_bytes Total number of bytes scanned for changes.
+# TYPE resticprofile_backup_processed_bytes gauge
+resticprofile_backup_processed_bytes{profile="root"} 3.8524672e+07
+# HELP resticprofile_backup_status Backup status: 0=fail, 1=warning, 2=success.
+# TYPE resticprofile_backup_status gauge
+resticprofile_backup_status{profile="root"} 1
+# HELP resticprofile_build_info resticprofile build information.
+# TYPE resticprofile_build_info gauge
+resticprofile_build_info{goversion="go1.16.6",version="0.15.0-dev"} 1
+
+```
+
+## User defined labels
+
+You can add your own prometheus labels. Please note they will be applied to **all** the metrics.
+Here's an example:
+
+```yaml
+root:
+    inherit: default
+    prometheus-save-to-file: "root.prom"
+    prometheus-push: "http://localhost:9091/"
+    prometheus-labels:
+      - host: {{ .Hostname }}
+    backup:
+        extended-status: true
+        no-error-on-warning: true
+        source:
+          - /
+```
+
+which will add the `host` label to all your metrics.
+
+
 # Variable expansion in configuration file
 
 You might want to reuse the same configuration (or bits of it) on different environments. One way of doing it is to create a generic configuration where specific bits will be replaced by a variable.
@@ -1914,6 +1999,8 @@ Flags used by resticprofile only
 * **run-after**: string OR list of strings
 * **run-after-fail**: string OR list of strings
 * **status-file**: string
+* **prometheus-save-to-file**: string
+* **prometheus-push**: string
 
 Flags passed to the restic command line
 

command_error.go

@@ -0,0 +1,45 @@
+package main
+
+import (
+	"errors"
+	"fmt"
+	"strings"
+)
+
+type commandError struct {
+	scd    shellCommandDefinition
+	stderr string
+	err    error
+}
+
+func newCommandError(command shellCommandDefinition, stderr string, err error) *commandError {
+	return &commandError{
+		scd:    command,
+		stderr: stderr,
+		err:    err,
+	}
+}
+
+func (c *commandError) Error() string {
+	return c.err.Error()
+}
+
+func (c *commandError) Commandline() string {
+	args := ""
+	if c.scd.args != nil && len(c.scd.args) > 0 {
+		args = fmt.Sprintf(" \"%s\"", strings.Join(c.scd.args, "\" \""))
+	}
+	return fmt.Sprintf("\"%s\"%s", c.scd.command, args)
+}
+
+func (c *commandError) Stderr() string {
+	return c.stderr
+}
+
+func (c *commandError) ExitCode() (int, error) {
+	if exitError, ok := asExitError(c.err); ok {
+		return exitError.ExitCode(), nil
+	} else {
+		return 0, errors.New("exit code not available")
+	}
+}

config/flag.go

@@ -123,6 +123,6 @@ func stringifyValue(value reflect.Value) ([]string, bool) {
 		return stringifyValue(value.Elem())
 
 	default:
-		panic(fmt.Errorf("unexpected type %s", value.Kind()))
+		return []string{fmt.Sprintf("ERROR: unexpected type %s", value.Kind())}, false
 	}
 }

config/flag_test.go

@@ -8,12 +8,11 @@ import (
 	"github.com/stretchr/testify/assert"
 )
 
-func TestPointerValueShouldPanic(t *testing.T) {
+func TestPointerValueShouldReturnErrorMessage(t *testing.T) {
 	concrete := "test"
 	value := &concrete
-	assert.PanicsWithError(t, "unexpected type ptr", func() {
-		stringifyValueOf(value)
-	})
+	argValue, _ := stringifyValueOf(value)
+	assert.Equal(t, []string{"ERROR: unexpected type ptr"}, argValue)
 }
 
 func TestNilValueFlag(t *testing.T) {

config/profile.go

@@ -10,33 +10,36 @@ import (
 
 // Profile contains the whole profile configuration
 type Profile struct {
-	config        *Config
-	Name          string
-	Description   string                    `mapstructure:"description"`
-	Quiet         bool                      `mapstructure:"quiet" argument:"quiet"`
-	Verbose       bool                      `mapstructure:"verbose" argument:"verbose"`
-	Repository    string                    `mapstructure:"repository" argument:"repo"`
-	PasswordFile  string                    `mapstructure:"password-file" argument:"password-file"`
-	CacheDir      string                    `mapstructure:"cache-dir" argument:"cache-dir"`
-	CACert        string                    `mapstructure:"cacert" argument:"cacert"`
-	TLSClientCert string                    `mapstructure:"tls-client-cert" argument:"tls-client-cert"`
-	Initialize    bool                      `mapstructure:"initialize"`
-	Inherit       string                    `mapstructure:"inherit"`
-	Lock          string                    `mapstructure:"lock"`
-	ForceLock     bool                      `mapstructure:"force-inactive-lock"`
-	RunBefore     []string                  `mapstructure:"run-before"`
-	RunAfter      []string                  `mapstructure:"run-after"`
-	RunAfterFail  []string                  `mapstructure:"run-after-fail"`
-	StatusFile    string                    `mapstructure:"status-file"`
-	OtherFlags    map[string]interface{}    `mapstructure:",remain"`
-	Environment   map[string]string         `mapstructure:"env"`
-	Backup        *BackupSection            `mapstructure:"backup"`
-	Retention     *RetentionSection         `mapstructure:"retention"`
-	Check         *OtherSectionWithSchedule `mapstructure:"check"`
-	Prune         *OtherSectionWithSchedule `mapstructure:"prune"`
-	Snapshots     map[string]interface{}    `mapstructure:"snapshots"`
-	Forget        *OtherSectionWithSchedule `mapstructure:"forget"`
-	Mount         map[string]interface{}    `mapstructure:"mount"`
+	config               *Config
+	Name                 string
+	Description          string                    `mapstructure:"description"`
+	Quiet                bool                      `mapstructure:"quiet" argument:"quiet"`
+	Verbose              bool                      `mapstructure:"verbose" argument:"verbose"`
+	Repository           string                    `mapstructure:"repository" argument:"repo"`
+	PasswordFile         string                    `mapstructure:"password-file" argument:"password-file"`
+	CacheDir             string                    `mapstructure:"cache-dir" argument:"cache-dir"`
+	CACert               string                    `mapstructure:"cacert" argument:"cacert"`
+	TLSClientCert        string                    `mapstructure:"tls-client-cert" argument:"tls-client-cert"`
+	Initialize           bool                      `mapstructure:"initialize"`
+	Inherit              string                    `mapstructure:"inherit"`
+	Lock                 string                    `mapstructure:"lock"`
+	ForceLock            bool                      `mapstructure:"force-inactive-lock"`
+	RunBefore            []string                  `mapstructure:"run-before"`
+	RunAfter             []string                  `mapstructure:"run-after"`
+	RunAfterFail         []string                  `mapstructure:"run-after-fail"`
+	StatusFile           string                    `mapstructure:"status-file"`
+	PrometheusSaveToFile string                    `mapstructure:"prometheus-save-to-file"`
+	PrometheusPush       string                    `mapstructure:"prometheus-push"`
+	PrometheusLabels     map[string]string         `mapstructure:"prometheus-labels"`
+	OtherFlags           map[string]interface{}    `mapstructure:",remain"`
+	Environment          map[string]string         `mapstructure:"env"`
+	Backup               *BackupSection            `mapstructure:"backup"`
+	Retention            *RetentionSection         `mapstructure:"retention"`
+	Check                *OtherSectionWithSchedule `mapstructure:"check"`
+	Prune                *OtherSectionWithSchedule `mapstructure:"prune"`
+	Snapshots            map[string]interface{}    `mapstructure:"snapshots"`
+	Forget               *OtherSectionWithSchedule `mapstructure:"forget"`
+	Mount                map[string]interface{}    `mapstructure:"mount"`
 }
 
 // BackupSection contains the specific configuration to the 'backup' command

examples/dev.yaml

@@ -24,6 +24,12 @@ default:
     repository: "/Volumes/RAMDisk/{{ .Profile.Name }}"
     lock: "/Volumes/RAMDisk/resticprofile-{{ .Profile.Name }}.lock"
 
+space:
+    description: Repository contains space
+    initialize: false
+    password-file: key
+    repository: "/Volumes/RAMDisk/with space"
+
 documents:
     inherit: default
     backup:
@@ -80,7 +86,7 @@ self:
     force-inactive-lock: true
     initialize: true
     inherit: default
-    status-file: /Volumes/RAMDisk/status.json
+    # status-file: /Volumes/RAMDisk/status.json
     backup:
         extended-status: false
         check-before: false
@@ -104,6 +110,24 @@ self:
         schedule: "weekly"
         schedule-priority: standard
 
+prom:
+    force-inactive-lock: true
+    initialize: true
+    inherit: default
+    prometheus-save-to-file: "self.prom"
+    prometheus-push: "http://localhost:9091/"
+    prometheus-labels:
+      - host: {{ .Hostname }}
+    status-file: /Volumes/RAMDisk/status.json
+    backup:
+        extended-status: true
+        no-error-on-warning: true
+        source:
+          - {{ .CurrentDir }}
+        {{ template "tags" . }}
+        # exclude:
+        #   - examples/private
+
 system:
     initialize: true
     no-cache: true

go.mod

@@ -14,6 +14,7 @@ require (
 	github.com/hashicorp/go-version v1.3.0 // indirect
 	github.com/mackerelio/go-osstat v0.2.0
 	github.com/mitchellh/mapstructure v1.4.1
+	github.com/prometheus/client_golang v1.11.0
 	github.com/rickb777/date v1.15.3
 	github.com/shirou/gopsutil/v3 v3.21.6
 	github.com/smartystreets/assertions v1.0.0 // indirect