Added resticprofile flags --no-lock & --lock-wait (#33)

* Added resticprofile flags --no-lock & --lock-wait

* Handle lock-wait and stale locks for restic

"--lock-wait" also waits on non-stale restic remote locks
Stale locks are unlocked when "force-inactive-locks" is true
Uses global section to configure the behaviour (can also be disabled)

* Allow to set lock-wait and no-lock within schedule

README.md

@@ -421,6 +421,7 @@ source = [ "/" ]
 # if scheduled, will run every day at midnight
 schedule = "daily"
 schedule-permission = "system"
+schedule-lock-wait = "2h"
 # run this after a backup to share a repository between a user and root (via sudo)
 run-after = "chown -R $SUDO_USER $HOME/.cache/restic /backup"
 # ignore restic warnings (otherwise the backup is considered failed when restic couldn't read some files)
@@ -466,6 +467,7 @@ run-after = "sync"
 # if scheduled, will run every 30 minutes
 schedule = "*:0,30"
 schedule-permission = "user"
+schedule-lock-wait = "10m"
 
 # retention policy for profile src
 [src.retention]
@@ -622,7 +624,8 @@ A few environment variables will be set before running these commands:
 Additionally for the `run-after-fail` commands, these environment variables will also be available:
 - `ERROR` containing the latest error message
 - `ERROR_COMMANDLINE` containing the command line that failed
-- `RESTIC_STDERR` containing any message that restic sent to the standard error (stderr)
+- `ERROR_EXIT_CODE` containing the exit code of the command line that failed
+- `ERROR_STDERR` containing any message that the failed command sent to the standard error (stderr)
 
 ## run before and after order during a backup
 
@@ -678,16 +681,50 @@ For this profile, a lock will be set using the file `/tmp/resticprofile-profile-
 
 **Please note restic locks and resticprofile locks are completely independent**
 
-In some cases, you might want to override the resticprofile lock if the process died (or the machine rebooted) leaving a lockfile behind.
+## Stale locks
 
-For that matter, if you add the flag `force-inactive-lock` to your profile, resticprofile will check for the presence of a process with the PID indicated in the lockfile. If it can't find any, it will try to delete the lock and continue the operation (locking again, running profile and so on...)
+In some cases, resticprofile as well as restic may leave a lock behind if the process died (or the machine rebooted).
+
+For that matter, if you add the flag `force-inactive-lock` to your profile, resticprofile will detect and remove stale locks: 
+* **resticprofile locks**: Check for the presence of a process with the PID indicated in the lockfile. If it can't find any, it will try to delete the lock and continue the operation (locking again, running profile and so on...)
+* **restic locks**: Evaluate if a restic command failed on acquiring a lock. If the lock is older than `restic-stale-lock-age`, invoke `restic unlock` and retry the command that failed (can be disabled by setting `restic-stale-lock-age` to 0, default is 2h).
 
 ```yaml
+global:
+  restic-stale-lock-age: 2h
+
 src:
     lock: "/tmp/resticprofile-profile-src.lock"
     force-inactive-lock: true
 ```
 
+## Lock wait
+
+By default, restic and resticprofile fail when a lock cannot be acquired as another process is currently holding it.
+
+Depending on the use case (e.g. scheduled backups), it may be more appropriate to wait on another process to finish instead of failing immediately.
+
+For that matter, if you add the commandline flag `--lock-wait` or configure schedules with `schedule-lock-wait`, resticprofile will wait on other backup processes:
+* **resticprofile locks**: Retry acquiring the lockfile until it either succeeds (when the other resticprofile process released the lock) or fail as the lock-wait duration has passed without success.
+* **restic locks**: Evaluate if a restic command failed on acquiring a lock. If the lock is not considered stale, retry the restic command every `restic-lock-retry-after` (default 1 minute) until it acquired the lock, or fail as the lock-wait duration has passed.
+
+Note: The lock wait duration is cumulative. If various locks in one profile-run require lock wait, the total wait time may not exceed the duration that was specified. 
+
+## restic lock management
+
+resticprofile can retry restic commands that fail on acquiring a lock and can also ask restic to unlock stale locks. The behaviour is controlled by 2 settings inside the `global` section:
+
+```yaml
+global:
+  # Retry a restic command that failed on acquiring a lock every minute 
+  # (at least), for up to the time specified in "--lock-wait duration". 
+  restic-lock-retry-after: 1m
+  # Ask restic to unlock a stale lock when its age is more than 2 hours
+  # and the option "force-inactive-lock" is enabled in the profile.
+  restic-stale-lock-age: 2h
+```
+
+If restic lock management is not desired, it can be disabled by setting both values to 0.
 
 # Using resticprofile
 
@@ -745,19 +782,21 @@ Usage of resticprofile:
 	resticprofile [resticprofile flags] [resticprofile command] [command specific flags]
 
 resticprofile flags:
-  -c, --config string   configuration file (default "profiles")
-      --dry-run         display the restic commands instead of running them
-  -f, --format string   file format of the configuration (default is to use the file extension)
-  -h, --help            display this help
-  -l, --log string      logs into a file instead of the console
-  -n, --name string     profile name (default "default")
-      --no-ansi         disable ansi control characters (disable console colouring)
-      --no-prio         don't set any priority on load: used when started from a service that has already set the priority
-  -q, --quiet           display only warnings and errors
-      --theme string    console colouring theme (dark, light, none) (default "light")
-      --trace           display even more debugging information
-  -v, --verbose         display some debugging information
-  -w, --wait            wait at the end until the user presses the enter key
+  -c, --config string        configuration file (default "profiles")
+      --dry-run              display the restic commands instead of running them
+  -f, --format string        file format of the configuration (default is to use the file extension)
+  -h, --help                 display this help
+      --lock-wait duration   wait up to duration to acquire a lock (syntax "1h5m30s")
+  -l, --log string           logs into a file instead of the console
+  -n, --name string          profile name (default "default")
+      --no-ansi              disable ansi control characters (disable console colouring)
+      --no-lock              skip profile lock file
+      --no-prio              don't set any priority on load: used when started from a service that has already set the priority
+  -q, --quiet                display only warnings and errors
+      --theme string         console colouring theme (dark, light, none) (default "light")
+      --trace                display even more debugging information
+  -v, --verbose              display some debugging information
+  -w, --wait                 wait at the end until the user presses the enter key
 
 resticprofile own commands:
    version       display version (run in verbose mode for detailed information)
@@ -787,8 +826,10 @@ There are not many options on the command line, most of the options are in the c
 * **[-q | --quiet]**: Force resticprofile and restic to be quiet (override any configuration from the profile)
 * **[-v | --verbose]**: Force resticprofile and restic to be verbose (override any configuration from the profile)
 * **[--no-ansi]**: Disable console colouring (to save output into a log file)
+* **[--no-lock]**: Disable resticprofile locks, neither create nor fail on a lock. restic locks are unaffected by this option.
 * **[--theme]**: Can be `light`, `dark` or `none`. The colours will adjust to a 
 light or dark terminal (none to disable colouring)
+* **[--lock-wait] duration**: Retry to acquire resticprofile and restic locks for up to the specified amount of time before failing on a lock failure. 
 * **[-l | --log] log_file**: To write the logs in file instead of displaying on the console
 * **[-w | --wait]**: Wait at the very end of the execution for the user to press enter. This is only useful in Windows when resticprofile is started from explorer and the console window closes automatically at the end.
 * **[resticprofile OR restic command]**: Like snapshots, backup, check, prune, forget, mount, etc.
@@ -869,7 +910,10 @@ The schedule configuration consists of a few parameters which can be added on ea
 [profile.backup]
 schedule = "*:00,30"
 schedule-permission = "system"
+schedule-priority = "background"
 schedule-log = "profile-backup.log"
+schedule-lock-mode = "default"
+schedule-lock-wait = "15m30s"
 ```
 
 
@@ -884,6 +928,18 @@ schedule-log = "profile-backup.log"
 
 * *empty*: resticprofile will try its best guess based on how you started it (with sudo or as a normal user) and fallback to `user`
 
+### schedule-lock-mode
+
+Starting from version 0.14.0, `schedule-lock-mode` accepts 3 values:
+- `default`: Wait on acquiring a lock for the time duration set in `schedule-lock-wait`, before failing a schedule.
+   Behaves like `fail` when `schedule-lock-wait` is "0" or not specified.
+- `fail`: Any lock failure causes a schedule to abort immediately. 
+- `ignore`: Skip resticprofile locks. restic locks are not skipped and can abort the schedule.
+
+### schedule-lock-wait
+
+Sets the amount of time to wait for a resticprofile and restic lock to become available. Is only used when `schedule-lock-mode` is unset or `default`.
+
 ### schedule-log
 
 Allow to redirect all output from resticprofile and restic to a file
@@ -957,15 +1013,20 @@ default:
 
 self:
     inherit: default
+    retention:
+      after-backup: true
+      keep-within: 14d
     backup:
         source: "."
         schedule:
         - "Mon..Fri *:00,15,30,45" # every 15 minutes on weekdays
         - "Sat,Sun 0,12:00"        # twice a day on week-ends
         schedule-permission: user
-    retention:
+        schedule-lock-wait: 10m
+    prune:
         schedule: "sun 3:30"
         schedule-permission: user
+        schedule-lock-wait: 1h
 ```
 
 ## Scheduling commands
@@ -1123,9 +1184,11 @@ test1:
         source: ./
         schedule: "*:00,15,30,45"
         schedule-permission: user
+        schedule-lock-wait: 15m
     check:
         schedule: "*-*-1"
         schedule-permission: user
+        schedule-lock-wait: 15m
 
 ```
 
@@ -1820,6 +1883,8 @@ None of these flags are passed on the restic command line
 * **default-command**: string
 * **initialize**: true / false
 * **restic-binary**: string
+* **restic-lock-retry-after**: duration
+* **restic-stale-lock-age**: duration
 * **min-memory**: integer (MB)
 * **scheduler**: string (`crond` is the only non-default value)
 
@@ -1867,6 +1932,8 @@ Flags used by resticprofile only
 * **check-after**: true / false
 * **schedule**: string OR list of strings
 * **schedule-permission**: string (`user` or `system`)
+* **schedule-lock-mode**: string (`default`, `fail` or `ignore`) 
+* **schedule-lock-wait**: duration 
 * **schedule-log**: string
 * **extended-status**: true / false
 * **no-error-on-warning**: true / false
@@ -1899,6 +1966,8 @@ Flags used by resticprofile only
 * **after-backup**: true / false
 * **schedule**: string OR list of strings
 * **schedule-permission**: string (`user` or `system`)
+* **schedule-lock-mode**: string (`default`, `fail` or `ignore`)
+* **schedule-lock-wait**: duration
 * **schedule-log**: string
 
 Flags passed to the restic command line
@@ -1936,6 +2005,8 @@ Flags used by resticprofile only
 
 * **schedule**: string OR list of strings
 * **schedule-permission**: string (`user` or `system`)
+* **schedule-lock-mode**: string (`default`, `fail` or `ignore`)
+* **schedule-lock-wait**: duration
 * **schedule-log**: string
 
 Flags passed to the restic command line
@@ -1962,6 +2033,8 @@ Flags used by resticprofile only
 
 * **schedule**: string OR list of strings
 * **schedule-permission**: string (`user` or `system`)
+* **schedule-lock-mode**: string (`default`, `fail` or `ignore`)
+* **schedule-lock-wait**: duration
 * **schedule-log**: string
 
 Flags passed to the restic command line
@@ -1977,6 +2050,8 @@ Flags used by resticprofile only
 
 * **schedule**: string OR list of strings
 * **schedule-permission**: string (`user` or `system`)
+* **schedule-lock-mode**: string (`default`, `fail` or `ignore`)
+* **schedule-lock-wait**: duration
 * **schedule-log**: string
 
 `[profile.mount]`

config/config.go

@@ -43,8 +43,14 @@ type Config struct {
 // For that matter, viper creates a slice of maps instead of a map for the other configuration file formats
 // This configOptionHCL deals with the slice to merge it into a single map
 var (
-	configOption    = viper.DecodeHook(nil)
-	configOptionHCL = viper.DecodeHook(sliceOfMapsToMapHookFunc())
+	configOption = viper.DecodeHook(mapstructure.ComposeDecodeHookFunc(
+		mapstructure.StringToTimeDurationHookFunc(),
+	))
+
+	configOptionHCL = viper.DecodeHook(mapstructure.ComposeDecodeHookFunc(
+		mapstructure.StringToTimeDurationHookFunc(),
+		sliceOfMapsToMapHookFunc(),
+	))
 )
 
 // newConfig instantiate a new Config object
@@ -212,7 +218,7 @@ func (c *Config) getCommandListHCL(sectionRawValue interface{}) []string {
 
 // GetGlobalSection returns the global configuration
 func (c *Config) GetGlobalSection() (*Global, error) {
-	global := newGlobal()
+	global := NewGlobal()
 	err := c.unmarshalKey(constants.SectionConfigurationGlobal, global)
 	if err != nil {
 		return nil, err

config/flag.go

@@ -53,10 +53,24 @@ func stringifyValueOf(value interface{}) ([]string, bool) {
 
 // stringifyValue returns a string representation of the value, and if it has any value at all
 func stringifyValue(value reflect.Value) ([]string, bool) {
+	// Check if the value can convert itself to String() (e.g. time.Duration)
+	stringer := fmt.Stringer(nil)
+	if value.CanInterface() {
+		vi := value.Interface()
+		if s, ok := vi.(fmt.Stringer); ok {
+			stringer = s
+		}
+	}
+
+	var stringVal string
 
 	switch value.Kind() {
 	case reflect.String:
-		stringVal := value.String()
+		if stringer != nil {
+			stringVal = stringer.String()
+		} else {
+			stringVal = value.String()
+		}
 		return []string{stringVal}, stringVal != ""
 
 	case reflect.Bool:
@@ -65,17 +79,29 @@ func stringifyValue(value reflect.Value) ([]string, bool) {
 
 	case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
 		intVal := value.Int()
-		stringVal := strconv.FormatInt(intVal, 10)
+		if stringer != nil {
+			stringVal = stringer.String()
+		} else {
+			stringVal = strconv.FormatInt(intVal, 10)
+		}
 		return []string{stringVal}, intVal != 0
 
 	case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
 		intVal := value.Uint()
-		stringVal := strconv.FormatUint(intVal, 10)
+		if stringer != nil {
+			stringVal = stringer.String()
+		} else {
+			stringVal = strconv.FormatUint(intVal, 10)
+		}
 		return []string{stringVal}, intVal != 0
 
 	case reflect.Float32, reflect.Float64:
 		floatVal := value.Float()
-		stringVal := strconv.FormatFloat(floatVal, 'f', -1, 64)
+		if stringer != nil {
+			stringVal = stringer.String()
+		} else {
+			stringVal = strconv.FormatFloat(floatVal, 'f', -1, 64)
+		}
 		return []string{stringVal}, floatVal != 0
 
 	case reflect.Slice, reflect.Array:

config/global.go

@@ -1,30 +1,36 @@
 package config
 
 import (
+	"time"
+
 	"github.com/creativeprojects/resticprofile/constants"
 )
 
 // Global holds the configuration from the global section
 type Global struct {
-	IONice         bool   `mapstructure:"ionice"`
-	IONiceClass    int    `mapstructure:"ionice-class"`
-	IONiceLevel    int    `mapstructure:"ionice-level"`
-	Nice           int    `mapstructure:"nice"`
-	Priority       string `mapstructure:"priority"`
-	DefaultCommand string `mapstructure:"default-command"`
-	Initialize     bool   `mapstructure:"initialize"`
-	ResticBinary   string `mapstructure:"restic-binary"`
-	MinMemory      uint64 `mapstructure:"min-memory"`
-	Scheduler      string `mapstructure:"scheduler"`
+	IONice               bool          `mapstructure:"ionice"`
+	IONiceClass          int           `mapstructure:"ionice-class"`
+	IONiceLevel          int           `mapstructure:"ionice-level"`
+	Nice                 int           `mapstructure:"nice"`
+	Priority             string        `mapstructure:"priority"`
+	DefaultCommand       string        `mapstructure:"default-command"`
+	Initialize           bool          `mapstructure:"initialize"`
+	ResticBinary         string        `mapstructure:"restic-binary"`
+	ResticLockRetryAfter time.Duration `mapstructure:"restic-lock-retry-after"`
+	ResticStaleLockAge   time.Duration `mapstructure:"restic-stale-lock-age"`
+	MinMemory            uint64        `mapstructure:"min-memory"`
+	Scheduler            string        `mapstructure:"scheduler"`
 }
 
-// newGlobal instantiates a new Global with default values
-func newGlobal() *Global {
+// NewGlobal instantiates a new Global with default values
+func NewGlobal() *Global {
 	return &Global{
-		IONice:         constants.DefaultIONiceFlag,
-		Nice:           constants.DefaultStandardNiceFlag,
-		DefaultCommand: constants.DefaultCommand,
-		ResticBinary:   constants.DefaultResticBinary,
-		MinMemory:      constants.DefaultMinMemory,
+		IONice:               constants.DefaultIONiceFlag,
+		Nice:                 constants.DefaultStandardNiceFlag,
+		DefaultCommand:       constants.DefaultCommand,
+		ResticBinary:         constants.DefaultResticBinary,
+		ResticLockRetryAfter: constants.DefaultResticLockRetryAfter,
+		ResticStaleLockAge:   constants.DefaultResticStaleLockAge,
+		MinMemory:            constants.DefaultMinMemory,
 	}
 }