Configuration File

[scoddy]

Configuration File to store the default settings. Just run "restic" to create next snapshot based on settings in config file.

Connect #288

[fd0]

This is a proposal for a configuration file for restic, in TOML (see https://github.com/toml-lang/toml#example for an example).

The configuration file can be used to define global default values and aliases to repositories, which apply configuration values.

[global]

[global.backup]
exclude = [ "*.swp" ]

[gibson]
location = "sftp://gibson.domain.tld//data/backup/restic-repo"

[gibson.backup]
exclude = [ "/home/user/work", "*.go" ]
target = [ "/home/user" ]

[mainframe]
location = "sftp://mainframe.internal//home/user/repo"

[mainframe.backup]
exclude = [ "/tmp", "/proc", "/sys" ]

This leads to the following behaviour:

• restic -r gibson uses sftp://gibson.domain.tld//data/backup/restic-repo as the repository location and the backup exclude list is populatated with the following patterns: *.swp, /home/user/work and *.go. If no other target is specified on the command line, restic creates a backup of /home/user.
• restic -r secondary uses sftp://mainframe.internal//home/user/repo as the repository location and the backup exclude list is populatated with the following patterns: *.swp, /tmp, /proc and /sys
• For every other repository location, restic works as normal, except that for the backup command *.swp is added to any other excludes defined on the command line

So if I have a good config file, creating a backup is a matter of running restic -r gibson backup. :)

Thoughts? @fw42 @scoddy

[ckemper67]

Have you considered using JSON as the format for your configuration files? After all you are already reading and writing JSON for the repository itself. Tools that want to interact with restic will already need to understand JSON etc.

The biggest downside to JSON is that it does not make it easy to add comments to the configuration files

[fd0]

Yes, I've considered and discarded JSON for the config file, writing JSON by hand is tiresome and very error-prone (at least in my experience). I'll probably use HCL by Hashicorp or a custom format for the config file. HCL supports comments and is machine-readable.

[fwilhe]

I think TOML is well suited for this situation because it is easy to read and write for humans and easy to parse (compared to YAML).

[middelink]

+1 for the config but then please add global.forget as well, to supply defaults for forgetting. Any person using restic forget dreads the long line of flags we need to give... :)

btw is the intent to have each restic cmd have it's own sub section, or can we combine them all in one? as long as the flags (now keywords) do not overlap between the cmds, that should work, no?

[global]
exclude = [ "*.swp" ]
forget_daily = 7
forget_weekly= 4
forget_monthly = 36
forget_yearly = 99

[gibson]
location = "sftp://gibson.domain.tld//data/backup/restic-repo"
exclude = [ "/home/user/work", "*.go" ]
target = [ "/home/user" ]

[mainframe]
location = "sftp://mainframe.internal//home/user/repo"
exclude = [ "/tmp", "/proc", "/sys" ]

Ok, you prob don't like that :) Lets just keep the forget_xxx's

[Edit] Oh, nasty. How do I remove a global setting? Say I do need swapfiles in mainframe. Something like no_inherit = [ "exclude" ] perhaps?

[jniggemann]

Like fwilhe (and for the very same reasons), I think TOML is well suited for this.

I also think that this issue is important: Since obnam was retired a couple of weeks ago, many obnam users either switch to borg or restic. I am one of those, currently making the transition fro obnam to restic and the support for a configuration file is what I'm missing most.
That, and compression :-)

Thank you for your work!

[leak]

Is there any progress in this area? Gentoo just masked obnam for removal and this is the No. 1 missing feature to choose restic as replacement.

[fd0]

No progress yet, sorry.

[leak]

Sample implementation for config file support:

Dislcaimer: This is my first time touching Go, so there might be dragons.

https://github.com/leak/restic/tree/config-file

Quick facts:
https://github.com/leak/restic/commit/a657466ae9f6daa7e46e2c20e5c19b6c5a6a1ede

Concerns:
spf13/viper is a feature monster, probably way beyond what restic requires, so the dependencies it pulls in are substantial, see here
https://github.com/leak/restic/commit/9d64bbd96619fe97a4f63c5f41b68221aba4a83c

Proposal:
What I really liked about obnam was its config file behavior. Basically you would create a /etc/obnam.conf, the cli would automatically pick up that file and you could issue obnam commands anywhere without specifying a repository or config file on every single command you use.

I wanted to continue and enhance this experience, thus my proposal:

A single config file per repository paired with default config search behavior, enabling the following scenarios:

Single repository:

• Configure your repository either in /etc/restic/restic.toml or ~/.restic/restic.toml
• Use any restic command anywhere without ever specifying the repo, the password or the config file location again

Multi repository:

• The directory containing the restic.toml config will now become your repository context, containing config, password-file, cacert, etc.
• Configure multiple config files, e.g. /etc/restic/nas/restic.toml, /etc/restic/amazon-aws/restic.toml, ...
• Drop into a given directory of the repository you want to work with cd /etc/restic/nas/
• Use any restic command without specifiying repo, pass or config-file

The --config-file parameter will allow you to override the default search behavior, so even if you are in say ~/.restic and you run restic --config-file ~/somewhere/magic-restic.toml it would work as expected.

[fd0]

Hm, nice idea, let me think about it. What I had in mind for the config file would be to be able to configure multip repositories, with a name for each repo. So you could have a repo config nas that you can use with restic --repo nas and it would then pick up all settings from the config file. One repo then could be the default, when you don't set --repo, it'll use the default one.

I'm still busy with #1494, but I'll think about it. :)

[evgeni]

I must admit I don't like the context-awareness of the multi-repo proposal.

Tools like Git have this feature, but there you are actually inside the repo when it loads the repo-specific configuration parts, not inside a folder that (maybe) refers to a repo, but is not by design the single-point-of-entry to that repo.

I think I like @fd0's idea with --repo nas to load /etc/restic/nas/restic.toml better. But maybe call it --repo-config or --repo-profile so that it does not clash with the folder ./nas that is the actual repo?

[fd0]

I think I like @fd0's idea with --repo nas to load /etc/restic/nas/restic.toml better. But maybe call it --repo-config or --repo-profile so that it does not clash with the folder ./nas that is the actual repo?

That's easy to solve: The name may not start with a . or /, and you can always use --repo local:foobar or --repo ./foobar to access a local repo in the folder foobar.

[evgeni]

Nah, ./repo is ugly, just think how often everyone copied the wrong stuff with rsync with its special-casing of foo vs foo/.

local:foo sounds sane, tho

[leak]

I like the git client analogy a lot. You can just drop into a folder and use commands without specifying the same parameters over and over again, or alternatively, scroll back and edit your previously issued commands so you can avoid retyping the same things again. To me this is very user unfriendly.

Btw. this feature only requires for restic to look for a "restic.toml" in the current working dir, nothing more or less, it could perfectly live side-by-side with all other proposed options.

As for the --repo nas suggestion. I'm not a fan of the reuse of the --repo flag, to me flags should have a dedicated purpose to avoid confusion or parsing nightmares/limitations.

Maybe a new flag could be called --named-repo nas or short--nr nas loading from /etc/restic/nas/restic.toml

[fd0]

Let's talk about that when we get there.

[maxstoyanov]

I just want to drop some lines about my usecase here:
For server backups I need to automate everything. Not only doing file but also database backups. Having to script everything together in one ugly shell script is ugly. I would really enjoy having a restic native configuration file that works cross-platform.

• Multiple repositories
• Multiple folders
• Input via stdin piping (e.g. for mysqldump)
• Forget policies
• All config and secret related variables (read: no need for environment variable)

As config format is toml preferrable. YAML is too complicated for many users and toml feels more natural (looks like ini).

[leak]

So it's been a while already. Meanwhile I'm using restic in more and more places across multiple OS without any problems, which is very pleasant.

The only thing that keeps coming up is the lack of a configuration file that would remove the need for shell scripts and whatnot.

I got a bit spare time on my hands. I could turn my sample implementation above into a pull request. I believe my implementation could already cover a lot of use cases and the advanced features could be handled/discussed in a new github issue.

Would that potential pull request have a chance?

[fd0]

Thank you very much for the offer (and for asking before spending more time on it)! I think I'd like to try this myself, using HCL instead of TOML and avoiding viper. I've looked at it and I think it has way more features than what we want.

[avimar]

I have all my secrets in a non-checked in secrets file, in JSON format. So I wrote a simple wrapper to pull those out and export it, keeping all my secrets in one place.

Lightweight setting wrapper to set the environment variables via a JSON file config file using fx

Dependency: npm install -g fx

#!/bin/bash

cd /your/working/directory || exit

#Not a secret, just hard code it if you want.
export RESTIC_REPOSITORY="s3:https://s3.amazonaws.com/your-db-or-configure" 
export AWS_ACCESS_KEY_ID=$(fx common.json "d => d['s3Backup']['accessKeyId']")
export AWS_SECRET_ACCESS_KEY=$(fx common.json "d => d['s3Backup']['secretAccessKey']")
export RESTIC_PASSWORD=$(fx common.json "d => d['RESTIC_PASSWORD']")

#https://stackoverflow.com/a/1696062/1278519 -- exec means just hand the process over, less chance of zombie processes
#POTENTIAL GOTCHA: nothing can be executed after this line!

#Pass all args on, we're just a wrapper to send environment variables
exec restic "$@"

EDIT: This didn't work as a wrapper, when used in a pipe.
When I piped from mongodump, I guess the pipe wasn't ready yet?
So I just put that code into my original mongodump bash script.

Suggestions for fixing this are welcome...

[kam1sh]

I understand that now there is no RFC/feedback needed labels, but if you think of the restic as a "low-level" backup program, then configuration should be done either in some higher-level language.

I tried to design restic configuration in toml, but it's kinda hard because nested mappings in toml is too much verbose IMO. But yaml is nice both to write and read, I think:

repos: 
  host:
    path: /backups/host
    password-file: /mnt/host-password.txt

global:
  repo: host
  ignore:
    caches: true

jobs:
  etc: /etc
  home:
    path: /home
    exclude: .local/share/Steam
  postgres:
    cmd: sudo -u postgres pg_dumpall
    save-as: pgdumpall.bin

But I see that there are people who don't like yaml. Then, maybe, we should look at interpretable languages, like Python?

from resticrc.models import Repository, Job, FileRunner

repo = Repository(name="host", path="/backups/host")

job = Job(
    repo=repo,
    tag="gitea",
    runner=FileRunner(paths=["/home/git", "/var/lib/gitea"]),
    exclude={"logs": True}
)
job.run()

[mjameswh]

So, I came here, too, looking for configuration file supports… Well, it isn’t ready yet, too bad. But I want this to be very clear: yes, that is perfectly fair, for this is Open Source. I haven't yet put a single ounce of effort or money in developing this great software, so please: no guilt anyone.

Now, I have just gone through this thread several times, as well as several others threads related to it and to restic development in general. I have also looked at most propositions and many existing wrappers. And I think it might be helpful to share some insights I gained from this exercise.

First, the existence of all these wrappers clearly demonstrates that strictly speaking, restic does not need native support for configuration files. Of course, this appears to be desirable, but if there are so many wrappers providing support for configuration files over restic, it is indeed because it is reasonably easy to do so. In comparison, compression can’t be implemented outside of restic’s core, and purge’s performance can’t be improved by an external command. But configuration files support can. Therefore, in term of development priority, it is not unreasonable for configuration files support to be in the backlog.

Second, most propositions I have seen differ essentially by their configuration languages, and most existing wrappers differ mainly by their programming languages and by their implementers. But aside from these minor differences, I think most suggestions and existing wrappers are essentially equivalent from a functional point of view (that is, with the notable exception of schedic and replica, which both store configuration files in the cloud and internally handle scheduling of backup execution).

Then why isn’t everyone converging to a unique configuration wrapper? That’s a broad question, indeed, but I personally think that it is in large part because there is no clear winner, and therefore, no payback to convert existing configurations and scripts that are working. Everybody is waiting for an “official” solution to move on… A solution that has received agreement from restic’s core developers, and that will be maintained in future restic development.

On the opposite side, there are also use cases for which configuration files as those envisioned for restic will simply never be fit… That is notably the case of schedic and replica, but there will certainly be many others. For these use cases, providing parameters through environment variables and/or command line arguments is just fine.

Now, from the discussion in this thread, it appears that @fd0 has some reasonable ideas of how he would like the configuration language to be designed. I must admit that, personally, I am not a huge fan of the HCL syntax, but my opinion on this matter doesn’t count, and it is true that HCL would indeed make restic’s configuration much more powerful. But in the end, what really matter is that 1) community efforts should be put on converging toward a unique configuration language, 2) that this configuration language must have support from @fd0 and other core developers, and 3) that a usable prototype of that configuration language be developed.

There is however nothing that require that the configuration language should be designed by @fd0 himself or any of the other core developers (they have other, more important things to do, whatever they are, and we must respect that). Core developers certainly must be in the loop, because obtaining their agreement is critical, but they should not be the ones doing the hard work on that.

Also, the prototype for that configuration language can very well be implemented as a wrapper over restic, rather than being implemented in restic’s source code. Tens of wrappers have already proven that this is possible, so why not do it for a prototype of the official configuration file support if that means we can get a working implementation much sooner, with very little effort from the core developers? Obviously, it would be highly desirable for that prototype to be constructed on the same technologies as restic itself (so that this wrapper can actually be used by anyone who would be using restic, without any supplementary requirement), but the code can be written by distinct, capable developers, without involving reviews by restic’s core developers.

Once this is done, this ticket can finally be closed, and everyone will be invited to start using the new “official” configuration file support, based on the “official” wrapper, instead of those numerous work around and third-party wrappers. All of this can be done even while restic’s code is in flux, since this is a distinct, external code base.

Then, eventually, once restic’s code and the configuration language have both stabilized, and free time flows in (ok, maybe I’m exaggerating a little bit on that one…), well, at that time, restic’s core developers shall take decisions on how to best integrate both softwares. Maybe they will choose to rebundle the configuration-file aware cli as a module to be import from restic’s core… Or maybe the opposite: the wrapper could be promoted to become the actual restic binary, importing the restic’s core module… For sure, there will be several reasonable options at that time. And what could be the worst case? Well, if the prototype’s source code isn’t clean enough, it can all be rewritten from scratch, at that time… That would still be a win compared to have no solution at all until then.

So this was my two cents.

@fd0, I think this has to be your decision. Does that all make sense to you? Do you feel comfortable with what I propose? There are certainly a few details I might have missed… After all, I have known this project for only a few days yet, so I’ll stay very humble if there are factors I have not considered… But I truly believe that it would be in everyone’s interest if we can get this ticket to move on without involving you more than necessary.

James Watkins-Harvey

[nils-werner]

Maybe also, instead of coming up with a solution to all problems at once and at the same time, how about dividing the problem into its components and tackling one at a time? I.e. we could do

1. Formulate a problem statement
2. Come up with a config schema
3. Decide on the language
4. Do the actual implementation

[creativeprojects]

I was looking at spf13/viper the other day. The best thing about it is that it can load all sort of configuration file formats out of the box. So the user would be able to choose his favourite format.

I'm quite tempted to rewrite resticprofile in go using viper. Mostly for the fun of it (and also to learn a bit more of go) but also because maybe it makes sense to keep the tool and the configuration separated after all.

[nils-werner]

I decided, just for good measure, to write my own super simple configuration wrapper, crestic :-)

[PrplHaz4]

Another solution is here: autorestic

It seems crestic may be the simplest approach - not needing updates every time the restic api moves forward...

[mjameswh]

In case it can help, I have published a few notes I had on an HCL configuration language for restic. I used HCL because I was hoping that we could finally all agree on a syntax that would get @fd0's official support… But I don't know if he agrees on this direction…

https://gist.github.com/mjameswh/9e9f7ddfdcdc2e5f30c4547ad72b402e

[nils-werner]

For such a simple prototype I must say I am really happy with the way crestic works, and I am using it daily now.

It's an almost-one-click install with no dependencies, does not try to be too clever, speaks a language you already know (the restic CLI parameters become keys in an INI file), and parameters are easily overrideable when you need to.

[dimejo]

@nils-werner:

For such a simple prototype I must say I am really happy with the way crestic works, and I am using it daily now.

If you have the time please create a post on the forum to introduce your project.

[fancsali]

I really like @nils-werner's approach with crestic. Big thanks!

Also, I can't wait to get this as a native restic feature... 😉

[lucmos]

I like crestic too!

However, it does not allow the cd folder; restic backup . workflow suggested here, to remove leading paths.

[dimejo]

However, it does not allow the cd folder; restic backup . workflow suggested here, to remove leading paths.

@aawsome proposed PR #3200 just 2 days ago, which allows to set a custom path and make this workaround obsolete. Let's hope it doesn't take too long for it to get merged. 😄

[nettnikl]

@lucmos
Was also looking for this feature - the dev implemented it lately, you might want to check out the newer versions of crestic!

[lucmos]

@nettnikl thanks for the heads up!

Do you mean in crestic? Because I am not finding anything relative to that

[nettnikl]

@lucmos Yes, in crestic. Was talking to the dev about this exact issue and later i found you asking for the exact same feature. It's not yet megred, but i invite you to join the discussion, a working branch can also be found there!

[nils-werner]

However, it does not allow the cd folder; restic backup . workflow suggested here, to remove leading paths.

Are you sure? I don't think that should be the case. Can you please open an issue in https://github.com/nils-werner/crestic to help me reproduce this?

Do you mean in crestic? Because I am not finding anything relative to that

There is a dedicated issue and branch for this.

[ogarcia]

One possible current restic configuration option is to simply use bash scripting. For example, using pass to fetch secrets, put this into your .bash_profile or .bashrc.

_config-restic () {
  local _pass_file="${1}"
  local _pass_contents="$(pass ${_pass_file})"
  export AWS_ACCESS_KEY_ID=$(echo ${_pass_contents} | sed -e '/^aws_access_key_id:/s/aws_access_key_id:[[:space:]]*//p' -e d)
  export AWS_SECRET_ACCESS_KEY=$(echo ${_pass_contents} | sed -e '/^aws_secret_access_key:/s/aws_secret_access_key:[[:space:]]*//p' -e d)
  export RESTIC_REPOSITORY=$(echo ${_pass_contents} | sed -e "/^${2}:/s/${2}:[[:space:]]*//p" -e d)
  export RESTIC_PASSWORD=$(echo ${_pass_contents} | head -1)
}

unconfig-restic () {
  unset AWS_ACCESS_KEY_ID
  unset AWS_SECRET_ACCESS_KEY
  unset RESTIC_REPOSITORY
  unset RESTIC_PASSWORD
}

_mount-restic () {
  _config-restic ${@}
  local _temp_dir="$(mktemp -d --tmpdir restic.XXX)"
  echo "Backup will be mounted in ${_temp_dir}"
  restic mount ${_temp_dir}
  rmdir ${_temp_dir}
  unconfig-restic
}

# Configuration aliases
alias -- config-restic-desktop='_config-restic Restic desktop_repository'
alias -- config-restic-server='_config-restic Restic server_repository'

# Mount aliases
alias -- mount-restic-desktop='_mount-restic Restic desktop_repository'
alias -- mount-restic-server='_mount-restic Restic server_repository'

The Restic pass file would look like this:

repository_password
---
aws_access_key_id: key_id
aws_secret_access_key: secret
desktop_repository: s3:https://server:port/bucket
server_repository: s3:https://other/bucket
sftp_repository: sftp:server:/path/to/repo

As you can see you can even use different repository backends since restic simply ignores environment variables that it does not need. Moreover, this is extensible to as many environment variables as needed.

Then to use it, simply run:

$ config-restic-desktop
$ restic snapshots # or any restic command
$ unconfig-restic # if you want to clear environment variables (optional)

Or for fast mount (after umount or Ctrl-c the tempdir is deleted 😉):

$ mount-restic-desktop
Backup will be mounted in /tmp/restic.o0o
repository bebecafe00 opened (version 2, compression level auto)
[0:00] 100.00%  666 / 666 index files loaded
Now serving the repository at /tmp/restic.o0o
Use another terminal or tool to browse the contents of this folder.
When finished, quit with Ctrl-c here or umount the mountpoint.

Another option that can also be used (pure bash script and without pass) is a small script that I have created for Arch Linux but that can be used in any Linux distribution, see here for full info.