Pattern docs are misleading/wrong on how relative patterns work in practice

[Atemu]

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

ISSUE

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.2.6

Operating system (distribution) and version.

NixOS 24.05

Hardware / network configuration, and filesystems used.

x86_64, local, tmpfs

How much data is handled by borg?

0B

Full borg commandline that lead to the problem (leave away excludes and passwords)

(Explained below.)

Describe the problem you're observing.

The docs state that:

The path/filenames used as input for the pattern matching start from the currently active recursion root.

Thus if I wanted to back up /tmp/foo but not /tmp/foo/d1, I'd expect the following pattern file to work:

R /tmp/foo
- d1

It does not however:

$ borg create --list --patterns-from /tmp/patterns.lst /tmp/borg/::{now}
d /tmp/foo/d1
d /tmp/foo

In order to exclude d1, I must specify its full path relative to /, rather than the recursion root /tmp/foo:

R /tmp/foo
- tmp/foo/d1

This is not "relative to the currently active recursion root", it's relative to /; unconditionally.

At this point I might aswell specify its full path, so I'm a bit confused why relative paths are even a thing when they're not actually relative to the recursion root?

I hope you can see how this is extremely confusing. The docs should be changed to reflect the actual behaviour and perhaps relative paths should be removed/disallowed until they actually do something that an absolute path wouldn't.

At the very least, the example pattern file should be marked with a note that relative paths only work the way they're portrayed when the root is /, not in any other case, as it also suggests the patterns work like described in the first sentence.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

mkdir -p /tmp/foo/d1
mkdir /tmp/borg
cd /tmp/borg
borg init -e none .
# create patterns files above
borg create --list --patterns-from /tmp/patterns.lst /tmp/borg/::{now}

Include any warning/errors/backtraces from the system logs

[ThomasWaldmann]

Did you read the next sentence from the docs after the one you quoted?

It should make clear how that is meant.

[jdchristensen]

I agree that the docs are confusing. The first two paragraph are:

The path/filenames used as input for the pattern matching start from the currently active recursion root. You usually give the recursion root(s) when invoking borg and these can be either relative or absolute paths.

Starting with Borg 1.2, paths that are matched against patterns always appear relative. If you give /absolute/ as root, the paths going into the matcher will start with absolute/. If you give ../../relative as root, the paths will be normalized as relative/.

I don't know what the first sentence is trying to convey, as I don't know what it means for a path to "start from" a root. The first sentence of the second paragraph is also potentially misleading. Saying that the paths are "relative" suggests that they are relative to the roots just mentioned, but they are not. How about something like this:

Borg generates paths to back up using the recursion roots and matches these paths against patterns. The recursion roots can be given as either relative or absolute paths, and are included in the paths that are matched against patterns, except that leading slashes and leading occurrences of ../ are removed. For example, if you give /absolute/path/ as a root, then the paths going into the matcher will start with absolute/path/. If you give relative/path/ as a root, then the paths going into the matcher will start with relative/path/. And if you give ../../relative/path/ as a root, then the paths going into the matcher will start with relative/path/.

I also recommend including an example of a relative root, e.g.

# An example with a relative root:
$ borg create -e Documents/Junk backup Documents

[jdchristensen]

The last example should answer one of @Atemu's questions: if that command is executed in /home/user, the exclude is not given as home/user/Documents/Junk but as Documents/Junk, so relative excludes are supported in this precise sense.

[Atemu]

It should absolutely be documented that patterns are relative to the working directory rather than the recursion root.

That is not at all intuitive; I would have expected them to be relative to the recursion root and perhaps the recursion root to then be the CWD by default if not otherwise specified.

(Perhaps that might be a better and more flexible mode of operation to begin with.)

[jdchristensen]

It should absolutely be documented that patterns are relative to the working directory rather than the recursion root.

This is not correct. If I do

cd /home/user
borg create repo::backup /usr/lib

then exclusion patterns must match usr/lib/.... So they are not relative to the working directory /home/user. One could say that patterns are relative to the start of the recursion root. But what is written in the documentation says it more precisely (especially in my proposed rephrasing).

There is a reason that patterns aren't relative to the recursion roots. For example, if we do

borg create repo::backup /path1 /path2

and want to exclude /path1/myfile but want to keep /path2/myfile, we can do that with -e path1/myfile. If exclusions were relative to the recursion root, we could try to use -e myfile, but that would match both files.

[Atemu]

This is not correct. If I do

cd /home/user
borg create repo::backup /usr/lib

then exclusion patterns must match usr/lib/.... So they are not relative to the working directory /home/user. One could say that patterns are relative to the start of the recursion root. But what is written in the documentation says it more precisely (especially in my proposed rephrasing).

I see.

So if I'm getting this right, when you give it an absolute path as its recursion root, the "pattern root" becomes / but when you give it a relative path, the "pattern root" becomes the CWD? (modulo relative path normalisation)

There is a reason that patterns aren't relative to the recursion roots. For example, if we do

borg create repo::backup /path1 /path2

Indeed. That is a case I hadn't considered.

I wasn't really aware Borg had such a feature. Is it intended to make include-lists possible without the user needing to build a pattern?

I.e. does borg create reop::backup /usr /home /var do the same as:

R /

+ usr/
+ home/
+ var/
- **

[jdchristensen]

So if I'm getting this right, when you give it an absolute path as its recursion root, the "pattern root" becomes / but when you give it a relative path, the "pattern root" becomes the CWD? (modulo relative path normalisation)

I think you are understanding things correctly, but I wouldn't describe it in this way. Each pattern is used for paths generated by all recursion roots, so I wouldn't say that the patterns have a specific root. Instead, we describe how paths are generated from the recursion roots, and say that the patterns must match the paths in that form. Incidentally, the same rules apply when giving a path name to extract from a backup.

I suspect the answer to your question about + patterns is correct, but I don't know for sure.

[jdchristensen]

Maybe the possibility of multiple recursion roots should be emphasized, and used to justify the way pattern matching is designed. Here is an updated proposal:

Borg generates paths to back up using the recursion roots and matches these paths against patterns. The recursion roots can be given as either relative or absolute paths. Since there can be more than one recursion root, the recursion roots are included in the paths that are matched against patterns, except that leading slashes and leading occurrences of ../ are removed. For example, if you give /absolute/path/ as a root, then the paths going into the matcher will start with absolute/path/. If you give relative/path/ as a root, then the paths going into the matcher will start with relative/path/. And if you give ../../relative/path/ as a root, then the paths going into the matcher will start with relative/path/.

@Atemu Does this seem clear to you?