Merge pull request #7646 from ThomasWaldmann/rel200b6

Release 2.0.0b6

docs/changes.rst

@@ -12,8 +12,8 @@ This section provides information about security and corruption issues.
 Change Log 2.x
 ==============
 
-Version 2.0.0b6 (not released yet)
-----------------------------------
+Version 2.0.0b6 (2023-06-11)
+----------------------------
 
 Please note:
 
@@ -118,6 +118,7 @@ New features:
 - diff: include changes in ctime and mtime, #7248
 - diff: sort JSON output alphabetically
 - diff --content-only: option added to ignore metadata changes
+- diff: add --format option, #4634
 - import-tar --ignore-zeros: new option to support importing concatenated tars, #7432
 - debug id-hash / parse-obj / format-obj: new debug commands, #7406
 - transfer --compression=C --recompress=M: recompress while transferring, #7529
@@ -173,6 +174,7 @@ Other changes:
 - borg.logger: add logging debugging functionality
 - add function to clear empty directories at end of compact process
 - unify scanning and listing of segment dirs / segment files, #7597
+- replace `LRUCache` internals with `OrderedDict`
 - docs:
 
   - add installation instructions for Windows
@@ -202,6 +204,7 @@ Other changes:
   - tox: package = editable-legacy, #7580
   - tox under fakeroot: fix finding setup_docs, #7391
   - check buzhash chunksize distribution, #7586
+  - use debian/bookworm64 box
 
 
 Version 2.0.0b5 (2023-02-27)

docs/deployment/central-backup-server.rst

@@ -1,5 +1,6 @@
 .. include:: ../global.rst.inc
 .. highlight:: none
+.. _central-backup-server:
 
 Central repository server with Ansible or Salt
 ==============================================

docs/deployment/pull-backup.rst

@@ -470,13 +470,13 @@ Make sure to use port ``8022`` and ``localhost`` for the repository as this inst
 remote forwarded ssh connection.
 
 SSH Keys
-~~~~~~~~
+--------
 
 If you want to automate backups when using this method, the ssh ``known_hosts`` and ``authorized_keys`` need to be set up
 to allow connections.
 
 Security Considerations
-~~~~~~~~~~~~~~~~~~~~~~~
+-----------------------
 
 Opening up SSH access this way can pose a security risk as it effectively opens remote access to your
 backup server on the client even if it is located outside of your company network.
@@ -496,7 +496,7 @@ All the additional security considerations for borg should be applied, see :ref:
 hints.
 
 More information
-~~~~~~~~~~~~~~~~
+----------------
 
 See `remote forwarding`_ and the `ssh man page`_ for more information about remote forwarding.
 

docs/faq.rst

@@ -25,7 +25,7 @@ running on the remote side, so *every* operation needs to go over the network,
 which is slower.
 
 Can I back up from multiple servers into a single repository?
-------------------------------------------------------------
+-------------------------------------------------------------
 
 In order for the deduplication used by Borg to work, it
 needs to keep a local cache containing checksums of all file
@@ -162,7 +162,7 @@ file is being processed.
 
 
 How can I back up huge file(s) over a unstable connection?
----------------------------------------------------------
+----------------------------------------------------------
 
 Yes. For more details, see :ref:`checkpoints_parts`.
 
@@ -744,7 +744,7 @@ This has some pros and cons, though:
 The long term plan to improve this is called "borgception", see :issue:`474`.
 
 Can I back up my root partition (/) with Borg?
----------------------------------------------
+----------------------------------------------
 
 Backing up your entire root partition works just fine, but remember to
 exclude directories that make no sense to back up, such as /dev, /proc,

docs/man/borg-benchmark-cpu.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "BORG-BENCHMARK-CPU" 1 "2023-02-26" "" "borg backup tool"
+.TH "BORG-BENCHMARK-CPU" 1 "2023-06-11" "" "borg backup tool"
 .SH NAME
 borg-benchmark-cpu \- Benchmark CPU bound operations.
 .SH SYNOPSIS

docs/man/borg-benchmark-crud.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "BORG-BENCHMARK-CRUD" 1 "2023-02-26" "" "borg backup tool"
+.TH "BORG-BENCHMARK-CRUD" 1 "2023-06-11" "" "borg backup tool"
 .SH NAME
 borg-benchmark-crud \- Benchmark Create, Read, Update, Delete for archives.
 .SH SYNOPSIS

docs/man/borg-benchmark.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "BORG-BENCHMARK" 1 "2023-02-26" "" "borg backup tool"
+.TH "BORG-BENCHMARK" 1 "2023-06-11" "" "borg backup tool"
 .SH NAME
 borg-benchmark \- benchmark command
 .SH SYNOPSIS

docs/man/borg-break-lock.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "BORG-BREAK-LOCK" 1 "2023-02-26" "" "borg backup tool"
+.TH "BORG-BREAK-LOCK" 1 "2023-06-11" "" "borg backup tool"
 .SH NAME
 borg-break-lock \- Break the repository lock (e.g. in case it was left by a dead borg.
 .SH SYNOPSIS

docs/man/borg-check.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "BORG-CHECK" 1 "2023-02-26" "" "borg backup tool"
+.TH "BORG-CHECK" 1 "2023-06-11" "" "borg backup tool"
 .SH NAME
 borg-check \- Check repository consistency
 .SH SYNOPSIS

docs/man/borg-common.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "BORG-COMMON" 1 "2023-02-26" "" "borg backup tool"
+.TH "BORG-COMMON" 1 "2023-06-11" "" "borg backup tool"
 .SH NAME
 borg-common \- Common options of Borg commands
 .SH SYNOPSIS
@@ -93,6 +93,9 @@ Write execution profile in Borg format into FILE. For local use a Python\-compat
 .BI \-\-rsh \ RSH
 Use this command to connect to the \(aqborg serve\(aq process (default: \(aqssh\(aq)
 .TP
+.BI \-\-socket \ PATH
+Use UNIX DOMAIN (IPC) socket at PATH for client/server communication with socket: protocol.
+.TP
 .BI \-r \ REPO\fR,\fB \ \-\-repo \ REPO
 repository to use
 .UNINDENT

docs/man/borg-compact.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "BORG-COMPACT" 1 "2023-02-26" "" "borg backup tool"
+.TH "BORG-COMPACT" 1 "2023-06-11" "" "borg backup tool"
 .SH NAME
 borg-compact \- compact segment files in the repository
 .SH SYNOPSIS

docs/man/borg-compression.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "BORG-COMPRESSION" 1 "2023-02-26" "" "borg backup tool"
+.TH "BORG-COMPRESSION" 1 "2023-06-11" "" "borg backup tool"
 .SH NAME
 borg-compression \- Details regarding compression
 .SH DESCRIPTION
@@ -82,29 +82,70 @@ being any valid compression specifier.
 .TP
 .B obfuscate,SPEC,C[,L]
 Use compressed\-size obfuscation to make fingerprinting attacks based on
-the observable stored chunk size more difficult.
-Note:
-\- you must combine this with encryption or it won\(aqt make any sense.
-\- your repo size will be bigger, of course.
+the observable stored chunk size more difficult. Note:
+.INDENT 7.0
+.IP \(bu 2
+You must combine this with encryption, or it won\(aqt make any sense.
+.IP \(bu 2
+Your repo size will be bigger, of course.
+.IP \(bu 2
+A chunk is limited by the constant \fBMAX_DATA_SIZE\fP (cur. ~20MiB).
+.UNINDENT
+.sp
+The SPEC value determines how the size obfuscation works:
 .sp
-The SPEC value will determine how the size obfuscation will work:
+\fIRelative random reciprocal size variation\fP (multiplicative)
 .sp
-Relative random reciprocal size variation:
 Size will increase by a factor, relative to the compressed data size.
-Smaller factors are often used, larger factors rarely.
-1: factor 0.01 .. 100.0
-2: factor 0.1 .. 1000.0
-3: factor 1.0 .. 10000.0
-4: factor 10.0 .. 100000.0
-5: factor 100.0 .. 1000000.0
-6: factor 1000.0 .. 10000000.0
+Smaller factors are used often, larger factors rarely.
+.sp
+Available factors:
+.INDENT 7.0
+.INDENT 3.5
+.sp
+.nf
+.ft C
+1:     0.01 ..        100
+2:     0.1  ..      1,000
+3:     1    ..     10,000
+4:    10    ..    100,000
+5:   100    ..  1,000,000
+6: 1,000    .. 10,000,000
+.ft P
+.fi
+.UNINDENT
+.UNINDENT
+.sp
+Example probabilities for SPEC \fB1\fP:
+.INDENT 7.0
+.INDENT 3.5
+.sp
+.nf
+.ft C
+90   %  0.01 ..   0.1
+ 9   %  0.1  ..   1
+ 0.9 %  1    ..  10
+ 0.09% 10    .. 100
+.ft P
+.fi
+.UNINDENT
+.UNINDENT
+.sp
+\fIRandomly sized padding up to the given size\fP (additive)
+.INDENT 7.0
+.INDENT 3.5
 .sp
-Add a randomly sized padding up to the given size:
-110: 1kiB
+.nf
+.ft C
+110: 1kiB (2 ^ (SPEC \- 100))
 \&...
 120: 1MiB
 \&...
 123: 8MiB (max.)
+.ft P
+.fi
+.UNINDENT
+.UNINDENT
 .UNINDENT
 .sp
 Examples:
@@ -120,7 +161,7 @@ borg create \-\-compression zlib REPO::ARCHIVE data
 borg create \-\-compression zlib,1 REPO::ARCHIVE data
 borg create \-\-compression auto,lzma,6 REPO::ARCHIVE data
 borg create \-\-compression auto,lzma ...
-borg create \-\-compression obfuscate,3,none ...
+borg create \-\-compression obfuscate,110,none ...
 borg create \-\-compression obfuscate,3,auto,zstd,10 ...
 borg create \-\-compression obfuscate,2,zstd,6 ...
 .ft P

docs/man/borg-config.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "BORG-CONFIG" 1 "2023-02-26" "" "borg backup tool"
+.TH "BORG-CONFIG" 1 "2023-06-11" "" "borg backup tool"
 .SH NAME
 borg-config \- get, set, and delete values in a repository or cache config file
 .SH SYNOPSIS

docs/man/borg-create.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "BORG-CREATE" 1 "2023-02-26" "" "borg backup tool"
+.TH "BORG-CREATE" 1 "2023-06-11" "" "borg backup tool"
 .SH NAME
 borg-create \- Create new archive
 .SH SYNOPSIS
@@ -169,15 +169,15 @@ set mode to M in archive for stdin data (default: 0660)
 interpret PATH as command and store its stdout. See also section Reading from stdin below.
 .TP
 .B  \-\-paths\-from\-stdin
-read DELIM\-separated list of paths to back up from stdin. Will not recurse into directories.
+read DELIM\-separated list of paths to back up from stdin. All control is external: it will back up all files given \- no more, no less.
 .TP
 .B  \-\-paths\-from\-command
 interpret PATH as command and treat its output as \fB\-\-paths\-from\-stdin\fP
 .TP
 .BI \-\-paths\-delimiter \ DELIM
-set path delimiter for \fB\-\-paths\-from\-stdin\fP and \fB\-\-paths\-from\-command\fP (default: n)
+set path delimiter for \fB\-\-paths\-from\-stdin\fP and \fB\-\-paths\-from\-command\fP (default: \fB\en\fP)
 .UNINDENT
-.SS Exclusion options
+.SS Include/Exclude options
 .INDENT 0.0
 .TP
 .BI \-e \ PATTERN\fR,\fB \ \-\-exclude \ PATTERN
@@ -208,7 +208,7 @@ exclude files flagged NODUMP
 .INDENT 0.0
 .TP
 .B  \-x\fP,\fB  \-\-one\-file\-system
-stay in the same file system and do not store mount points of other file systems.  This might behave different from your expectations, see the docs.
+stay in the same file system and do not store mount points of other file systems \- this might behave different from your expectations, see the description below.
 .TP
 .B  \-\-numeric\-ids
 only store numeric user and group identifiers
@@ -368,13 +368,13 @@ through using the \fB\-\-keep\-exclude\-tags\fP option.
 The \fB\-x\fP or \fB\-\-one\-file\-system\fP option excludes directories, that are mountpoints (and everything in them).
 It detects mountpoints by comparing the device number from the output of \fBstat()\fP of the directory and its
 parent directory. Specifically, it excludes directories for which \fBstat()\fP reports a device number different
-from the device number of their parent. Be aware that in Linux (and possibly elsewhere) there are directories
-with device number different from their parent, which the kernel does not consider a mountpoint and also the
-other way around. Examples are bind mounts (possibly same device number, but always a mountpoint) and ALL
-subvolumes of a btrfs (different device number from parent but not necessarily a mountpoint). Therefore when
-using \fB\-\-one\-file\-system\fP, one should make doubly sure that the backup works as intended especially when using
-btrfs. This is even more important, if the btrfs layout was created by someone else, e.g. a distribution
-installer.
+from the device number of their parent.
+In general: be aware that there are directories with device number different from their parent, which the kernel
+does not consider a mountpoint and also the other way around.
+Linux examples for this are bind mounts (possibly same device number, but always a mountpoint) and ALL
+subvolumes of a btrfs (different device number from parent but not necessarily a mountpoint).
+macOS examples are the apfs mounts of a typical macOS installation.
+Therefore, when using \fB\-\-one\-file\-system\fP, you should double\-check that the backup works as intended.
 .SS Item flags
 .sp
 \fB\-\-list\fP outputs a list of all files, directories and other

docs/man/borg-delete.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "BORG-DELETE" 1 "2023-02-26" "" "borg backup tool"
+.TH "BORG-DELETE" 1 "2023-06-11" "" "borg backup tool"
 .SH NAME
 borg-delete \- Delete archives
 .SH SYNOPSIS
@@ -70,9 +70,6 @@ consider checkpoint archives for deletion (default: not considered).
 .B  \-s\fP,\fB  \-\-stats
 print statistics for the deleted archive
 .TP
-.B  \-\-cache\-only
-delete only the local cache for the given repository
-.TP
 .B  \-\-force
 force deletion of corrupted archives, use \fB\-\-force \-\-force\fP in case \fB\-\-force\fP does not work.
 .TP
@@ -118,13 +115,13 @@ $ borg delete Monday
 $ borg compact
 
 # delete all archives whose names begin with the machine\(aqs hostname followed by \(dq\-\(dq
-$ borg delete \-a \(aq{hostname}\-*\(aq
+$ borg delete \-a \(aqsh:{hostname}\-*\(aq
 
 # delete all archives whose names contain \(dq\-2012\-\(dq
-$ borg delete \-a \(aq*\-2012\-*\(aq
+$ borg delete \-a \(aqsh:*\-2012\-*\(aq
 
 # see what would be deleted if delete was run without \-\-dry\-run
-$ borg delete \-\-list \-\-dry\-run \-a \(aq*\-May\-*\(aq
+$ borg delete \-\-list \-\-dry\-run \-a \(aqsh:*\-May\-*\(aq
 .ft P
 .fi
 .UNINDENT