refine the documentation

doc/accessories.rst

@@ -13,3 +13,30 @@ Borg-Space
 `Borg-Space <https://github.com/KenKundert/borg-space>`_ is a utility that 
 reports and tracks the space required by your *Borg* repositories.
 It also allows you to graph the space used over time.
+
+
+.. _ntlog accessory:
+
+Logging with ntLog
+------------------
+
+`ntLog <https://github.com/KenKundert/ntlog>`_ is a log file aggregation 
+utility.
+
+When run *Emborg* writes over a previously generated logfile.  This becomes 
+problematic if you have one  cron script that runs *create* frequently and 
+another that runs a command like *prune* less frequently. If there is trouble 
+with the *prune* command it will be difficult to see and resolve because its 
+logfile will be overwritten by subsequent *create* commands.
+
+*ntlog* can be run after each *Emborg* run to aggregate the individual logfile 
+from each run into a single accumulating log file.  To arrange this you can use 
+:ref:`run_after_borg <run_after_borg>`::
+
+    run_after_borg = 'ntlog --delete --keep-for 7 ~/.local/share/emborg/{config_name}.log'
+
+This accumulates the log files as they are created to 
+~/.local/share/emborg/{config_name}.log.nt.
+
+
+

doc/configuring.rst

@@ -342,6 +342,54 @@ features are described in the `Borg patterns documentation
 <https://borgbackup.readthedocs.io/en/stable/usage/help.html>`_.
 
 
+.. _retention:
+
+Archive Retention
+-----------------
+
+You use the retention limits (the ``keep_X`` settings) to specify how long to 
+keep archives after they have been created.  A good description of the use of 
+these settings can be found on the `Borg Prune Command 
+<https://borgbackup.readthedocs.io/en/stable/usage/prune.html>`_ page.
+
+Generally you want to thin the archives out more and more as they age.  When 
+choosing your retention limits you need to consider the nature of the files you 
+are archiving.  Specifically you need to consider how often the files change, 
+whether you would want to recover prior versions of the files you keep and if so 
+how many prior versions are of interest, and how long precious files may be 
+missing or damaged before you notice that they need to be restored.
+
+If files are changing all the time, long high retention limits result in high 
+storage requirements.  If you want to make sure you retain the latest version of 
+a file but you do not need prior versions, then you can reduce your retention 
+limits to reduce your storage requirements.  For example, consider a directory 
+of log files.  Log files generally change all the time, but they also tend to be 
+cumulative, meaning that the latest file contains the information contained in 
+prior versions of the same file, so keeping those prior versions is of low 
+value.  In this situation using “*keep_last N*” where *N* is small is a good 
+approach.
+
+Now consider a directory of files that should be kept forever, such as family 
+photos or legal documents.  The loss of these files due to disk corruption or 
+accidental deletion might not be noticed for years.  In this case you would want 
+to specify “*keep_yearly N*” where *N* is large.  These files never change, so 
+the de-duplication feature of *Borg* avoids growth in storage requirements 
+despite high retention limits.
+
+You cannot specify retention limits on a per file or per directory basis within 
+a single configuration.  Instead, if you feel it is necessary, you would create 
+individual configurations for files with different retention needs.  For 
+example, as a system administrator you might want to create separate 
+configurations for operating system files, which tend to need low retention 
+limits, and users home directories, which benefit from longer retention limits.
+
+Remember that your retention limits are not enforced until you run the 
+:ref:`prune command <prune>`.  Furthermore, with *Borg 1.2* and later, after 
+running the *prune command*, the disk space is not reclaimed until you run the 
+:ref:`compact command <compact>`.  You can automate pruning and compaction using 
+the :ref:`prune_after_create` and :ref:`compact_after_delete` settings.
+
+
 .. _confirming_configuration:
 
 Confirming Your Configuration

emborg/emborg.py

@@ -20,17 +20,15 @@
 import errno
 import os
 import re
-from textwrap import dedent
-
 import arrow
-
 from inform import (
     Color,
     Error,
     LoggingCache,
     codicil,
     comment,
     conjoin,
+    dedent,
     display,
     done,
     errors_accrued,
@@ -50,7 +48,6 @@
     Run, cd, cwd, getmod, mv, render_command, rm, to_path,
     set_prefs as set_shlib_prefs
 )
-
 from . import __version__
 from .collection import Collection, split_lines
 from .hooks import Hooks
@@ -368,17 +365,15 @@ def fail(self, *msg, cmd='<unknown>'):
                 Run(
                     ["mail", "-s", f"{PROGRAM_NAME} failed on {username}@{hostname}"]
                     + self.notify.split(),
-                    stdin=dedent(
-                        f"""\
+                    stdin=dedent(f"""\
                         {PROGRAM_NAME} fails.
 
                         command: {cmd}
                         config: {self.config_name}
                         source: {username}@{fullhostname}:{', '.join(str(d) for d in self.src_dirs)}
                         destination: {self.repository!s}
                         error message:
-                        """
-                    ) + indent(msg) + "\n",
+                    """) + indent(msg) + "\n",
                     modes="soeW",
                     encoding="ascii",
                 )
@@ -552,16 +547,14 @@ def borg_options(self, cmd, borg_opts, emborg_opts, strip_prefix):
                     warn("passphrase given but not needed as encryption set to none.")
                 if encryption.startswith("keyfile"):
                     warn(
-                        dedent(
-                            f"""
+                        dedent(f"""
                             you should use “borg key export” to export the
                             encryption key, and then keep that key in a safe
                             place.  You can do this with emborg using “emborg
                             --config {self.config_name} borg key export @repo
                             ❬/.../out-file❭”.  If you lose this key you lose
                             access to your backups.
-                            """
-                        ).strip(),
+                        """).strip(),
                         wrap=True,
                     )
             else:
@@ -928,12 +921,10 @@ def __enter__(self):
             now = arrow.now()
             pid = os.getpid()
             lockfile.write_text(
-                dedent(
-                    f"""
+                dedent(f"""
                     started = {now!s}
                     pid = {pid}
-                    """
-                ).lstrip()
+                """).lstrip()
             )
 
         # open logfile