Merge branch 'topic/christian/add-path-docs'

* topic/christian/add-path-docs:
  Add developer docs about zkg's internal use of directories

CHANGES

@@ -1,3 +1,7 @@
+2.13.0-67 | 2024-01-24 09:31:19 -0800
+
+  * Add developer docs about zkg's internal use of directories (Christian Kreibich, Corelight)
+
 2.13.0-64 | 2023-11-06 12:49:34 +0100
 
   * manager/package: Use is_valid_name() for alias validation (Arne Welzel, Corelight)

VERSION

@@ -1 +1 @@
-2.13.0-64
+2.13.0-67

doc/developers.rst

@@ -88,3 +88,116 @@ INI File Option            ```name```                      `name`
 ========================== =============================== ===========================
 
 Python API docstrings roughly follow the `Google Style Docstrings`_ format.
+
+Internals
+---------
+
+``zkg``'s view of a package
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``zkg`` maintains copies of a Zeek package in up to four places:
+
+- A long-lived clone of the package's git repo in
+  ``$STATEDIR/clones/package/<name>``. This clone (not its installed version,
+  see below) is ``zkg``'s "authoritative" view of a package.
+
+- A conceptually short-lived clone in ``$STATEDIR/scratch/<name>``, for
+  retrieving information about a package ("short-lived", because access to those
+  copies is momentary -- not because ``zkg`` necessarily cleans up after those
+  copies).
+
+- A "stage". A stage is any place in which ``zkg`` actually installs a package
+  for running in Zeek. This can be the local Zeek installation, or locally below
+  ``$STATEDIR/testing`` when testing a package. So "staging" here does not mean
+  "not yet live"; rather the opposite: it's closer to "any place where the
+  package may actually run". A stage allows customization of many of the
+  directories involved (for internal cloning, installing Zeek scripts and
+  plugins, binary files, etc). Installation into a stage is also where ``zkg``
+  adds its management of ``packages.zeek``, in the stage's scripts directory.
+
+- In ``$STATEDIR/testing/<name>/clones/<names>``, for testing a given package
+  along with any dependencies it might have.
+
+``zkg`` captures state about installed packages in ``$STATEDIR/manifest.json``.
+This does not capture all knowable information about packages, though.
+More on this next.
+
+Directory usage
+~~~~~~~~~~~~~~~
+
+``zkg`` populates its internal state directory (dubbed ``$STATEDIR`` below) with
+several subdirectories.
+
+``$STATEDIR/clones``
+""""""""""""""""""""
+
+This directory keeps git clones of installed packages
+(``$STATEDIR/clones/package/<name>``), packages sources
+(``$STATEDIR/clones/source/<name>``), and package template repositories
+(``$STATEDIR/clones/template/<name>``).
+
+``zkg`` clones the relevant repositories as needed. It can dynamically re-create
+some of these directories as needed, but interfering in that space is not
+recommended. For example, if you remove a clone of an installed package, the
+installation itself will remain live (via the staging mechanism), but ``zkg``
+will no longer be able to refer to all information about the installed package
+(because anything not explicitly captured about the package in
+``$STATEDIR/manifest.json`` is now gone).
+
+Removal of a package (``zkg remove``) removes its clone in ``$STATEDIR/clones``.
+
+``$STATEDIR/scratch``
+"""""""""""""""""""""
+
+When retrieving information on a package that isn't yet installed, or where
+``zkg`` doesn't want to touch the installed code, ``$STATEDIR/scratch/<name>``
+is a clone of the package's git repo at a version (often a git tag) of
+interest. This clone is shallow for any versions that aren't raw SHA-1
+hashes. The information parsed includes the ``zkg.meta`` as well as git
+branch/tag/commit info.
+
+During package installation, ``zkg`` places backups of user-tweakable files into
+``$STATEDIR/scratch/tmpcfg``. ``zkg`` restores these after package installation
+to preserve the user's edits. During package source aggregation, ``zkg`` places
+temporary versions of ``aggregate.meta`` directly into ``$STATEDIR/scratch``.
+
+Creation or unbundling of a package happens via ``$STATEDIR/scratch/bundle``, to
+compose or retrieve information about the bundle. The directory is deleted and
+re-created at the beginning of those operations:
+
+- During bundling, ``zkg`` copies installed package repos from
+  ``$STATEDIR/clones/<name>`` into ``$STATEDIR/scratch/bundle/<name>``, and
+  creates fresh git clones in the ``bundle`` directory for any packages not
+  currently installed. It creates a ``.tar.gz`` of the whole directory,
+  initially in the bundle directory, and moves it to where the user specified.
+
+- During unbundling, ``zkg`` reads the bundle manifest as well as the git repos
+  of the contained packages, and moves the package repos from the scratch space
+  into ``$STATEDIR/clones/package/<name>``.
+
+When installing a package's ``script_dir`` or ``plugin_dir`` into a staging area
+and the source file is a tarfile, ``zkg`` temporarily extracts the tarball into
+``$STATEDIR/scratch/untar/``.
+
+There's little or no cleanup of files in the scratch space after the operations
+creating them complete. ``zkg`` only deletes, then (re-)creates, the directories
+involved upon next use.
+
+When ``zkg`` isn't in the middle of executing any commands, you can always
+delete the scratch space without negatively affecting ``zkg``.
+
+``$STATEDIR/testing``
+"""""""""""""""""""""
+
+When testing a package (during installation, or when explicitly running ``zkg
+test``), ``zkg`` creates a staging area ``$STATEDIR/testing/<name>`` for the
+package under test, clones the package and its dependencies into
+``$STATEDIR/testing/<name>/clones/``, installs them from there into
+``$STATEDIR/testing/<name>``, and then runs the package's ``test_command`` from
+its clone (``$STATEDIR/testing/<name>/clones/<name>``), with an environment set
+such that it finds the installation in the local stage. The stdout and stderr of
+those testing runs is preserved into ``zkg.test_command.stdout`` and
+``zkg.test_command.stderr`` in that directory.
+
+As in ``STATEDIR/scratch``, there's no cleanup, and you can delete the testing
+space as needed after a test run is complete.

doc/man/zkg.1

@@ -27,7 +27,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
 .in \\n[rst2man-indent\\n[rst2man-indent-level]]u
 ..
-.TH "ZKG" "1" "Nov 06, 2023" "2.13.0-64" "Zeek Package Manager"
+.TH "ZKG" "1" "Jan 24, 2024" "2.13.0-67" "Zeek Package Manager"
 .SH NAME
 zkg \- Zeek Package Manager
 .sp
@@ -38,7 +38,9 @@ A command\-line package manager for Zeek.
 .sp
 .nf
 .ft C
-usage: zkg [\-h] [\-\-version] [\-\-configfile FILE | \-\-user] [\-\-verbose] [\-\-extra\-source NAME=URL] {test,install,bundle,unbundle,remove,uninstall,purge,refresh,upgrade,load,unload,pin,unpin,list,search,info,config,autoconfig,env,create,template} ...
+usage: zkg [\-h] [\-\-version] [\-\-configfile FILE | \-\-user] [\-\-verbose] [\-\-extra\-source NAME=URL]
+           {test,install,bundle,unbundle,remove,uninstall,purge,refresh,upgrade,load,unload,pin,unpin,list,search,info,config,autoconfig,env,create,template}
+           ...
 .ft P
 .fi
 .UNINDENT
@@ -119,7 +121,9 @@ Installs packages from a configured package source or directly from a git URL.
 .sp
 .nf
 .ft C
-usage: zkg install [\-h] [\-\-skiptests] [\-\-nodeps] [\-\-nosuggestions] [\-\-version VERSION] [\-\-ignore\-dirty\-git] [\-\-force] [\-\-user\-var NAME=VAL] package [package ...]
+usage: zkg install [\-h] [\-\-skiptests] [\-\-nodeps] [\-\-nosuggestions] [\-\-version VERSION] [\-\-ignore\-dirty\-git]
+                   [\-\-force] [\-\-user\-var NAME=VAL]
+                   package [package ...]
 .ft P
 .fi
 .UNINDENT
@@ -230,7 +234,9 @@ This command creates a bundle file containing a collection of Zeek packages.  If
 .sp
 .nf
 .ft C
-usage: zkg bundle [\-h] [\-\-force] [\-\-nodeps] [\-\-nosuggestions] [\-\-manifest MANIFEST [MANIFEST ...] \-\-] [\-\-ignore\-dirty\-git] filename.bundle
+usage: zkg bundle [\-h] [\-\-force] [\-\-nodeps] [\-\-nosuggestions] [\-\-manifest MANIFEST [MANIFEST ...] \-\-]
+                  [\-\-ignore\-dirty\-git]
+                  filename.bundle
 .ft P
 .fi
 .UNINDENT
@@ -308,7 +314,8 @@ Retrieve latest package metadata from sources and checks whether any installed p
 .sp
 .nf
 .ft C
-usage: zkg refresh [\-h] [\-\-aggregate] [\-\-fail\-on\-aggregate\-problems] [\-\-push] [\-\-sources SOURCES [SOURCES ...]]
+usage: zkg refresh [\-h] [\-\-aggregate] [\-\-fail\-on\-aggregate\-problems] [\-\-push]
+                   [\-\-sources SOURCES [SOURCES ...]]
 .ft P
 .fi
 .UNINDENT
@@ -340,7 +347,8 @@ Uprades the specified package(s) to latest available version.  If no specific pa
 .sp
 .nf
 .ft C
-usage: zkg upgrade [\-h] [\-\-skiptests] [\-\-nodeps] [\-\-nosuggestions] [\-\-force] [\-\-user\-var NAME=VAL] [package ...]
+usage: zkg upgrade [\-h] [\-\-skiptests] [\-\-nodeps] [\-\-nosuggestions] [\-\-force] [\-\-user\-var NAME=VAL]
+                   [package ...]
 .ft P
 .fi
 .UNINDENT
@@ -549,7 +557,9 @@ Shows detailed information/metadata for given packages. If the package is curren
 .sp
 .nf
 .ft C
-usage: zkg info [\-h] [\-\-version VERSION] [\-\-nolocal] [\-\-include\-builtin] [\-\-json] [\-\-jsonpretty SPACES] [\-\-allvers] package [package ...]
+usage: zkg info [\-h] [\-\-version VERSION] [\-\-nolocal] [\-\-include\-builtin] [\-\-json] [\-\-jsonpretty SPACES]
+                [\-\-allvers]
+                package [package ...]
 .ft P
 .fi
 .UNINDENT
@@ -658,7 +668,8 @@ This command creates a new Zeek package in the directory provided via \-\-packag
 .sp
 .nf
 .ft C
-usage: zkg create [\-h] \-\-packagedir DIR [\-\-version VERSION] [\-\-features FEATURE [FEATURE ...]] [\-\-template URL] [\-\-force] [\-\-user\-var NAME=VAL]
+usage: zkg create [\-h] \-\-packagedir DIR [\-\-version VERSION] [\-\-features FEATURE [FEATURE ...]]
+                  [\-\-template URL] [\-\-force] [\-\-user\-var NAME=VAL]
 .ft P
 .fi
 .UNINDENT

zeekpkg/__init__.py

@@ -9,7 +9,7 @@
 
 import logging
 
-__version__ = "2.13.0-64"
+__version__ = "2.13.0-67"
 __all__ = ["manager", "package", "source", "template", "uservar"]
 
 LOG = logging.getLogger(__name__)