Merge branch '2.9' into stable

.readthedocs.yaml

@@ -0,0 +1,29 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-20.04
+  tools:
+    python: "3.9"
+    # You can also specify other tool versions:
+    # nodejs: "16"
+    # rust: "1.55"
+    # golang: "1.17"
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+   configuration: conf.py
+
+# If using Sphinx, optionally build your docs in additional formats such as PDF
+# formats:
+#    - pdf
+
+# Optionally declare the Python requirements required to build your docs
+python:
+   install:
+   - requirements: requirements.txt

conf.py

@@ -30,7 +30,8 @@
 # ones.
 extensions = [
     'sphinx.ext.todo',
-    'sphinx.ext.mathjax'
+    'sphinx.ext.mathjax',
+    'sphinxcontrib.mermaid'
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -57,9 +58,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = u'2.8'
+version = u'2.9'
 # The full version, including alpha/beta/rc tags.
-release = u'2.8.2'
+release = u'2.9.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

cpt-configure.rst

@@ -603,6 +603,8 @@ the host or by the proxy.
 * Failures of proxy name resolution and any connection/timeout error, partial
   file transfer, or non 2XX return code (except 5XX and 404) are classified as
   proxy failure if a proxy server is used.
+* Explicit proxy errors (indicated via the `X-Squid-Error` or `Proxy-Status`
+  headers) will always be classified as proxy failure.
 
 If CernVM-FS detects a host failure, it will fail-over to the next host in the
 list while keeping the proxy server untouched. If it detects a proxy failure, it

cpt-containers.rst

@@ -222,15 +222,77 @@ There are several lines of development that we are pursuing to improve
 the CernVM-FS container integration.
 
 ``containerd`` remote-snapshotter plugin
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+----------------------------------------
 
-This will allow running images from Kubernetes looking for the layers first in
-CernVM-FS and if the layers are not to be found, downloading them from the
-standard docker registry.
+CernVM-FS integration with ``containerd`` is achieved by the snapshotter plugin,
+a specialized component responsible for assembling all the layers of container
+images into a stacked filesystem that ``containerd`` can use.
+The snapshotter takes as input the list of required layers and outputs a directory
+containing the final filesystem. It is also responsible to clean-up the output
+directory when containers using it are stopped.
+
+We focus on the layers provided by CernVM-FS, but with minor changes it is possible to mount layers from any
+filesystem, like NFS. In CernVM-FS, the available layers are stored at ``/cvmfs/<repo_name>/.layers``.
+If the desired layers are not in the local filesystem, ``containerd`` simply follows the
+standard path downloading them from the standard docker registry.
+
+Configuration
+~~~~~~~~~~~~~
+
+The CernVM-FS remote snapshotter communicates with ``containerd`` via gRPC over a UNIX domain socket.
+The default socket is ``/run/containerd-cvmfs-grpc/containerd-cvmfs-grpc.sock``.
+This socket is created automatically by the snapshotter when building the binary, if it does not exist.
+
+To build the binary, use the following commands:
+
+::
+
+    cd <source directory>
+    make
+
+A new ``/out`` folder is created with the binary ``cvmfs-snapshotter``.
+
+How to use the CernVM-FS Snapshotter
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The binary accepts different command line options:
+-  ``--address``: address for the snapshotter's GRPC server.
+The default one is ``/run/containerd-cvmfs-grpc/containerd-cvmfs-grpc.sock``
+-  ``--config``: path to the configuration file.
+Creating a configuration file is useful to customize the default values.
+-  ``--log-level``: logging level [trace, debug, info, warn, error, fatal, panic].
+The default values is ``info``.
+-  ``--root``: path to the root directory for this snapshotter.
+The default one is ``/var/lib/containerd-cvmfs-grpc``.
+
+By default, the repository used to search for the layers is ``unpacked.cern.ch``.
+The default values can be overrided by running the binary indicating the path to a
+configuration file ``config.toml`` using the ``--config`` option. A basic configuration
+file with the default values would look like:
+
+::
+    # tell containerd the repository and the mountpoint
+    repository = "unpacked.cern.ch"
+    absolute-mountpoint = "/cvmfs/unpacked.cern.ch"
+
+    # tell containerd to use this particular snapshotter
+    [plugins."io.containerd.grpc.v1.cri".containerd]
+    snapshotter = "cvmfs-snapshotter"
+    disable_snapshot_annotations = false
+
+    # tell containerd how to communicate with this snapshotter
+    [proxy_plugins]
+    [proxy_plugins.cvmfs-snapshotter]
+        type = "snapshot"
+        address = "/run/containerd-cvmfs-grpc/containerd-cvmfs-grpc.sock"
+
+
+Note that if only the repository is specified under the key value ``repository``, the mountpoint
+(under the key value ``absolute-mountpoint``) is by default constructed as ``/cvmfs/<repo_name>``.
 
 
 ``podman`` integration
-~~~~~~~~~~~~~~~~~~~~~~
+----------------------
 
 Similarly to the ``containerd`` integration, this development will allow running
 a standard docker image using podman fetching the layers, unpacked, from a
@@ -239,7 +301,7 @@ registry if necessary.
 
 
 DUCC registry interface
-~~~~~~~~~~~~~~~~~~~~~~~
+-----------------------
 
 This development will allow for pushing the image to a special registry and
 for finding the image in the CernVM-FS repository as soon as the push

cpt-ducc.rst

@@ -139,10 +139,6 @@ refer to the input image.
 The current wish list format requires all the images to be stored in the same
 CernVM-FS repository and have the same thin output image format.
 
-DUCC Requirements
-=============
-DUCC requires Singularity to be installed, and requires a cvmfs_server publishing interface.
-
 DUCC Commands
 =============
 
@@ -181,6 +177,21 @@ iteration, the wish list file is read again in order to pick up changes.
 
     cvmfs_ducc loop recipe.yaml
 
+convert-single-image
+********************
+
+The `convert-single-image` command is useful when only a single image need to
+be converted and pushed into a CernVM-FS repository.
+
+::
+
+    cvmfs_ducc convert-single-image image-to-convert repository.cern.ch
+
+The command takes two arguments as input, the image to convert and the CernVM-FS
+repository where to store it.
+
+The `image-to-convert` argument follow the same syntax of the wishlist, for
+instance it could be something like `https://registry.hub.docker.com/library/fedora:latest`.
 
 
 Incremental Conversion
@@ -193,3 +204,46 @@ while others will need to be converted ex-novo.
 An image that has been already unpacked in CernVM-FS will be skipped. For
 unconverted images, only the missing layers will be unpacked.
 
+Layer Aware
+===========
+
+DUCC is now aware that containers images are build incrementally on top of
+smaller layers.
+
+Converting an image based on an image already inside the repository will skip
+most of the work.
+
+As long as the lower layers of an image don't change this allows a very fast
+ingestion of software images, irrespectively of their size.
+
+Notification
+============
+
+DUCC provides a basic notification system to alert external services of
+updates in the filesystem.
+
+The notifications are appended to a simple text file as JSON objects.
+
+Human operator or software can follow the file and react on notification of
+interest.
+
+The notification file, eventually can grow large. The suggestion is to treat it
+as a standard log file with tools like `logrotate`.
+
+Multiple DUCC processes can write on the same notification file at the same
+time, multiple consumer can read from it.
+
+The notification are activated if and only if the user ask for them providing
+a file where to write them.
+To provide a notification file the flag `-n/--notification-file` is available.
+
+Multiprocess
+============
+
+DUCC is able to run multiprocess against the same CernVM-FS repository.
+
+Before to interact with the CernVM-FS repository, DUCC takes a filesystem
+level lock against `/tmp/DUCC.lock`.
+
+This allows to run multiple instances of DUCC at the same time, one instance
+could listen to a web socket, while one could be doing wishlist conversion.

cpt-quickstart.rst

@@ -48,7 +48,7 @@ To install the CVMFS package run
 
 ::
 
-    sudo dnf install https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.8.2/cvmfs-2.8.2-1.fc32.x86_64.rpm https://ecsft.cern.ch/dist/cvmfs/cvmfs-config/cvmfs-config-default-latest.noarch.rpm
+    sudo dnf install https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.9.0/cvmfs-2.9.0-1.fc34.x86_64.rpm https://ecsft.cern.ch/dist/cvmfs/cvmfs-config/cvmfs-config-default-latest.noarch.rpm
 
 
 Docker Container
@@ -65,7 +65,7 @@ or with
 
 ::
 
-    curl https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.8.2/cvmfs-service-2.8.2-1.x86_64.docker.tar.gz | docker load
+    curl https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.9.0/cvmfs-service-2.9.0-1.x86_64.docker.tar.gz | docker load
 
 Run the container as a system service with
 
@@ -98,7 +98,7 @@ Download the CernVM-FS client package in the terminal in order to avoid signatur
 
 ::
 
-    curl -o ~/Downloads/cvmfs-2.8.2.pkg https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.8.2/cvmfs-2.8.2.pkg
+    curl -o ~/Downloads/cvmfs-2.9.0.pkg https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.9.0/cvmfs-2.9.0.pkg
 
 Install the CernVM-FS package by opening the .pkg file and reboot.
 Future releases will provide a signed and notarized package.