Merge branch '2.10' into stable

cvmfs · Nov 7, 2022 · 642493b · 642493b
2 parents c036478 + d2eaaf1
commit 642493b
Show file tree

Hide file tree

Showing 8 changed files with 158 additions and 193 deletions.
diff --git a/_static/css/custom.css b/_static/css/custom.css
@@ -0,0 +1,16 @@
+/* modified from https://github.com/readthedocs/sphinx_rtd_theme/issues/295#issuecomment-560895037 */
+
+.wy-nav-content {
+    max-width: 900px !important;
+}
+
+/* and fix wrap bug per https://rackerlabs.github.io/docs-rackspace/tools/rtd-tables.html */
+.wy-table-responsive table td {
+    /* !important prevents the common CSS stylesheets from overriding
+    this as on RTD they are loaded after this stylesheet */
+    white-space: normal !important;
+}
+
+.wy-table-responsive {
+    overflow: visible !important;
+}
diff --git a/apx-parameters.rst b/apx-parameters.rst
@@ -62,10 +62,10 @@ CVMFS_INSTRUMENT_FUSE           | When set to *true* gather performance statisti
                                 | The results are displayed with `cvmfs_talk internal affairs`.
 CVMFS_NFS_INTERLEAVED_INODES    In NFS mode, use only inodes of the form :math:`an+b`, specified as "b%a".
 CVMFS_IPFAMILY_PREFER           Which IP protocol to prefer when connecting to proxies.  Can be either 4 or 6.
-CVMFS_KCACHE_TIMEOUT            Timeout for path names and file attributes in the kernel file system buffers.
+CVMFS_KCACHE_TIMEOUT            Timeout in seconds for path names and file attributes in the kernel file system buffers.
 CVMFS_KEYS_DIR                  | Directory containing \*.pub files used as repository signing keys.
                                 | If set, this parameter has precedence over ``CVMFS_PUBLIC_KEY``.
-CVMFS_LOW_SPEED_LIMIT           Minimum transfer rate a server or proxy must provide.
+CVMFS_LOW_SPEED_LIMIT           Minimum transfer rate in bytes/second a server or proxy must provide.
 CVMFS_MAX_EXTERNAL_SERVERS      | Limit the number of (geo sorted) stratum 1 servers for external data
                                 | that are effectively used.
 CVMFS_MAX_IPADDR_PER_PROXY      | Limit the number of IP addresses a proxy names resolves into.
@@ -85,6 +85,8 @@ CVMFS_OOM_SCORE_ADJ             | Set the Linux kernel's out-of-memory killer pr
                                 | for the CernVM-FS client [-1000 - 1000].
 CVMFS_PROXY_RESET_AFTER         | Delay in seconds after which CernVM-FS will retry the primary proxy group
                                 | in case of a fail-over to another group.
+CVMFS_PROXY_SHARD               | If set to *yes*, shard requests across all proxies within the current
+                                | load-balancing group using consistent hashing.
 CVMFS_PROXY_TEMPLATE            Overwrite the default proxy template in Geo-API calls.  Only needed for debugging.
 CVMFS_PUBLIC_KEY                Colon-separated list of repository signing keys.
 CVMFS_QUOTA_LIMIT               Soft-limit of the cache in Megabyte.
@@ -169,17 +171,14 @@ CVMFS_FILE_MBYTE_LIMIT              | Maximum number of megabytes for a publishe
 CVMFS_FORCE_REMOUNT_WARNING         | Enable/disable warning through ``wall`` and grace period before forcefully
                                     | remounting a CernVM-FS repository on the release managere machine.
 CVMFS_GARBAGE_COLLECTION            Enables repository garbage collection |br| (Stratum~0 only | if set to *true*)
-CVMFS_GENERATE_LEGACY_BULK_CHUNKS   | Deprecated, set to *true* to enable generation of whole-file objects for large files.
 CVMFS_GC_DELETION_LOG               | Log file path to track all garbage collected objects during sweeping
                                     | for bookkeeping or debugging
 CVMFS_GEO_DB_FILE                   Path to externally updated location of geolite2 city database, or 'None' for no database.
 CVMFS_GEO_LICENSE_KEY               A license key for downloading the geolite2 city database from maxmind.
 CVMFS_GID_MAP                       Path of a file for the mapping of file owner group ids.
 CVMFS_HASH_ALGORITHM                | Define which secure hash algorithm should be used by CernVM-FS for CAS objects
                                     | (supported are: *sha1*, *rmd160* and *shake128*)
-CVMFS_IGNORE_SPECIAL_FILES          Set to *true* to skip special files during publish without aborting.
-CVMFS_IGNORE_XDIR_HARDLINKS         | Deprecated, defaults to *true*
-                                    | hardlinks are found. Instead automatically break the hardlinks across directories.
+CVMFS_IGNORE_SPECIAL_FILES          Set to *true* to skip special files (pipes, sockets, block device and character device files) during publish without aborting.
 CVMFS_INCLUDE_XATTRS                Set to *true* to process extended attributes
 CVMFS_MAX_CHUNK_SIZE                Maximal size of a file chunk in bytes (see also *CVMFS_USE_FILE_CHUNKING*)
 CVMFS_MAXIMAL_CONCURRENT_WRITES     Maximal number of concurrently processed files during publishing.
@@ -233,6 +232,20 @@ X509_CERT_DIR                       | Directory file with CA certificates for HT
                                     | defaults to /etc/grid-security/certificates (see :ref:`sct_data`)
 =================================== ====================================================================================
 
+Deprecated parameters
+^^^^^^^^^^^^^^^^^^^^^
+
+Will be removed in future versions.
+
+=================================== ====================================================================================
+**Parameter**                       **Meaning**
+=================================== ====================================================================================
+CVMFS_GENERATE_LEGACY_BULK_CHUNKS   | Deprecated, set to *true* to enable generation of whole-file objects for large files.
+CVMFS_IGNORE_XDIR_HARDLINKS         | Deprecated, defaults to *true*
+                                    | hardlinks are found. Instead automatically break the hardlinks across directories.
+=================================== ====================================================================================
+
+
 Format of CVMFS_UPSTREAM_STORAGE
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

diff --git a/conf.py b/conf.py
@@ -51,17 +51,17 @@
 
 # General information about the project.
 project = u'CernVM-FS'
-copyright = u'2021, CernVM Team'
+copyright = u'2022, CernVM Team'
 author = u'CernVM Team'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The short X.Y version.
-version = u'2.9'
+version = u'2.10'
 # The full version, including alpha/beta/rc tags.
-release = u'2.9.4'
+release = u'2.10.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -129,7 +129,7 @@
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
-#html_theme_options = {}
+# html_theme_options = {}
 
 # Add any paths that contain custom themes here, relative to this directory.
 #html_theme_path = []
@@ -155,6 +155,12 @@
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ['_static']
 
+# These paths are either relative to html_static_path
+# or fully qualified paths (eg. https://...)
+html_css_files = [
+    'css/custom.css',
+]
+
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied
 # directly to the root of the documentation.

diff --git a/cpt-configure.rst b/cpt-configure.rst
@@ -440,14 +440,18 @@ system-wide settings. Instead of a single proxy, CernVM-FS uses a *chain
 of load-balanced proxy groups*. The CernVM-FS proxies are set by the
 ``CVMFS_HTTP_PROXY`` parameter.
 
-Proxy groups are used for load-balancing among several proxies of equal priority.
-Starting with the first group, one proxy within a group is selected at random.
-If it fails, CernVM-FS automatically switches to another proxy from the current
-group. If all proxies in a group have failed, CernVM-FS switches to
-the next proxy group. After probing the last proxy group in the chain,
-the first is probed again. To avoid endless loops, for each file
-download the number of switches is limited by the total number of
-proxies.
+Proxy groups are used for load-balancing among several proxies of equal
+priority.  Starting with the first group, one proxy within a group is selected
+at random.  By default, this randomly selected proxy will used for all requests.
+If :ref:`proxy sharding <sct_proxy_sharding>` is enabled, then the proxy is
+instead selected on a per-request basis to distribute the requests across all
+proxies within the current group.
+
+If a proxy fails, CernVM-FS automatically switches to another proxy from the
+current group.  If all proxies in a group have failed, CernVM-FS switches to the
+next proxy group.  After probing the last proxy group in the chain, the first is
+probed again. To avoid endless loops, for each file download the number of
+switches is limited by the total number of proxies.
 
 Proxies within the same group are separated by a pipe character ``|``, while
 groups are separated from each other by a semicolon character ``;`` [#]_.
@@ -568,13 +572,13 @@ configuration. It is by default turned on on macOS and turned off on Linux.
 Network Path Selection
 ~~~~~~~~~~~~~~~~~~~~~~
 
-This section summarized the CernVM-FS mechanics to select a network path from
+This section summarizes the CernVM-FS mechanics to select a network path from
 the client through an HTTP forward proxy to an HTTP endpoint. At any given point
-in time, there is only one combination of web proxy and web host that all new
-requests are going to utilize. In this section, it is this combination of proxy
-and host that is called "network path". The network path is chosen from the
-collection of web proxies and hosts in the CernVM-FS configuration according to
-the following rules.
+in time, there is only one combination of web proxy and web host that a new
+request will utilize. In this section, it is this combination of proxy and host
+that is called "network path". The network path is chosen from the collection of
+web proxies and hosts in the CernVM-FS configuration according to the following
+rules.
 
 Host Selection
 ^^^^^^^^^^^^^^
@@ -591,6 +595,24 @@ load-balance group, a proxy is chosen at random. DNS proxy names that resolve to
 multiple IP addresses are automatically transformed into a proxy load-balance
 group, whose maximum size can be limited by ``CVMFS_MAX_IPADDR_PER_PROXY``.
 
+.. _sct_proxy_sharding:
+
+Proxy Sharding
+^^^^^^^^^^^^^^
+
+In the default (non-sharded) configuration, each CernVM-FS client will
+independently choose a single proxy to be used for all requests.  For sites with
+many clients that are likely to access the same content, this can result in
+unnecessary duplication of cached content across multiple proxies.
+
+If proxy sharding is enabled via the ``CVMFS_PROXY_SHARD`` parameter, all
+proxies within a load-balancing group are used concurrently.  Each proxy handles
+a subset of the requests.  Proxies are selected using consistent hashing so that
+multiple clients will independently select the same proxy for a given request,
+to maximise cache efficiency.  If any proxy fails, CernVM-FS automatically
+removes it from the load-balancing group and distributes its requests evenly
+across the remaining proxies.
+
 Failover Rules
 ^^^^^^^^^^^^^^
 
@@ -1151,7 +1173,7 @@ The ``cvmfs_talk`` command provides a way to control a currently running
 CernVM-FS process and to extract information about the status of the
 corresponding mount point. Most of the commands are for special purposes
 only or covered by more convenient commands, such as
-``cvmfs_config showconfig`` or ``cvmfs_config stat``. Three commands might
+``cvmfs_config showconfig`` or ``cvmfs_config stat``. Four commands might
 be of particular interest though.
 
 ::
@@ -1176,6 +1198,13 @@ the client experiences cache thrashing.
 prints the internal status information and performance counters. It can
 be helpful for performance engineering.
 
+::
+
+    cvmfs_talk -i <repo> remount
+
+starts the catalog update routine.
+When using ``remount sync`` the system waits for the new file system snapshot to be served (if there is a new one).
+
 Other
 ~~~~~
 

diff --git a/cpt-containers.rst b/cpt-containers.rst
@@ -240,20 +240,22 @@ A template ``config.toml`` file looks like this:
 
 ::
 
+    version = 2
+
     # Source of image layers
     repository = "unpacked.cern.ch"
     absolute-mountpoint = "/cvmfs/unpacked.cern.ch"
 
     # Ask containerd to use this particular snapshotter
     [plugins."io.containerd.grpc.v1.cri".containerd]
-    snapshotter = "cvmfs-snapshotter"
-    disable_snapshot_annotations = false
+        snapshotter = "cvmfs-snapshotter"
+        disable_snapshot_annotations = false
 
     # Set the communication endpoint between containerd and the snapshotter
     [proxy_plugins]
-    [proxy_plugins.cvmfs-snapshotter]
-        type = "snapshot"
-        address = "/run/containerd-cvmfs-grpc/containerd-cvmfs-grpc.sock"
+        [proxy_plugins.cvmfs]
+            type = "snapshot"
+            address = "/run/containerd-cvmfs-grpc/containerd-cvmfs-grpc.sock"
 
 
 Note that if only the repository is specified under the key value ``repository``, the mountpoint

diff --git a/cpt-quickstart.rst b/cpt-quickstart.rst
@@ -48,7 +48,7 @@ To install the CVMFS package run
 
 ::
 
-    sudo dnf install https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.9.4/cvmfs-2.9.4-1.fc34.x86_64.rpm https://ecsft.cern.ch/dist/cvmfs/cvmfs-config/cvmfs-config-default-latest.noarch.rpm
+    sudo dnf install https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.10.0/cvmfs-2.10.0-1.fc34.x86_64.rpm https://ecsft.cern.ch/dist/cvmfs/cvmfs-config/cvmfs-config-default-latest.noarch.rpm
 
 
 Docker Container
@@ -65,7 +65,7 @@ or with
 
 ::
 
-    curl https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.9.4/cvmfs-service-2.9.4-1.x86_64.docker.tar.gz | docker load
+    curl https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.10.0/cvmfs-service-2.10.0-1.x86_64.docker.tar.gz | docker load
 
 Run the container as a system service with
 
@@ -98,7 +98,7 @@ Download the CernVM-FS client package in the terminal in order to avoid signatur
 
 ::
 
-    curl -o ~/Downloads/cvmfs-2.9.4.pkg https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.9.4/cvmfs-2.9.4.pkg
+    curl -o ~/Downloads/cvmfs-2.10.0.pkg https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.10.0/cvmfs-2.10.0.pkg
 
 Install the CernVM-FS package by opening the .pkg file and reboot.
 Future releases will provide a signed and notarized package.
@@ -195,6 +195,28 @@ On Intel architectures, it needs a gcc :math:`\geq 4.2` compiler, on ARMv7 a gcc
     make
     sudo make install
 
+
+Building with local libraries
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+For development purposes it might be useful to use locally installed libraries instead of using default system libraries.
+This can be done by defining variables during the ``cmake`` configuration step.
+The correct naming of the variables can be found in ``cmake/Modules``.
+For example, in case of Fuse3 following variables must be set: ``FUSE3_INCLUDE_DIR`` and ``FUSE3_LIBRARY``.
+
+Furthermore, ``CMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON`` must be set, otherwise will ``sudo make install`` strip all linked libraries that point to none-system libraries.
+
+Example code for building CernVM-FS with locally built Fuse3 and including the CernVM-FS unittests and gateway:
+::
+
+    cmake -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON \
+          -D BUILD_UNITTESTS=ON -D BUILD_GATEWAY=ON \
+          -D FUSE3_INCLUDE_DIR=/usr/local/include/ \
+          -D FUSE3_LIBRARY=/usr/local/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 \
+          ../
+    make
+    sudo make install
+
+
 Troubleshooting
 ---------------