Merge remote-tracking branch 'upstream/2.11' into stable

apx-contact.rst

@@ -4,7 +4,7 @@ Contact Information
 For support requests and bug reports, please submit a GitHub issue in our
 `issue tracker <https://github.com/cvmfs/cvmfs/issues>`_.
 
-Together with bug reports, please attach a "bugreport tarball", which is created
+Together with bug reports, please attach a "bug report tarball", which is created
 with ``sudo cvmfs_config bugreport``.
 
 Discourse Forum

apx-issues.rst

@@ -1,9 +1,18 @@
 Known Issues
 ============
 
+Slow Performance with a Very Large File Descriptor Limit
+--------------------------------------------------------
+
+Before CernVM-FS 2.11, having the file descriptor limit set to a very large number
+will result in a very slow ``cvmfs`` performance in certain situations.
+This is due to certain operations looping over all possible file descriptors,
+instead of just the used/opened ones.
+This issue is resolved in CernVM-FS 2.11.
+
 Publisher nodes with AUFS and XFS
 ---------------------------------
 
-If the /tmp file system is on xfs, the publisher node cannot be used with AUFS.
+If the ``/tmp`` file system is on xfs, the publisher node cannot be used with AUFS.
 On such systems, adding the mount option ``xino=/dev/shm/aufs.xino`` can be
 a workaround. In general, new repositories should use OverlayFS if available.

apx-parameters.rst

@@ -29,8 +29,11 @@ CVMFS_BACKOFF_MAX               Maximum backoff in seconds when retrying to down
 CVMFS_CATALOG_WATERMARK         | Try to release pinned catalogs when their number surpasses the given watermark.
                                 | Defaults to 1/4 CVMFS_NFILES; explicitly set by shrinkwrap.
 CVMFS_CACHE_BASE                Location (directory) of the CernVM-FS cache.
+CVMFS_CACHE_REFCOUNT            If set to *yes*, deduplicate open file descriptors by refcounting.
+CVMFS_CACHE_SYMLINKS            If set to *yes*, enables symlink caching in the kernel. 
 CVMFS_CHECK_PERMISSIONS         If set to *no*, disable checking of file ownership and permissions (open all files).
 CVMFS_CLAIM_OWNERSHIP           If set to *yes*, allows CernVM-FS to claim ownership of files and directories.
+CVMFS_CPU_AFFINITY              Comma-separated list to set CPU affinity for all ``cvmfs`` components. 
 CVMFS_DEBUGLOG                  If set, run CernVM-FS in debug mode and write a verbose log the the specified file.
 CVMFS_DEFAULT_DOMAIN            | The default domain will be automatically appended to repository names
                                 | when given without a domain.
@@ -60,8 +63,13 @@ CVMFS_HTTP_PROXY                | Chain of HTTP proxy groups used by CernVM-FS.
 CVMFS_IGNORE_SIGNATURE          When set to *yes*, don't verify CernVM-FS file catalog signatures.
 CVMFS_INITIAL_GENERATION        Initial inode generation.  Used for testing.
 CVMFS_INSTRUMENT_FUSE           | When set to *true* gather performance statistics about the FUSE callbacks.
-                                | The results are displayed with `cvmfs_talk internal affairs`.
+                                | The results are displayed with ``cvmfs_talk internal affairs``.
 CVMFS_NFS_INTERLEAVED_INODES    In NFS mode, use only inodes of the form :math:`an+b`, specified as "b%a".
+CVMFS_INFLUX_EXTRA_FIELDS       Static fields always attached to the (absolute) output of the InfluxDB Telemetry Aggregator
+CVMFS_INFLUX_EXTRA_TAGS         Static tags always attached to the (absolute + delta) output of the InfluxDB Telemetry Aggregator
+CVMFS_INFLUX_HOST               Host name or IP address of the receiver of the InfluxDB Telemetry Aggregator
+CVMFS_INFLUX_METRIC_NAME        Name of the measurement of the InfluxDB Telemetry Aggregator
+CVMFS_INFLUX_PORT               Port of the host (receiver) of the InfluxDB Telemetry Aggregator
 CVMFS_IPFAMILY_PREFER           Which IP protocol to prefer when connecting to proxies.  Can be either 4 or 6.
 CVMFS_KCACHE_TIMEOUT            Timeout in seconds for path names and file attributes in the kernel file system buffers.
 CVMFS_KEYS_DIR                  | Directory containing \*.pub files used as repository signing keys.
@@ -74,7 +82,7 @@ CVMFS_MAX_IPADDR_PER_PROXY      | Limit the number of IP addresses a proxy names
 CVMFS_MAX_RETRIES               Maximum number of retries for a given proxy/host combination.
 CVMFS_MAX_SERVERS               Limit the number of (geo sorted) stratum 1 servers that are effectively used.
 CVMFS_MAX_TTL                   Maximum file catalog TTL in minutes.  Can overwrite the TTL stored in the catalog.
-CVMFS_MEMCACHE_SIZE             Size of the CernVM-FS meta-data memory cache in Megabyte.
+CVMFS_MEMCACHE_SIZE             Size of the CernVM-FS metadata memory cache in Megabyte.
 CVMFS_MOUNT_RW                  | Mount CernVM-FS as a read/write file system.  Write operations will fail
                                 | but this option can workaround faulty ``open()`` flags.
 CVMFS_NFILES                    Maximum number of open file descriptors that can be used by the CernVM-FS process.
@@ -105,6 +113,7 @@ CVMFS_SEND_INFO_HEADER          If set to *yes*, include the cvmfs path of downl
 CVMFS_SERVER_CACHE_MODE         Enable special cache semantics for a client used as a publisher's repository base line.
 CVMFS_SERVER_URL                Semicolon-separated chain of Stratum~1 servers.
 CVMFS_SHARED_CACHE              If set to *no*, makes a repository use an exclusive cache.
+CVMFS_STREAMING_CACHE           If set to *yes*, use a download manager to download regular files on read.
 CVMFS_STRICT_MOUNT              If set to *yes*, mount only repositories that are listed in ``CVMFS_REPOSITORIES``.
 CVMFS_SUID                      If set to *yes*, enable suid magic on the mounted repository. Requires mounting as root.
 CVMFS_SYSLOG_FACILITY           | If set to a number between 0 and 7, uses the corresponding
@@ -113,17 +122,22 @@ CVMFS_SYSLOG_LEVEL              | If set to 1 or 2, sets the syslog level for Ce
                                 | LOG_DEBUG or LOG_INFO respectively.
 CVMFS_SYSTEMD_NOKILL            | If set to *yes*, modify the command line to ``@vmfs2 ...`` in order to
                                 | act as a systemd lowlevel storage manager.
+CVMFS_TELEMETRY_RATE            Rate in seconds for Telemetry Aggregator to send the telemetry. Minimum send rate >= 5 sec.
+CVMFS_TELEMETRY_SEND            ``ON`` to activate Telemetry Aggregator.
 CVMFS_TIMEOUT                   Timeout in seconds for HTTP requests with a proxy server.
 CVMFS_TIMEOUT_DIRECT            Timeout in seconds for HTTP requests without a proxy server.
 CVMFS_TRACEFILE                 If set, enables the tracer and trace file system calls to the given file.
 CVMFS_USE_GEOAPI                Request order of Stratum 1 servers and fallback proxies via Geo-API.
+CVMFS_USE_SSL_SYSTEM_CA         | When connecting to an HTTPS endpoints,
+                                | it will load the certificates provided by the system.
 CVMFS_USER                      Sets the ``gid`` and ``uid`` mount options. Don't touch or overwrite.
 CVMFS_USYSLOG                   | All messages that normally are logged to syslog are re-directed to the given file.
                                 | This file can grow up to 500kB and there is one step of log rotation.
                                 | Required for $\mu$CernVM.
+CVMFS_XATTR_PRIVILEGED_GIDS     Comma-separated list of (main) group IDs that are allowed to access the extended attributes listed by ``CVMFS_XATTR_PROTECTED_XATTRS``.
+CVMFS_XATTR_PROTECTED_XATTRS    Comma-separated list of extended attributes (full name, e.g. ``user.fqrn``) that are only accessible by ``root`` and the group IDs listed by ``CVMFS_XATTR_PRIVILEGED_GIDS``.
 CVMFS_WORKSPACE                 Set the local directory for storing special files (defaults to the cache directory).
-CVMFS_USE_SSL_SYSTEM_CA         | When connecting to an HTTPS endpoints,
-                                | it will load the certificates provided by the system.
+CVMFS_WORLD_READABLE            Override posix read permissions to make files in repository globally readable
 =============================== ========================================================================================
 
 
@@ -179,6 +193,8 @@ CVMFS_GEO_LICENSE_KEY               A license key for downloading the geolite2 c
 CVMFS_GID_MAP                       Path of a file for the mapping of file owner group ids.
 CVMFS_HASH_ALGORITHM                | Define which secure hash algorithm should be used by CernVM-FS for CAS objects
                                     | (supported are: *sha1*, *rmd160* and *shake128*)
+CVMFS_HTTP_TRACING                  Activates that a tracing header is attached to each CURL request. Consists of ``uid``, ``pid``, and ``gid``. Default is ``off``.
+CVMFS_HTTP_TRACING_HEADERS          Adds additional static, user-defined tracing headers. Format: ``key1:val1|key2:val2|key3:val3``. Needs ``CVMFS_HTTP_TRACING`` to be set to ``on``.
 CVMFS_IGNORE_SPECIAL_FILES          Set to *true* to skip special files (pipes, sockets, block device and character device files) during publish without aborting.
 CVMFS_INCLUDE_XATTRS                Set to *true* to process extended attributes
 CVMFS_MAX_CHUNK_SIZE                Maximal size of a file chunk in bytes (see also *CVMFS_USE_FILE_CHUNKING*)
@@ -215,7 +231,7 @@ CVMFS_UNION_DIR                     | Mount point of the union file system for c
                                     | Here, changes to the repository are performed
                                     | (see :ref:`sct_repocreation_update`).
 CVMFS_UNION_FS_TYPE                 | Defines the union file system to be used for the repository.
-                                    | (currently `aufs` and `overlayfs` are fully supported)
+                                    | (only ``overlayfs`` is fully supported, ``aufs`` has no active support anymore)
 CVMFS_UPLOAD_STATS_DB               | Publish repository statistics data file to the Stratum 0 /stats location
 CVMFS_UPLOAD_STATS_PLOTS            | Publish repository statistics plots and webpage to the Stratum 0 /stats location (requires ROOT)
 CVMFS_UPSTREAM_STORAGE              | Upstream spooler description defining the basic upstream storage type
@@ -225,6 +241,8 @@ CVMFS_USER                          The user name that owns and manipulates the
 CVMFS_VIRTUAL_DIR                   | Set to *true* to enable the hidden, virtual ``.cvmfs/snapshots`` directory
                                     | containing entry points to all named tags.
 CVMFS_VOMS_AUTHZ                    Membership requirement (e.g. VOMS authentication) to be added into the file catalogs
+CVMFS_STATFS_CACHE_TIMEOUT          | Caching time of  ``statfs()`` in seconds (no caching by default).
+                                    | Calling ``statfs()`` in high frequency can be expensive.
 CVMFS_STATISTICS_DB                 | SQLite file path to store the statistics. Default is
                                     | ``/var/spool/cvmfs/<REPO_NAME>/stats.db`` .
 CVMFS_PRINT_STATISTICS              Set to *true* to enable statistics printing to the standard output.

apx-rpms.rst

@@ -14,12 +14,12 @@ The CernVM-FS software is available in form of several packages:
     cern.ch, egi.eu, and opensciencegrid.org domains.
 
 **cvmfs-config-none**
-    Empty package to satisfy the cvmfs-config requirement of the cvmfs
+    Empty package to satisfy the ``cvmfs-config`` requirement of the cvmfs
     package without actually installing any configuration.
 
 **cvmfs**
     Contains the Fuse module and additional client tools. It has
-    dependencies to at least one of the cvmfs-config-\ :math:`\cdots`
+    dependencies to at least one of the ``cvmfs-config-...``
     packages.
 
 **cvmfs-fuse3**
@@ -50,12 +50,12 @@ The CernVM-FS software is available in form of several packages:
     on a publisher node.
 
 **cvmfs-notify**
-    Websockets frontend for used for repository update notifications. Supposed
+    WebSockets frontend for used for repository update notifications. Supposed
     to be co-located with a RabbitMQ service.
 
 **kernel-...-.aufs21**
-    Scientific Linux 6 kernel with aufs. Required for SL6 based
-    Stratum 0 servers.
+    Scientific Linux 6 kernel with ``aufs``. Required for SL6 based
+    Stratum 0 servers. (Note: no active support for ``aufs`` anymore)
 
 **cvmfs-shrinkwrap**
     Stand-alone utility to export file system trees into containers for HPC