Merge branch 'stable' of github.com:cvmfs/doc-cvmfs into stable

cvmfs · Feb 15, 2021 · 778867d · 778867d
2 parents acc2777 + 47f8e46
commit 778867d
Show file tree

Hide file tree

Showing 13 changed files with 274 additions and 130 deletions.
diff --git a/apx-contact.rst b/apx-contact.rst
@@ -9,10 +9,8 @@ a CERN account, you can register yourself for an
 Together with bug reports, please attach a "bugreport tarball", which is created
 with ``sudo cvmfs_config bugreport``.
 
-Mailing Lists
--------------
+Discourse Forum
+---------------
 
-The ``cvmfs-talk@cern.ch`` mailing list is used for general discussions about
-CernVM-FS. The ``cvmfs-announce@cern.ch`` mailing list is a low-volume list used
-for important news such as new releases. The ``cvmfs-testing@cern.ch`` mailing
-list is used to announce new releases in pre-production state.
+For announcements, discussions, and support please join us in the
+`CernVM Forum <https://cernvm-forum.cern.ch>`_.
diff --git a/apx-parameters.rst b/apx-parameters.rst
@@ -117,7 +117,7 @@ CVMFS_USYSLOG                   | All messages that normally are logged to syslo
                                 | This file can grow up to 500kB and there is one step of log rotation.
                                 | Required for $\mu$CernVM.
 CVMFS_WORKSPACE                 Set the local directory for storing special files (defaults to the cache directory).
-CVMFS_USE_SSL_SYSTEM_CA         | When connecting to an HTTPS endpoints, 
+CVMFS_USE_SSL_SYSTEM_CA         | When connecting to an HTTPS endpoints,
                                 | it will load the certificates provided by the system.
 =============================== ========================================================================================
 
@@ -157,15 +157,15 @@ CVMFS_DONT_CHECK_OVERLAYFS_VERSION  | Disable checking of OverlayFS version befo
                                     | (see :ref:`sct_reporequirements`)
 CVMFS_ENFORCE_LIMITS                | Set to *true* to cause exceeding \*LIMIT variables to be fatal to a publish
                                     | instead of a warning
+CVMFS_EXTENDED_GC_STATS             | Set to *true* to keep track of the volume of garbage collected files (increases GC running time)
 CVMFS_EXTERNAL_DATA                 | Set to *true* to mark repository to contain external data
                                     | that is served from an external HTTP server
 CVMFS_FILE_MBYTE_LIMIT              | Maximum number of megabytes for a published file, default value: 1024
                                     | (see also *CVMFS_ENFORCE_LIMITS*)
 CVMFS_FORCE_REMOUNT_WARNING         | Enable/disable warning through ``wall`` and grace period before forcefully
                                     | remounting a CernVM-FS repository on the release managere machine.
 CVMFS_GARBAGE_COLLECTION            Enables repository garbage collection |br| (Stratum~0 only | if set to *true*)
-CVMFS_GENERATE_LEGACY_BULK_CHUNKS   | Set to *false* to disable generation of whole-file objects for large files.
-                                    | Requires clients >= 2.1.7.
+CVMFS_GENERATE_LEGACY_BULK_CHUNKS   | Deprecated, set to *true* to enable generation of whole-file objects for large files.
 CVMFS_GC_DELETION_LOG               | Log file path to track all garbage collected objects during sweeping
                                     | for bookkeeping or debugging
 CVMFS_GEO_DB_FILE                   Path to externally updated location of geolite2 city database, or 'None' for no database.
@@ -174,7 +174,7 @@ CVMFS_GID_MAP                       Path of a file for the mapping of file owner
 CVMFS_HASH_ALGORITHM                | Define which secure hash algorithm should be used by CernVM-FS for CAS objects
                                     | (supported are: *sha1*, *rmd160* and *shake128*)
 CVMFS_IGNORE_SPECIAL_FILES          Set to *true* to skip special files during publish without aborting.
-CVMFS_IGNORE_XDIR_HARDLINKS         | If set to *true*, do not abort the publish operation when cross-directory
+CVMFS_IGNORE_XDIR_HARDLINKS         | Deprecated, defaults to *true*
                                     | hardlinks are found. Instead automatically break the hardlinks across directories.
 CVMFS_INCLUDE_XATTRS                Set to *true* to process extended attributes
 CVMFS_MAX_CHUNK_SIZE                Maximal size of a file chunk in bytes (see also *CVMFS_USE_FILE_CHUNKING*)
@@ -187,6 +187,7 @@ CVMFS_NUM_UPLOAD_TASKS              | Number of threads used to commit data to s
 CVMFS_NUM_WORKERS                   | Maximal number of concurrently downloaded files during a Stratum1 pull operation
                                     | (Stratum~1 only).
 CVMFS_PUBLIC_KEY                    Colon-separated path to the public key file(s) or directory(ies) of the repository to be replicated. (Stratum 1 only).
+CVMFS_PRINT_STATISTICS              | Set to *true* to show publisher statistics on the console
 CVMFS_REPLICA_ACTIVE                | Stratum1-only: Set to *no* to skip this repository when executing
                                     | ``cvmfs_server snapshot -a``
 CVMFS_REPOSITORY_NAME               The fully qualified name of the specific repository.
@@ -199,6 +200,8 @@ CVMFS_SNAPSHOT_GROUP                | Group name for subset of repositories used
                                     | Added with ``cvmfs_server add-replica -g``.
 CVMFS_SPOOL_DIR                     | Location of the upstream spooler scratch directories;
                                     | the read-only CernVM-FS moint point and copy-on-write storage reside here.
+CVMFS_STATISTICS_DB                 | Set a custom path for the publisher statistics database
+CVMFS_STATS_DB_DAYS_TO_KEEP         | Sets the pruning interval for the publisher statistics database
 CVMFS_STRATUM0                      URL of the master copy (*stratum0*) of this specific repository.
 CVMFS_STRATUM1                      URL of the Stratum1 HTTP server for this specific repository.
 CVMFS_SYNCFS_LEVEL                  | Controls how often ``sync`` will by called by ``cvmfs_server`` operations.
@@ -209,6 +212,7 @@ CVMFS_UNION_DIR                     | Mount point of the union file system for c
                                     | (see :ref:`sct_repocreation_update`).
 CVMFS_UNION_FS_TYPE                 | Defines the union file system to be used for the repository.
                                     | (currently `aufs` and `overlayfs` are fully supported)
+CVMFS_UPLOAD_STATS_DB               | Publish repository statistics plots to the Stratum 0 /stats location
 CVMFS_UPSTREAM_STORAGE              | Upstream spooler description defining the basic upstream storage type
                                     | and configuration.
 CVMFS_USE_FILE_CHUNKING             Allows backend to split big files into small chunks (*true* | *false*)

diff --git a/apx-serverinfra.rst b/apx-serverinfra.rst
@@ -22,8 +22,6 @@ setup and tools to be in place:
 
    -   kernel 4.2.x or later.
    -   RHEL7.3 kernel (for OverlayFS)
-   -   Custom kernel compilation with `aufs` support the kernel (see
-       Section :ref:`sct_customkernelinstall`)
 
 -  Backend storage location available through HTTP
 

diff --git a/cpt-details.rst b/cpt-details.rst
@@ -652,12 +652,18 @@ requests are translated into the ``pread()`` system call.
 getxattr
 ~~~~~~~~
 
-CernVM-FS uses extended attributes to display additional repository
-information. There are several supported attributes:
+CernVM-FS uses synthetic extended attributes to display additional repository
+information. There are the following supported magic attributes:
+
+**catalog\_counters**
+    Like ``repo_counters`` but only for the nested catalog that hosts the given path.
 
 **chunks**
     Number of chunks of a regular file.
 
+**chunk\_list**
+    Hashes and sizes of the chunks of a regular (large) file.
+
 **compression**
     Compression algorithm, for regular files only.  Either "zlib" or "none".
 

diff --git a/cpt-enter.rst b/cpt-enter.rst
@@ -0,0 +1,45 @@
+.. _cpt_enter:
+
+Ephemeral Writable Container
+============================
+
+**Note:** this feature is still considered experimental.
+
+The CernVM-FS ephemeral writable container can provide a short-lived shell with writable access to a regular, read-only CernVM-FS repository.
+A writable CernVM-FS mountpoint is normally a functionality that only publisher nodes provide.
+With the ephemeral writable container, this capability becomes available to every regular client.
+
+The ephemeral writable container requires the ``cvmfs-server`` package to be installed.
+Provided that the target repository is already mounted, a writable shell is opened with
+
+::
+
+    cvmfs_server enter <repository name> [-- <command>]
+
+Changes to the writable mountpoint are only stored locally.
+The changes are discarded when the shell is closed.
+In a future release it will be possible to publish changes directly to a gateway.
+
+Repository changes in the writable shell can be shown with
+
+::
+
+    cvmfs_server diff --worktree
+
+Before closing the shell, changes can be manually copied to a publisher node for publication.
+This helps with building and deploying non-relocatable packages to CernVM-FS.
+
+The ephemeral writable container uses Linux user namespaces and fuse-overlayfs in order to construct the writable repository mountpoint.
+Therefore, it requires a recent enough kernel.
+The vanilla kernel >= 4.18 and the EL 8 kernel are known to work.
+
+The container creates a session directory in ``$HOME/.cvmfs`` to store temporary files and changes to the repository.
+By default, the session directory is removed when exiting the shell.
+It can be preserved with the ``--keep-session`` parameter.
+If only the logs should be preserved, use the ``--keep-logs`` parameter instead.
+
+If necessary, the container can be opened as fake root user using the ``root`` option.
+
+Note that by default a dedicated CernVM-FS cache directory is created for the lifetime of the ephemeral container.
+It can be desireable to use a shared cache directory across several invocations of the ``cvmfs_server enter`` command.
+To do so, use the ``--cvmfs-config <config file>`` parameter and set ``CVMFS_CACHE_BASE=/common/path`` in the passed configuration file.
diff --git a/cpt-quickstart.rst b/cpt-quickstart.rst
@@ -2,8 +2,9 @@ Getting Started
 ===============
 
 This section describes how to install the CernVM-FS client.
-The CernVM-FS client is supported on x86, x86\_64, and ARM architectures running Linux or Mac OS X \ :math:`\geq 10.12`.
-There is experimental support for Power 8 and RISC-V.
+The CernVM-FS client is supported on x86, x86\_64, and ARM architectures running Linux and
+macOS \ :math:`\geq 10.14` as well as on Windows Services for Linux (WSL2).
+There is experimental support for Power and RISC-V architectures.
 
 Overview
 --------
@@ -14,6 +15,7 @@ That means that starting from the base directory /cvmfs different repositories a
 A repository will be automatically unmounted after some automount-defined idle time.
 On macOS, mounting and un-mounting of the CernVM-FS is done by the user with ``sudo mount -t cvmfs /cvmfs/...`` commands.
 
+
 Getting the Software
 --------------------
 The CernVM-FS source code and binary packages are available from the `CernVM website <https://cernvm.cern.ch/portal/filesystem/downloads>`_.
@@ -46,13 +48,74 @@ To install the CVMFS package run
 
 ::
 
-    sudo dnf install https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.7.5/cvmfs-2.7.5-1.fc29.x86_64.rpm https://ecsft.cern.ch/dist/cvmfs/cvmfs-config/cvmfs-config-default-latest.noarch.rpm
+    sudo dnf install https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.8.0/cvmfs-2.8.0-1.fc29.x86_64.rpm https://ecsft.cern.ch/dist/cvmfs/cvmfs-config/cvmfs-config-default-latest.noarch.rpm
+
+
+Docker Container
+~~~~~~~~~~~~~~~~
+
+The CernVM-FS service container can expose the /cvmfs directory tree to the host.
+Import the container with
+
+::
+
+    docker pull cvmfs/service
 
+or with
+
+::
+
+    curl https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.8.0/cvmfs-service-2.8.0-1.x86_64.docker.tar.gz | docker load
+
+Run the container as a system service with
+
+::
+
+    docker run -d --rm \
+      -e CVMFS_CLIENT_PROFILE=single \
+      -e CVMFS_REPOSITORIES=sft.cern.ch,... \
+      --cap-add SYS_ADMIN \
+      --device /dev/fuse \
+      --volume /cvmfs:/cvmfs:shared \
+      cvmfs/service:2.8.0-1
+
+Use ``docker stop`` to unmount the /cvmfs tree.
+Note that if you run multiple nodes (a cluster), you should use ``-e CVMFS_HTTP_PROXY`` to set a proper site proxy as described further down.
 
 Mac OS X
 ~~~~~~~~
 
-Install the CernVM-FS package by opening the .pkg file.
+On Mac OS X, CernVM-FS is based on `macFUSE <http://osxfuse.github.io>`_.
+Note that as of macOS 11 Big Sur, `kernel extensions need to be enabled <https://support.apple.com/guide/mac-help/change-startup-disk-security-settings-a-mac-mchl768f7291/mac>`_
+to install macFUSE.
+Verify that fuse is available with
+
+::
+
+    kextstat | grep -i fuse
+
+Download the CernVM-FS client package in the terminal in order to avoid signature warnings
+
+::
+
+    curl -o ~/Downloads/cvmfs-2.8.0.pkg https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.8.0/cvmfs-2.8.0.pkg
+
+Install the CernVM-FS package by opening the .pkg file and reboot.
+Future releases will provide a signed and notarized package.
+
+
+Windows / WSL2
+~~~~~~~~~~~~~~
+
+Follow the `Windows instructions <https://docs.microsoft.com/en-us/windows/wsl/install-win10>`_ to install the Windows Subsytem for Linux (WSL2).
+Install any of the Linux distributions and follow the instructions for the distribution in this guide.
+Whenever you open the Linux distribution, run
+
+::
+
+    sudo cvmfs_config wsl2_start
+
+to start the CernVM-FS service.
 
 
 Setting up the Software
@@ -75,13 +138,15 @@ NB: For OpenSUSE uncomment the line ``#+dir:/etc/auto.master.d/`` in the file /e
 Mac OS X
 ~~~~~~~~
 
-On Mac OS X, CernVM-FS is based on `OSXFuse <http://osxfuse.github.io>`_.
-It is not integrated with autofs hence mount the individual repositories using
+Due to the lack of autofs on macOS, mount the individual repositories manually like
 
 ::
 
-    sudo mkdir -p /cvmfs/cms.cern.ch
-    sudo mount -t cvmfs cms.cern.ch /cvmfs/cms.cern.ch
+    sudo mkdir -p /cvmfs/cvmfs-config.cern.ch
+    sudo mount -t cvmfs cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch
+
+For optimal configuration settings, mount the config repository before any other repositories.
+
 
 Create default.local
 ~~~~~~~~~~~~~~~~~~~~
@@ -93,7 +158,13 @@ Select the desired repositories by setting ``CVMFS_REPOSITORIES=repo1,repo2,...`
 
     CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch
 
-Specify the HTTP proxy servers on your site with
+For an individual workstation or laptop, set
+
+::
+
+    CVMFS_CLIENT_PROFILE=single
+
+If you setup a cluster of cvmfs nodes, specify the HTTP proxy servers on your site with
 
 ::
 
@@ -102,8 +173,7 @@ Specify the HTTP proxy servers on your site with
 If you're unsure about the proxy names, set ``CVMFS_HTTP_PROXY=DIRECT``.
 This should *only* be done for a small number of clients (< 5), because large numbers can put a heavy load on the Stratum 1 servers and result, amongst others, in poorer performance for the client.
 For the syntax of more complex HTTP proxy settings, see :ref:`sct_network`.
-
-If you install CernVM-FS on a single, possibly roaming computer, you can specifiy ``CVMFS_CLIENT_PROFILE=single`` in which case CernVM-FS will choose a suitable proxy automatically.
+If there are no HTTP proxies yet at your site, see :ref:`cpt_squid` for instructions on how to set them up.
 
 Verify the file system
 ~~~~~~~~~~~~~~~~~~~~~~

diff --git a/cpt-releasenotes.rst b/cpt-releasenotes.rst
@@ -36,11 +36,11 @@ CernVM-FS Client Service Container
 
 As of this release, we provide the CernVM-FS client in addition to the regular
 distribution packages as a minimal Docker container.  The container is available
-from Dockerhub and as a standalone tarball to be used with `docker load`.
+`from Dockerhub <https://hub.docker.com/r/cvmfs/service>`_ and as a standalone tarball to be used with `docker load`.
 
 The service container can be used to expose /cvmfs mountpoint to the container
 host. It is meant for fully containerized Linux distributions such as
-Fedora CoreOS. It can also be deployed as a _deamon set_ to provide /cvmfs
+Fedora CoreOS. It can also be deployed as a Kubernetes DaemonSet to provide /cvmfs
 to pods in kubernetes clusters.
 
 
@@ -62,7 +62,7 @@ be started like
     cvmfs_server publish
 
 Template transactions are an experimental feature for the time being.
-Note that template transaction do not yet work for remote publishers connected
+Note that template transactions do not yet work for remote publishers connected
 to a gateway. This limitation will be lifted in a future release.
 
 
@@ -73,10 +73,10 @@ This release provides the new ``cvmfs_server enter`` command that can open
 an ephemeral, writable container for a repository. The ephemeral container
 effectively promotes a regular, read-only client mountpoint under /cvmfs to
 a writable mountpoint. Starting an ephemeral publish container does not require
-a full installation of publisher node; availability of the ``cvmfs``,
+a full installation of a publisher node; availability of the ``cvmfs``,
 ``cvmfs-server``, and ``fuse-overlayfs`` packages is sufficient.
 
-Opening an ephemeral publish container is an unprivilged operation. It does
+Opening an ephemeral publish container is an unprivileged operation. It does
 require, however, the relatively recent "user namespaces" and "unprivileged
 fuse mounts" features from the Linux kernel.  CentOS 8, for instance, provides
 a recent enough kernel.
@@ -117,7 +117,7 @@ Bug Fixes
 
   * Server: fix ``cvmfs_server ingest`` into root directory
 
-  * Server" fix ingestion of hardlinked catalog markers
+  * Server: fix ingestion of hardlinked catalog markers
     (`CVM-1931 <https://sft.its.cern.ch/jira/browse/CVM-1931>`_)
 
   * Server: refuse non-regular .cvmfscatalog files during publish
@@ -128,6 +128,9 @@ Bug Fixes
   * Server, stratum 1: fix stuck Apache processes with disabled geo API
     (`CVM-1956 <https://sft.its.cern.ch/jira/browse/CVM-1956>`_)
 
+  * Gateway: fix corrupted catalog when a nested catalog is replaced by a symlink
+    (`CVM-1930 <https://sft.its.cern.ch/jira/browse/CVM-1930>`_)
+
   * Gateway: fix accidental creation of undeletable content caused by improper
     handling of the reflog