write 2.6 release notes

cvmfs · Mar 12, 2019 · 962d32d · 962d32d
1 parent 0aba57c
commit 962d32d
Show file tree

Hide file tree

Showing 3 changed files with 172 additions and 8 deletions.
diff --git a/apx-parameters.rst b/apx-parameters.rst
@@ -48,6 +48,7 @@ CVMFS_HOST_RESET_AFTER          See ``CVMFS_PROXY_RESET_AFTER``.
 CVMFS_HTTP_PROXY                Chain of HTTP proxy groups used by CernVM-FS. Necessary. Set to ``DIRECT`` if you don't use proxies.
 CVMFS_IGNORE_SIGNATURE          When set to *yes*, don't verify CernVM-FS file catalog signatures.
 CVMFS_INITIAL_GENERATION        Initial inode generation.  Used for testing.
+CVMFS_NFS_INTERLEAVED_INODES    In NFS mode, use only inodes of the form :math:`an+b`, specified as "b%a".
 CVMFS_IPFAMILY_PREFER           Which IP protocol to prefer when connecting to proxies.  Can be either 4 or 6.
 CVMFS_KCACHE_TIMEOUT            Timeout for path names and file attributes in the kernel file system buffers.
 CVMFS_KEYS_DIR                  Directory containing \*.pub files used as repository signing keys.  If set, this parameter has precedence over ``CVMFS_PUBLIC_KEY``.
@@ -132,6 +133,7 @@ CVMFS_MAX_CHUNK_SIZE                Maximal size of a file chunk in bytes |br| (
 CVMFS_MAXIMAL_CONCURRENT_WRITES     Maximal number of concurrently processed files during publishing.
 CVMFS_MIN_CHUNK_SIZE                Minimal size of a file chunk in bytes |br| (see also *CVMFS_USE_FILE_CHUNKING*)
 CVMFS_NESTED_KCATALOG_LIMIT         Maximum thousands of files allowed in nested catalogs, default 500 |br| (see also *CVMFS_ROOT_KCATALOG_LIMIT* and *CVMFS_ENFORCE_LIMITS*)
+CVMFS_NUM_UPLOAD_TASKS              Number of threads used to commit data to storage during publication. Currently only used by the local backend.
 CVMFS_NUM_WORKERS                   Maximal number of concurrently downloaded files during a Stratum1 pull operation (Stratum~1 only).
 CVMFS_PUBLIC_KEY                    Path to the public key file of the repository to be replicated. (Stratum 1 only).
 CVMFS_REPLICA_ACTIVE                Stratum1-only: Set to *no* to skip this repository when executing ``cvmfs_server snapshot -a``

diff --git a/cpt-configure.rst b/cpt-configure.rst
@@ -682,6 +682,37 @@ The performance will benefit from large RAM on the NFS server
 (:math:`\geq` 16GB) and CernVM-FS caches hosted on an SSD
 hard drive.
 
+.. _sct_nfs_interleaved:
+
+Export of /cvmfs with Cray DVS
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+On Cray DVS and possibly other systems that export /cvmfs as a whole instead of
+individual repositories as separate volumes, an additional effort is needed to
+ensure that inodes are distinct from each other across multiple repositories.
+The ``CVMFS_NFS_INTERLEAVED_INODES`` parameter can be used to configure
+repositories to only issue inodes of a particular residue class. To ensure
+pairwise distinct inodes across repositories, each repository should be
+configured with a different residue class.  For instance, in order to avoid
+inode clashes between the atlas.cern.ch and the cms.cern.ch repositories,
+there can be a configuration file /etc/cvmfs/config.d/atlas.cern.ch.local
+with
+
+::
+
+    CVMFS_NFS_INTERLEAVED_INODES=0%2 # issue inodes 0, 2, 4, ...
+
+and a configuration file /etc/cvmfs/config.d/cms.cern.ch.local with
+
+::
+
+    CVMFS_NFS_INTERLEAVED_INODES=1%2 # issue inodes 1, 3, 5, ...
+
+
+The maximum number of possibly exported repositories needs to be known in
+advance. The ``CVMFS_NFS_INTERLEAVED_INODES`` only has an effect in NFS mode.
+
+
 Shared NFS Maps (HA-NFS)
 ~~~~~~~~~~~~~~~~~~~~~~~~
 

diff --git a/cpt-releasenotes.rst b/cpt-releasenotes.rst
@@ -1,14 +1,145 @@
 Release Notes for CernVM-FS 2.6.0
 =================================
 
+CernVM-FS 2.6 is a feature release that comes with performance improvements,
+new functionality, and bugfixes. We would like to thank Dave Dykstra (FNAL),
+Brian Bockelman and Derek Weitzel (U. Nebraska) and Nick Hazekamp
+(U. Notre Dame) for their contributions to this release!
+
+This release comes with several new, experimental satellite serivces around
+the CernVM-FS core components:
+
+  1) DUCC (Daemon that unpacks container images into CernVM-FS). This new
+  component automatizes the publication of container images from a Docker
+  registry into CernVM-FS.
+
+  2) Repository change notification system, which is complementary to the
+  default, pull-based approach to propagate repository updates.
+
+  3) Repository shrinkwrap utility. This new utility allows for exporting large
+  parts of a CernVM-FS repository to an external file system or a "fat image" as
+  they are used in some HPC environments.
+
 Together with CernVM-FS 2.6.0, we also release the CernVM-FS Repository Gateway
-version 1.0. The increased version number indicates that we are ready to assure backward compatibility for the component, in line with `semantic versioning <https://semver.org/>`_ rules.
+version 1.0. The increased version number indicates that we are ready to assure
+backward compatibility for the component, in line with
+`semantic versioning <https://semver.org/>`_ rules.
+
+Other notable changes include
+
+  * A new server command ``cvmfs_server ingest``, that can be use to directly
+    publish tarballs without extracting them first.
+
+  * A new server parameter ``CVMFS_PRINT_STATISTICS``. If enabled, publication
+    and garbage collection runs report key metrics about processed data to the
+    standard output and to a sqlite file.
+
+  * Various improvements for the S3 backend.
+
+  * A file system call tracer that can be enabled on the client in order to
+    log the file system accesses to a repository.
+
+  * Support for baerer token authentication in addition to X.509 authentication
+    for protected repositories.
+
+  * Various new routines in libcvmfs that provide access to cvmfs specific
+    meta-data, such as the nested catalog structure.
+
+
+As with previous releases, upgrading should be seamless just by installing the
+new package from the repository. As usual, we recommend to update only a few
+worker nodes first and gradually ramp up once the new version proves to work
+correctly. Please take special care when upgrading a client in NFS mode.
+
+For Stratum 0 servers, all transactions must be closed before upgrading.
+For Stratum 1 servers, there should be no running snapshots during the upgrade.
+After the software upgrade, publisher nodes (``stratum 0'') require doing
+``cvmfs_server migrate`` for each repository.
+
+**Note**: if the configuration of the repostory publisher node is handled by a
+configuration management system (Puppet, Chef, ...), please see Section
+:ref:`sct_manual_migration`.
+
+
+Bug Fixes
+---------
+
+  * Client, macOS: fix hang during ``cvmfs_config reload``
+
+  * Client: fix credentials handling on HTTP retries for protected repositories
+    (`CVM-1660 <https://sft.its.cern.ch/jira/browse/CVM-1660>`_)
+
+  * Client: add support for ``CVMFS_NFS_INTERLEAVED_INODES`` parameter
+    (`CVM-1561 <https://sft.its.cern.ch/jira/browse/CVM-1561>`_, `Documentation <cpt-configure.html#sct-nfs-interleaved>`_)
+
+  * Server: prevent following dirtab entries that point outside the repository
+    (`CVM-1608 <https://sft.its.cern.ch/jira/browse/CVM-1608>`_)
+
+  * Server, S3: fix rare crash during file upload
+
+  * Server, S3: throttle upload frequency on HTTP 429 "too many requests"
+    replies (`CVM-1584 <https://sft.its.cern.ch/jira/browse/CVM-1584>`_)
+
+  * Fix building on macOS Mojave
+
+  * Fix warnings and errors in Debian packaging
+
+
+Other Improvements
+------------------
+
+  * Client: log more details on HTTP host and proxy connection errors
+    (`CVM-1662 <https://sft.its.cern.ch/jira/browse/CVM-1662>`_)
+
+  * Client: generally replace ``@fqrn@`` and ``@org@`` in configuration files
+    (`CVM-1526 <https://sft.its.cern.ch/jira/browse/CVM-1526>`_)
+
+  * Client: new parameter ``CVMFS_CATALOG_WATERMARK`` to unpin catalogs when
+    their number surpasses the given watermark
+
+  * Server: make publication process less likely to run out of file descriptors
+    on the read-only union file system mount
+
+  * Server, S3: retry upload requests on HTTP 502 errors in order to better
+    handle high load on load-balancers
+
+  * Server, S3: add support for CVMFS_S3_PEEK_BEFORE_PUT parameter, enabled by
+    default (`CVM-1584 <https://sft.its.cern.ch/jira/browse/CVM-1584>`_)
+
+  * Server: reduce number of I/O operations to the local storage backend
+
+  * Server: add support for ``CVMFS_NUM_UPLOAD_TASKS`` parameter for local
+    storage backend
+
+
+
+.. _sct_manual_migration:
+
+Manual Migration from 2.5.2 Release Manager Machines
+----------------------------------------------------
+
+If you do not want to use ``cvmfs_server migrate`` to automatically upgrade,
+publisher nodes that maintain Stratum 0 repositories can be migrated from
+version 2.5.2 with the following steps:
+
+  1. Ensure that there are no open transactions and garbage collection processes
+     before updating the server software and during the repository layout
+     migration.
+
+  2. Install the ``cvmfs-server`` 2.6.0 package.
+
+  3. For each repository: adjust
+     /etc/cvmfs/repositories.d/<REPOSITORY>/client.conf and add the
+     ``CVMFS_NFILES=65536`` parameter.
+
+  4. Update /etc/cvmfs/repositories.d/<REPOSITORY>/server.conf and set
+     ``CVMFS_CREATOR_VERSION=141``
+
+In agreement with the repository owner it's recommended to make a test publish
+
+::
 
-In addition to various bugfixes, this version introduces an improved repository
-configuration format, which should greatly simplify the initial setup of the
-repository gateway in a large number of cases. Please consult the updated
-repository gateway documentation: :ref:`cpt_repository_gateway`.
+    cvmfs_server transaction <REPOSITORY>
+    cvmfs_server publish <REPOSITORY>
 
-This release also includes an experimental preview of the new notification
-system. Please consult the relevant documentation page for more details:
-:ref:`cpt_notification_system`.
+before resuming normal operation.