diff --git a/docs/TheBook/pom.xml b/docs/TheBook/pom.xml index ce0b522640e..3768dcbb32a 100644 --- a/docs/TheBook/pom.xml +++ b/docs/TheBook/pom.xml @@ -42,7 +42,7 @@ ${html-header.path} ${html-footer.path} true - SMARTYPANTS,AUTOLINKS,TABLES,FENCED_CODE_BLOCKS,STRIKETHROUGH,TASKLISTITEMS,EXTANCHORLINKS + SMARTYPANTS,AUTOLINKS,TABLES,FENCED_CODE_BLOCKS,STRIKETHROUGH,TASKLISTITEMS,EXTANCHORLINKS,TOC css,images diff --git a/docs/TheBook/src/main/markdown/config-PoolManager.md b/docs/TheBook/src/main/markdown/config-PoolManager.md index 18959de63c4..b77d99635da 100644 --- a/docs/TheBook/src/main/markdown/config-PoolManager.md +++ b/docs/TheBook/src/main/markdown/config-PoolManager.md @@ -1,22 +1,6 @@ THE POOLMANAGER SERVICE ================================== -## Table of Contents - -* [The Pool Selection Mechanism](#the-pool-selection-mechanism) - - * [Links](#links) - * [Examples](#examples) - -* [The Partition Manager](#the-partition-manager) - - * [Overview](#overview) - * [Managing Partitions](#managing-partitions) - * [Using Partitions](#using-partitions) - * [Classic Partitions](#classic-partitions) - -* [Link Groups](#link-groups) - The heart of a dCache System is the `poolmanager`. When a user performs an action on a file - reading or writing - a `transfer request` is sent to the dCache system. The `poolmanager` then decides how to handle this request. If a file the user wishes to read resides on one of the storage-pools within the dCache system, it will be transferred from that pool to the user. If it resides on several pools, the file will be retrieved from one of the pools determined by a configurable load balancing policy. If all pools the file is stored on are busy, a new copy of the file on an idle pool will be created and this pool will answer the request. @@ -42,6 +26,10 @@ chapter we will describe the commands allowed in this file. > `poolmanager` in zookeeper and reads `poolmanager.conf` file only if > configuration in zookeeper is missing, e.g. on the first start. +----- +[TOC bullet hierarchy] +----- + ## THE POOL SELECTION MECHANISM The PSU is responsible for finding the set of pools which can be used for a specific transfer-request. By telling the PSU which pools are permitted for which type of transfer-request, the administrator of the dCache system can adjust the system to any kind of scenario: Separate organizations served by separate pools, special pools for writing the data to a tertiary storage system, pools in a DMZ which serves only a certain kind of data (e.g., for the grid). This section explains the mechanism employed by the PSU and shows how to configure it with several examples. diff --git a/docs/TheBook/src/main/markdown/config-SRM.md b/docs/TheBook/src/main/markdown/config-SRM.md index 30b5b233c96..fd065f7a8bf 100644 --- a/docs/TheBook/src/main/markdown/config-SRM.md +++ b/docs/TheBook/src/main/markdown/config-SRM.md @@ -1,43 +1,6 @@ CHAPTER 13. dCache STORAGE RESOURCE MANAGER =========================================== -## Table of Contents - -+ [Introduction](#introduction) -+ [Configuring the srm service](#configuring-the-srm-service) - - [The Basic Setup](#the-basic-setup) - [Important srm configuration options](#important-srm-configuration-options) - -+ [Utilization of Space Reservations for Data Storage](#utilization-of-space-reservations-for-data-storage) - - [Properties of Space Reservation](#properties-of-space-reservation) -+ [dCache specific concepts](#dcache-specific-concepts) - - [Activating SRM SpaceManager](#activating-srm-spacemanager) - [Explicit and Implicit Space Reservations for Data Storage in dCache](#explicit-and-implicit-space-reservations-for-data-storage-in-dcache) - -+ [SpaceManager configuration](#spacemanager-configuration) - - [SRM SpaceManager and Link Groups](#srm-spacemanager-and-link-groups) - [Making a Space Reservation](#making-a-space-reservation) - [SRM configuration for experts](#srm-configuration-for-experts) - -+ [Configuring the PostgreSQL Database](#configuring-the-postgresql-database) - - [SRM or srm monitoring on a separate node](#srm-or-srm-monitoring-on-a-separate-node) - -+ [General SRM Concepts (for developers)](#general-srm-concepts-(for-developers)) - - [The SRM service](#the-srm-service) - [Space Management Functions](#space-management-functions) - [Data Transfer Functions](#data-transfer-functions) - [Request Status Functions](#request-status-functions) - [Directory Functions](#directory-functions) - [Permission functions](#permission-functions) - -## Introduction - Storage Resource Managers (SRMs) are middleware components whose function is to provide dynamic space allocation and file management on shared storage components on the Grid. SRMs support protocol negotiation and a reliable replication mechanism. @@ -55,6 +18,10 @@ The main benefits of using HTTPS rather than HTTP over GSI is that HTTPS is a st and has support for sessions, improving latency in case a client needs to connect to the same server multiple times. +----- +[TOC bullet hierarchy] +----- + ## CONFIGURING THE SRM SERVICE ### BASIC SETUP diff --git a/docs/TheBook/src/main/markdown/config-acl.md b/docs/TheBook/src/main/markdown/config-acl.md index 8b78c81bc29..2aa3c239d7c 100644 --- a/docs/TheBook/src/main/markdown/config-acl.md +++ b/docs/TheBook/src/main/markdown/config-acl.md @@ -1,18 +1,14 @@ ACLs in dCache =========================== -Table of Contents ------------------ - -* [Introduction](#introduction) -* [Configuring ACL support](#configuring-acl-support) -* [Setting and getting ACLs](#setting-and-getting-acls) -* [Accessing ACL over NFS mount](#accessing-acl-over-nfs-mount) - dCache includes support for Access Control Lists (ACLs). This support is conforming to the [NFS version 4 Protocol specification](https://tools.ietf.org/rfc/rfc7530.txt). This chapter provides some background information and details on configuring dCache to use ACLs and how to administer the resulting system. +----- +[TOC bullet hierarchy] +----- + ## Introduction dCache allows control over namespace operations (e.g., creating new files and directories, deleting items, renaming items) and data operations (reading data, writing data) using the standard Unix permission model. In this model, files and directories have both owner and group-owner attributes and a set of permissions that apply to the owner, permissions for users that are members of the group-owner group and permissions for other users. diff --git a/docs/TheBook/src/main/markdown/config-admin.md b/docs/TheBook/src/main/markdown/config-admin.md index fd40240b411..40899fbcb93 100644 --- a/docs/TheBook/src/main/markdown/config-admin.md +++ b/docs/TheBook/src/main/markdown/config-admin.md @@ -5,6 +5,10 @@ THE ADMIN SERVICE > > Only commands described in this documentation should be used for the administration of a dCache system. +----- +[TOC bullet hierarchy] +----- + ## FIRST STEPS dCache has a powerful administration interface. Administration protocol is implemented as `admin` cell that diff --git a/docs/TheBook/src/main/markdown/config-alarms.md b/docs/TheBook/src/main/markdown/config-alarms.md index 701354ac880..c8b89571ecd 100644 --- a/docs/TheBook/src/main/markdown/config-alarms.md +++ b/docs/TheBook/src/main/markdown/config-alarms.md @@ -1,20 +1,6 @@ CHAPTER 16. THE ALARMS SERVICE ============================== -## Table of Contents - -* [The Basic Setup](#the-basic-setup) - [Configure where the alarms service is Running](#configure-where-the-alarms-service-is-running) - [Types of Alarms](#types-of-alarms) - [Alarm Priority](#alarm-priority) - [Working with Alarms: Shell Commands](#working-with-alarms-shell-commands) - [Working with Alarms: Admin Commands](#working-with-alarms-admin-commands) - [Working with Alarms: The DCache-View Alarms Tab](#working-with-alarms-the-dcache-view-alarms-tab) -* [Advanced Service Configuration: Enabling Automatic Cleanup](#advanced-service-configuration-enabling-automatic-cleanup) -* [Advanced Service Configuration: Enabling Email Alerts](#advanced-service-configuration-enabling-email-alerts) -* [Miscellaneous Properties of the Alarms Service](#miscellaneous-properties-of-the-alarm-service) -* [Alarms SPI](#alarms-spi--service-provider-interface-) - dCache has an `alarms` backend service which records failures (*alarms*) requiring more or less urgent intervention. The service stores alarms by either in an XML file or an RDBMS. The service is turned off by default. @@ -26,6 +12,10 @@ both the timestamp range and the filtering and sorting of alarms by fields. Admins also have access to buttons which allow them to mark alarms as closed or to delete them altogether. +----- +[TOC bullet hierarchy] +----- + ## THE BASIC SETUP It is not necessary to run the `alarms` service in a separate domain, though diff --git a/docs/TheBook/src/main/markdown/config-billing.md b/docs/TheBook/src/main/markdown/config-billing.md index 92cc873683c..9260687b37b 100644 --- a/docs/TheBook/src/main/markdown/config-billing.md +++ b/docs/TheBook/src/main/markdown/config-billing.md @@ -1,14 +1,6 @@ CHAPTER 15. THE BILLING SERVICE =============================== -## Table of Contents - -* [The Billing Log Files](#the-billing-log-files) -* [The Billing Database](#the-billing-database) -* [Billing Histogram Data](#billing-histogram-data) -* [Billing Records](#billing-records) -* [Upgrading a Previous Installation](#upgrading-a-previous-installation) - dCache has built-in monitoring capabilities which provide an overview of the activity and performance of the installation’s doors and pools. There are two options for how this data can be represented and stored: - a set of log files written to a known location @@ -18,6 +10,10 @@ dCache has built-in monitoring capabilities which provide an overview of the act These options can be enabled simultaneously. If the database option is selected, the data in those tables will also be displayed as a set of histogram plots on the installation's web page. +----- +[TOC bullet hierarchy] +----- + ## THE BILLING LOG FILES If you installed dCache following the instructions in the Chapter [Installing dCache](install.md) you enabled the BILLING in the domain where the HTTPD service is running (see the extract of the layout file). diff --git a/docs/TheBook/src/main/markdown/config-chimera.md b/docs/TheBook/src/main/markdown/config-chimera.md index ecf6589ae75..796b897b582 100644 --- a/docs/TheBook/src/main/markdown/config-chimera.md +++ b/docs/TheBook/src/main/markdown/config-chimera.md @@ -1,18 +1,9 @@ CHIMERA ================== -## Table of Contents - -- [Mounting Chimera through NFS](#mounting-chimera-through-nfs) -- [Using dCap with a mounted file system](#using-dcap-with-a-mounted-file-system) -- [Communicating with Chimera](#communicating-with-chimera) -- [IDs](#ids) -- [Directory Tags](#directory-tags) - - [Create, List and Read Directory Tags if the Namespace is not Mounted](#create-list-and-read-directory-tags-if-the-namespace-is-not-mounted) - - [Create, List and Read Directory Tags if the Namespace is Mounted](#create-list-and-read-directory-tags-if-the-namespace-is-mounted) - - [Directory Tags and Command Files](#directory-tags-and-command-files) - - [Directory Tags for dCache](#directory-tags-for-dcache) - - [Storage Class and Directory Tags](#storage-class-and-directory-tags) +----- +[TOC bullet hierarchy] +----- The inner dCache components talk to the namespace via a module called `PnfsManager`, which in turn communicates with the Chimera database using a thin Java layer. In addition to `PnfsManager` a direct access to the file system view is provided by an `NFSv3` and `NFSv4.1` server. Clients can `NFS`-mount the namespace locally. This offers the opportunity to use OS-level tools like `ls, mkdir, mv` for Chimera. Direct I/O-operations like `cp` and `cat` are possible with the `NFSv4.1 door`. diff --git a/docs/TheBook/src/main/markdown/config-frontend.md b/docs/TheBook/src/main/markdown/config-frontend.md index 5e1e43e2d69..86e46aa6f8a 100644 --- a/docs/TheBook/src/main/markdown/config-frontend.md +++ b/docs/TheBook/src/main/markdown/config-frontend.md @@ -1,5 +1,9 @@ CHAPTER 17. dCache Frontend Service -===================================== +=================================== + +----- +[TOC bullet hierarchy] +----- The Frontend service is the dCache service (cell) responsible for serving data to clients via HTTP/REST. The default port on which it diff --git a/docs/TheBook/src/main/markdown/config-ftp.md b/docs/TheBook/src/main/markdown/config-ftp.md index 84655be6b20..63bf6772a25 100644 --- a/docs/TheBook/src/main/markdown/config-ftp.md +++ b/docs/TheBook/src/main/markdown/config-ftp.md @@ -1,16 +1,13 @@ dCache as an FTP Server ======================= -Table of Contents - -- [Introduction](#introduction) -- [The Control Channel](#the-control-channel) -- [Data transfers](#data-transfers) -- [Configuration examples](#configuration-examples) - This chapter explains how to configure dCache to allow FTP, a common network protocol that many clients support. +----- +[TOC bullet hierarchy] +----- + ## Introduction FTP is a long established protocol that allows clients to transfer diff --git a/docs/TheBook/src/main/markdown/config-gplazma.md b/docs/TheBook/src/main/markdown/config-gplazma.md index 094e489fbf3..e37adc610b3 100644 --- a/docs/TheBook/src/main/markdown/config-gplazma.md +++ b/docs/TheBook/src/main/markdown/config-gplazma.md @@ -1,37 +1,12 @@ Chapter 10. Authorization in dCache =================================== -## Table of Contents - -+ [Basics](#basics) -+ [Configuration](#configuration) - - - [Plug-ins](#plug-ins) - -+ [Using X.509 Certificates](#using-x509-certificates) - - - [CA Certificates](#ca-certificates) - - [User Certificate](#user-certificate) - - [Host Certificate](#host-certificate) - - [VOMS Proxy Certificate](#voms-proxy-certificate) - -+ [Using OpenID Connect](#using-openid-connect) - -+ [Configuration files](#configuration-files) - - - [storage-authzdb](#storage-authzdb) - - [The gplazmalite-vorole-mapping plug-in](#the-gplazmalite-vorole-mapping-plug-in) - - [Authorizing a VO](#authorizing-a-vo) - - [The kpwd plug-in](#the-kpwd-plug-in) - - [The gridmap plug-in](#the-gridmap-plug-in) - -+ [gPlazma specific dCache configuration](#gplazma-specific-dcache-configuration) - - - [Enabling Username/Password Access for WebDAV](#enabling-username-password-access-for-webdav) - - [gPlazma config example to work with authenticated webadmin](#gplazma-config-example-to-work-with-authenticated-webadmin) - To limit access to data, dCache comes with an authentication and authorization interface called `gPlazma2`. gPlazma is an acronym for Grid-aware PLuggable AuthorZation Management. Earlier versions of dCache worked with `gPlazma1` which has now been completely removed from dCache. So if you are upgrading, you have to reconfigure `gPlazma` if you used `gPlazma1` until now. +----- +[TOC bullet hierarchy] +----- + ## Basics Though it is possible to allow anonymous access to dCache it is usually desirable to authenticate users. The user then has to connect to one of the different doors (e.g., `GridFTP door, dCap door`) and login with credentials that prove his identity. In Grid-World these credentials are very often `X.509` certificates, but dCache also supports other methods like username/password and kerberos authentication. diff --git a/docs/TheBook/src/main/markdown/config-history.md b/docs/TheBook/src/main/markdown/config-history.md index 33443e95ff9..501cdb35123 100644 --- a/docs/TheBook/src/main/markdown/config-history.md +++ b/docs/TheBook/src/main/markdown/config-history.md @@ -1,6 +1,10 @@ CHAPTER 18. dCache History Service ===================================== +----- +[TOC bullet hierarchy] +----- + The purpose of this service is to provide a disk-backed cache for time-windowed state data extracted from backend dCache components, most importantly, pools. diff --git a/docs/TheBook/src/main/markdown/config-hopping.md b/docs/TheBook/src/main/markdown/config-hopping.md index 79f5951d1f2..4148febd1b9 100644 --- a/docs/TheBook/src/main/markdown/config-hopping.md +++ b/docs/TheBook/src/main/markdown/config-hopping.md @@ -1,15 +1,9 @@ CHAPTER 9. FILE HOPPING ======================= - -## Table of Contents - -* [File Hopping on arrival from outside dCache](#file-hopping-on-arrival-from-outside-dcache) - - [File mode of replicated files](#file-mode-of-replicated-files) - [File Hopping managed by the PoolManager](#file-hopping-managed-by-the-poolmanager) - [File Hopping managed by the HoppingManager](#file-hopping-managed-by-the-hoppingmanager) - +----- +[TOC bullet hierarchy] +----- File hopping is a collective term in dCache, summarizing the possibility of having files being transferred between dCache pools triggered by a variety of conditions. The most prominent examples are: diff --git a/docs/TheBook/src/main/markdown/config-hsm.md b/docs/TheBook/src/main/markdown/config-hsm.md index 8f940826afb..b9da2b13e50 100644 --- a/docs/TheBook/src/main/markdown/config-hsm.md +++ b/docs/TheBook/src/main/markdown/config-hsm.md @@ -1,43 +1,12 @@ Chapter 8: The dCache Tertiary Storage System Interface ============================================ - -Table of Contents - -* [Introduction](#introduction) -* [Scope of this chapter](#scope-of-this-chapter) -* [Requirements for a Tertiary Storage System](#requirements-for-a-tertiary-storage-system) - - [Migrating Tertiary Storage Systems with a file system interface.](#migrating-tertiary-storage-systems-with-a-file-system-interface.) - [Tertiary Storage Systems with a minimalistic PUT, GET and REMOVE interface](#tertiary-storage-systems-with-a-minimalistic-put-get-and-remove-interface) - -* [How dCache interacts with a Tertiary Storage System](#how-dcache-interacts-with-a-tertiary-storage-system) -* [Details on the TSS-support executable](#details-on-the-tss-support-executable) - - [Summary of command line options](#summary-of-command-line-options) - [Summary of return codes](#summary-of-return-codes) - [The executable and the STORE FILE operation](#the-executable-and-the-store-file-operation) - [The executable and the FETCH FILE operation](#the-executable-and-the-fetch-file-operation) - [The executable and the REMOVE FILE operation](#the-executable-and-the-remove-file-operation) - -* [Configuring pools to interact with a Tertiary Storage System](#configuring-pools-to-interact-with-a-tertiary-storage-system) - - [The dCache layout files](#the-dcache-layout-files) - [What happens next](#what-happens-next) - -* [How to Store-/Restore files via the Admin Interface](#how-to-store-/restore-files-via-the-admin-interface) -* [How to monitor what’s going on](#how-to-monitor-what’s-going-on) - - [Log Files](#log-files) - [Obtain information via the dCache Command Line Admin Interface](#obtain-information-via-the-dcache-command-line-admin-interface) - -* [Example of an executable to simulate a tape backend](#example-of-an-executable-to-simulate-a-tape-backend) - - -## INTRODUCTION - One of the features dCache provides is the ability to migrate files from its disk repository to one or more connected Tertiary Storage Systems (TSS) and to move them back to disk when necessary. Although the interface between dCache and the TSS is kept simple, dCache assumes to interact with an intelligent TSS. dCache does not drive tape robots or tape drives by itself. More detailed requirements to the storage system are described in one of the subsequent paragraphs. +----- +[TOC bullet hierarchy] +----- + ## SCOPE OF THIS CHAPTER This document describes how to enable a standard dCache installation to interact with a Tertiary Storage System. In this description we assume that diff --git a/docs/TheBook/src/main/markdown/config-info-provider.md b/docs/TheBook/src/main/markdown/config-info-provider.md index ff70a2c031b..952b5c01489 100644 --- a/docs/TheBook/src/main/markdown/config-info-provider.md +++ b/docs/TheBook/src/main/markdown/config-info-provider.md @@ -1,15 +1,6 @@ CHAPTER 19. GLUE INFO PROVIDER ============================== -## Table of Contents -+ [Internal collection of information](#internal-collection-of-information) -+ [Configuring the info service](#configuring-the-info-service) -+ [Testing the info provider](#testing-the-info-provider) -+ [Publishing dCache information](#publishing-dcache-information) -+ [Troubleshooting BDII problems](#troubleshooting-bdii-problems) -+ [Updating information](#updating-information) - - The GLUE information provider supplied with dCache provides the information about the dCache instance in a standard format called GLUE. This is necessary so that WLCG infrastructure (such as FTS) and clients using WLCG tools can discover the dCache instance and use it correctly. The process of configuring the info-provider is designed to have the minimum overhead so you can configure it manually; however, you may prefer to use an automatic configuration tool, such as YAIM. @@ -24,6 +15,10 @@ This chapter describes how to enable and test the dCache-internal collection of > > Please be aware that changing information provider may result in a brief interruption to published information. This may have an adverse affect on client software that make use of this information. +----- +[TOC bullet hierarchy] +----- + ## INTERNAL COLLECTION OF INFORMATION The info-provider takes as much information as possible from dCache. To achieve this, it needs the internal information-collecting service, `info`, to be running and a means to collect that information: `httpd`. Make sure that both the `httpd` and `info` services are running within your dCache instance. By default, the `info` service is started on the admin-node; but it is possible to configure dCache so it runs on a different node. You should run only one `info` service per dCache instance. diff --git a/docs/TheBook/src/main/markdown/config-message-passing.md b/docs/TheBook/src/main/markdown/config-message-passing.md index 66f717e236a..38924acf6ce 100644 --- a/docs/TheBook/src/main/markdown/config-message-passing.md +++ b/docs/TheBook/src/main/markdown/config-message-passing.md @@ -1,6 +1,10 @@ Message passing =============== +----- +[TOC bullet hierarchy] +----- + The dCache system is divided into cells which communicate with each other via messages. Cells run inside domains and cells communicate by passing messages to each other. Domains are connected through cell tunnels which exchange messages over TCP. Each domain runs in a separate Java virtual machine and each cell is run as a separate thread therein. Domain names have to be unique. The domains communicate with each other via `TCP` using connections that are established at start-up. The topology is controlled by the location manager service. When configured, all domains connect with a core domain, which routes all messages to the appropriate domains. This forms a star topology. diff --git a/docs/TheBook/src/main/markdown/config-missing-files.md b/docs/TheBook/src/main/markdown/config-missing-files.md index 5379e817cb1..101e3a67e98 100644 --- a/docs/TheBook/src/main/markdown/config-missing-files.md +++ b/docs/TheBook/src/main/markdown/config-missing-files.md @@ -1,6 +1,10 @@ THE MISSING FILES SERVICE ========================= +----- +[TOC bullet hierarchy] +----- + ## Introduction When a user requests to read a file that doesn't exist, diff --git a/docs/TheBook/src/main/markdown/config-nfs.md b/docs/TheBook/src/main/markdown/config-nfs.md index e54035f9f3b..79d1ebd442c 100644 --- a/docs/TheBook/src/main/markdown/config-nfs.md +++ b/docs/TheBook/src/main/markdown/config-nfs.md @@ -1,20 +1,16 @@ dCache as NFSv4.1 Server ==================================== -Table of Contents - -- [Setting up](#setting-up) -- [Exporting filesystem](#exporting-filesystem) -- [Configuring NFSv4.1 door with GSS-API support](#configuring-nfsv4.1-door-with-gss-api-support) -- [Configuring principal-id mapping for NFS access](#configuring-principal-id-mapping-for-nfs-access) -- [Managing group ids](#managing-group-ids) - This chapter explains how to configure dCache in order to access it via the `NFSv4.1` protocol, allowing clients to mount dCache and perform POSIX IO using standard `NFSv4.1` clients. > **Important** > > The `pNFS` mentioned in this chapter is the protocol `NFSv4.1/pNFS` and not the namespace pnfs. +----- +[TOC bullet hierarchy] +----- + ## Setting up To allow file transfers in and out of dCache using NFSv4.1/pNFS, a new NFSv4.1 door must be started. This door acts then as the mount point for NFS clients. diff --git a/docs/TheBook/src/main/markdown/config-resilience.md b/docs/TheBook/src/main/markdown/config-resilience.md index 40e1562f429..827f24d455a 100644 --- a/docs/TheBook/src/main/markdown/config-resilience.md +++ b/docs/TheBook/src/main/markdown/config-resilience.md @@ -1,6 +1,10 @@ The Resilience Service ====================== +----- +[TOC bullet hierarchy] +----- + ## Configuring the resilience service ### Activating resilience diff --git a/docs/TheBook/src/main/markdown/config-stage-protection.md b/docs/TheBook/src/main/markdown/config-stage-protection.md index 8f8f8358334..1d54f1b9451 100644 --- a/docs/TheBook/src/main/markdown/config-stage-protection.md +++ b/docs/TheBook/src/main/markdown/config-stage-protection.md @@ -1,16 +1,15 @@ CHAPTER 20. STAGE PROTECTION ============================= -## Table of Contents - -+ [Configuration of Stage Protection](#configuration-of-stage-protection) -+ [Definition of the White List](#definition-of-the-white-list) - Access to tape is expensive. To avoid inefficient use of tape resources, or "stage mayhem" by random, chaotic user activity a mechanism exists in dCahe called "stage protection" that allows to control access to data on tape based on user identity (DN), VO group membership and VO role (defined in FQAN), storage group and protocol. Attempts to stage data that does not satisfy criteria of -stage permission configuration will result in permission denied errors +stage permission configuration will result in permission denied errors. + +----- +[TOC bullet hierarchy] +----- ## CONFIGURATION OF STAGE PROTECTION diff --git a/docs/TheBook/src/main/markdown/config-statistics.md b/docs/TheBook/src/main/markdown/config-statistics.md index 1a963ddb016..d8e31988bb3 100644 --- a/docs/TheBook/src/main/markdown/config-statistics.md +++ b/docs/TheBook/src/main/markdown/config-statistics.md @@ -1,19 +1,15 @@ CHAPTER 14. THE STATISTICS SERVICE ================================== -## Table of Contents - -[The Basic Setup](#the-basic-setup) -[The Statistics Web Page](#the-statistics-web-page) -[Explanation of the File Format of the xxx.raw Files](#explanation-of-the-file-format-of-the-xxxraw-files) - - The `statistics` service collects information on the amount of data stored on all pools and the total data flow including streams from and to tertiary storage systems. Once per hour an ASCII file is produced, containing a table with information on the amount of used disk space and the data transferred starting midnight up to this point in time. Data is sorted per pool and storage class. In addition to the hourly statistics, files are produced reporting on the daily, monthly and yearly dCache activities. An HTML tree is produced and updated once per hour allowing to navigate through the collected statistics information. +----- +[TOC bullet hierarchy] +----- ## THE BASIC SETUP diff --git a/docs/TheBook/src/main/markdown/config-write-token.md b/docs/TheBook/src/main/markdown/config-write-token.md index 03ee0a96786..6006218e64a 100644 --- a/docs/TheBook/src/main/markdown/config-write-token.md +++ b/docs/TheBook/src/main/markdown/config-write-token.md @@ -1,12 +1,6 @@ Chapter 21. Using Space Reservations without SRM ================================================= -## Table of Contents - -* [The Space Reservation](#the-space-reservation) -* [The WriteToken tag](#the-writetoken-tag) -* [Copy a File into the WriteToken](#copy-a-file-into-the-writetoken) - If you are using space reservations, i.e. you set ```ini @@ -15,6 +9,10 @@ dcache.enable.space-reservation=true in your configuration file and all of your pools are in [link groups](config-PoolManager.md#link-groups), then you can only write into dCache if a link group is available for your transfer. Using the `SRM` you can specify the link group to write into. If you want to use another protocol like `curl` or `xrootd` you cannot specify a link group. In this case you need to use the `WriteToken` directory tag. +----- +[TOC bullet hierarchy] +----- + ## The Space Reservation Before you can create a `WriteToken` tag you need to have a space reservation. diff --git a/docs/TheBook/src/main/markdown/config-xrootd.md b/docs/TheBook/src/main/markdown/config-xrootd.md index 2a36063b637..a25119906c5 100644 --- a/docs/TheBook/src/main/markdown/config-xrootd.md +++ b/docs/TheBook/src/main/markdown/config-xrootd.md @@ -1,34 +1,12 @@ Chapter 11. dCache as xRootd-Server =================================== -## Table of Contents - -* [Setting up](#setting-up) - - [Parameters](#parameters) - -* [Quick tests](#quick-tests) - - [Copying files with xrdcp](#copying-files-with-xrdcp) - [Accessing files from within ROOT](#accessing-files-from-within-root) - -* [xrootd security](#xrootd-security) - - [Read-Write access](#read-write-access) - [Permitting read/write access on selected directories](#permitting-read/write-access-on-selected-directories) - [Token-based authorization](#token-based-authorization) - [Strong authentication](#strong-authentication) - [Precedence of security mechanisms](#precedence-of-security-mechanisms) - [Other configuration options](#other-configuration-options) - -* [xrootd third-party transfer](#xrootd-third-party-transfer) - - [Changes to dCache configuration for authenticated (GSI) transfers](#changes-to-dcache-configuration-for-authenticated-gsi-transfers) - - [Signed Hash Verification Support](#signed-hash-verification-support) - This chapter explains how to configure dCache in order to access it via the `xrootd` protocol, allowing `xrootd`-Clients like ROOT’s TXNetfile and xrdcp to do file operations against a dCache instance in a transparent manner. dCache implements version 2.1.6 of `xrootd` protocol. +----- +[TOC bullet hierarchy] +----- + ## Setting up To allow file transfers in and out of dCache using xrootd, a new `xrootd door` must be started. This door acts then as the entry point to all xrootd requests. Compared to the native xrootd server-implementation (produced by SLAC), the `xrootd door` corresponds to the `redirector node`. diff --git a/docs/TheBook/src/main/markdown/config-zookeeper.md b/docs/TheBook/src/main/markdown/config-zookeeper.md index ef884753ec9..637336a25ed 100644 --- a/docs/TheBook/src/main/markdown/config-zookeeper.md +++ b/docs/TheBook/src/main/markdown/config-zookeeper.md @@ -3,6 +3,9 @@ ZooKeeper As mentioned earlier, dCache relies on [Apache ZooKeeper](https://zookeeper.apache.org), a distributed directory and coordination service. +----- +[TOC bullet hierarchy] +----- ## Deployment scenarios diff --git a/docs/TheBook/src/main/markdown/cookbook-advanced.md b/docs/TheBook/src/main/markdown/cookbook-advanced.md index 979504cd9a6..840f84c3c99 100644 --- a/docs/TheBook/src/main/markdown/cookbook-advanced.md +++ b/docs/TheBook/src/main/markdown/cookbook-advanced.md @@ -1,24 +1,12 @@ CHAPTER 26. ADVANCED TUNING =========================== -## Table of Contents - -+ [Multiple Queues for Movers in each Pool](#multiple-queues-for-movers-in-each-pool) - - [Description](#description) - [Solution](#solution) - [Configuration](#configuration) - [Tunable Properties for Multiple Queues](#tunable-properties-for-multiple-queues) - -+ [Tunable Properties](#tunable-properties) - - [dCap](#dcap) - [GridFTP](#gridftp) - [SRM](#srm) - - The use cases described in this chapter are only relevant for large-scale dCache instances which require special tuning according to a longer experience with client behaviour. +----- +[TOC bullet hierarchy] +----- + ## MULTIPLE QUEUES FOR MOVERS IN EACH POOL ### Description diff --git a/docs/TheBook/src/main/markdown/cookbook-classic-se-to-dcache.md b/docs/TheBook/src/main/markdown/cookbook-classic-se-to-dcache.md index 2d386f85571..c4600609d89 100644 --- a/docs/TheBook/src/main/markdown/cookbook-classic-se-to-dcache.md +++ b/docs/TheBook/src/main/markdown/cookbook-classic-se-to-dcache.md @@ -4,6 +4,10 @@ Migration of classic SE ( nfs, disk ) to dCache This chapter contains a guide how-to migrate a classic SE to dCache without coping the data. +----- +[TOC bullet hierarchy] +----- + The conversion of a classic SE is straightforward, but has to be done carefully to prevent data losses. We assume, that dCache is [installed](install.md), configured and NFS mounted on the classic SE node. diff --git a/docs/TheBook/src/main/markdown/cookbook-clients.md b/docs/TheBook/src/main/markdown/cookbook-clients.md index b91c838f040..39b3d36e7bd 100644 --- a/docs/TheBook/src/main/markdown/cookbook-clients.md +++ b/docs/TheBook/src/main/markdown/cookbook-clients.md @@ -1,39 +1,11 @@ CHAPTER 22. dCache CLIENTS. ========================== -## Table of Contents - -+ [GSI-FTP](#gsi-ftp) - - [Listing a directory](#listing-a-directory) - [Checking a file exists](#checking-a-file-exists) - [Deleting files](#deleting-files) - [Copying files](#copying-files) - -+ [dCap](#dcap) - - [dccp](#dccp) - [Using the dCache client interposition library.](#using-the-dcache-client-interposition-library.) - -+ [SRM](#srm) - - [Creating a new directory.](#creating-a-new-directory) - [Removing files from dCache](#removing-files-from-dcache) - [Removing empty directories from dCache](#removing-empty-directories-from-dcache) - [srmcp for SRM v1](#srmcp-for-srm-v1) - [srmcp for SRM v2.2](#srmcp-for-srm-v2.2) - -+ [ldap](#ldap) -+ [Using the LCG commands with dCache](#using-the-lcg-commands-with-dcache) - - [The lcg-gt Application](#the-lcg-gt-application) - [The lcg-sd Application](#the-lcg-sd-application) - There are many client tools for dCache. These can most easily be classified by communication protocol. - -As can be seen from above even a single node standard install of dCache returns a considerable number of lines and for this reason we have not included the output, in this case 205 lines where written. - +----- +[TOC bullet hierarchy] +----- ## GSI-FTP diff --git a/docs/TheBook/src/main/markdown/cookbook-dCap.md b/docs/TheBook/src/main/markdown/cookbook-dCap.md index a53c51f2838..e7c9afb5b33 100644 --- a/docs/TheBook/src/main/markdown/cookbook-dCap.md +++ b/docs/TheBook/src/main/markdown/cookbook-dCap.md @@ -1,6 +1,10 @@ Protocols ========= +----- +[TOC bullet hierarchy] +----- + ## DCAP options mover and client options DCAP is the native random access I/O protocol for files within dCache. In additition to the usual data transfer mechanisms, it supports all necessary file metadata and name space manipulation operations. diff --git a/docs/TheBook/src/main/markdown/cookbook-ha-with-replicable-services.md b/docs/TheBook/src/main/markdown/cookbook-ha-with-replicable-services.md index a6940a05d1b..fc95f33ea3e 100644 --- a/docs/TheBook/src/main/markdown/cookbook-ha-with-replicable-services.md +++ b/docs/TheBook/src/main/markdown/cookbook-ha-with-replicable-services.md @@ -22,6 +22,10 @@ low level communication infrastructure used by dCache is redundant. This involves setting up a [ZooKeeper cluster](config-zookeeper.md) of at least three nodes, as well as using a multipath topology for [Cell Message passing](config-message-passing.md). +----- +[TOC bullet hierarchy] +----- + ## Pools Pools contain the data files stored in dCache. If a pool is offline, any file diff --git a/docs/TheBook/src/main/markdown/cookbook-net.md b/docs/TheBook/src/main/markdown/cookbook-net.md index 4d008c3e1ba..f780fb51c74 100644 --- a/docs/TheBook/src/main/markdown/cookbook-net.md +++ b/docs/TheBook/src/main/markdown/cookbook-net.md @@ -1,26 +1,16 @@ Chapter 25. Complex Network Configuration ========================================= -Table of Contents - -+ [Firewall Configuration](#firewall-configuration) - - * [Basic Installation](#basic-installations) - * [Multi-Node with Firewalls](#multi-node-with-firewalls) - -+ [GridFTP Connections via two or more Network Interfaces](#gridftp-connections-cia-two-or-more-network-interfaces) - -+ [GridFTP with Pools in a Private Subnet](#gridftp-with-pools-in-a-private-subnet) - -+ [Using IPv6 with dCache](#using-ipv6-with-dcache) - - This chapter contains solutions for several non-trivial network configurations. The first section discusses the interoperation of dCache with firewalls and does not require any background knowledge about dCache other than what is given in the installation guide ([Chapter 2, Installing dCache](install.md)) and the first steps tutorial ([Chapter 3, Getting to know your dCache](intouch.md). The following sections will deal with more complex network topologies, e.g. private subnets. Even though not every case is covered, these cases might help solve other problems, as well. Intermediate knowledge about dCache is required. > **Warning** > > The TCP and UDP ports used for dCache internal communication (port `11111` by default) *MUST* be subject to firewall control so that only other dCache nodes can access them. Failure to do this will allow an attacker to issue arbitrary commands on any node within your dCache cluster, as whichever user the dCache process runs. +----- +[TOC bullet hierarchy] +----- + ## Firewall Configuration The components of a dCache instance may be distributed over several hosts (nodes). Some of these components are accessed from outside and consequently the firewall needs to be aware of that. We contemplate two communication types, the dCache internal communication and the interaction from dCache with clients. diff --git a/docs/TheBook/src/main/markdown/cookbook-pool.md b/docs/TheBook/src/main/markdown/cookbook-pool.md index cf99ffaf878..931391a9da3 100644 --- a/docs/TheBook/src/main/markdown/cookbook-pool.md +++ b/docs/TheBook/src/main/markdown/cookbook-pool.md @@ -1,22 +1,9 @@ Chapter 23. Pool Operations ========================================== -## Table of Contents - -+ [Checksums](#checksums) - - * [How to configure checksum calculation](#how-to-configure-checksum-calculation) - -+ [Migration Module](#migration-module) - - * [Overview and Terminology](#overview-and-terminology) - * [Command Summary](#command-summary) - * [Examples](#examples) - -+ [Renaming a Pool](#renaming-a-pool) -+ [Pinning Files to a Pool](#pinning-files-to-a-pool) -+ [Running pools with CEPH backends](#running-pools-with-ceph-backends) -+ [Keeping metadata on MongoDB](#keeping-metadata-on-mongodb) +----- +[TOC bullet hierarchy] +----- ## Checksums diff --git a/docs/TheBook/src/main/markdown/cookbook-postgres.md b/docs/TheBook/src/main/markdown/cookbook-postgres.md index 60f46ce58f6..35f02e29e6f 100644 --- a/docs/TheBook/src/main/markdown/cookbook-postgres.md +++ b/docs/TheBook/src/main/markdown/cookbook-postgres.md @@ -1,18 +1,14 @@ Chapter 24. PostgreSQL and dCache ================================= -## Table of Contents - -+ [Installing a PostgreSQL Server](#installing-a-postgresql-Server) -+ [Configuring Access to PostgreSQL](#configuring-access-tp-postgresql) -+ [Performance of the PostgreSQL Server](#performance-of-the-postgresql-server) - - - PostgreSQL is used for various things in a dCache system: The [SRM](rf-glossary.md#storage-resource-manager-srm), the [pin manager](rf-glossary.md#pin-manager), the [space manager](rf-glossary.md#space-manager), the [replica manager](rf-glossary.md#replica-manager), the [billing](rf-glossary.md#billing), and the pnfs server might make use of one or more databases in a single or several separate PostgreSQL servers. The `SRM`, the pin manager, the space manager and the replica manager will use the PostgreSQL database as configured at cell start-up in the corresponding batch files. The `billing` will only write the accounting information into a database if it is configured with the option `-useSQL`. The `pnfs` server will use a PostgreSQL server if the `pnfs-posgresql` version is used. It will use several databases in the PostgreSQL server. +----- +[TOC bullet hierarchy] +----- + ## Installing a PSQL Server The preferred way to set up a PSQL server should be the installation of the version provided by your OS distribution; however, version 8.3 or later is required. Version 9.5 is recommended, as it has a more efficient locking mechanism that improves performance. diff --git a/docs/TheBook/src/main/markdown/cookbook-transport-security.md b/docs/TheBook/src/main/markdown/cookbook-transport-security.md index 0b6e048756c..043b91aa1f0 100644 --- a/docs/TheBook/src/main/markdown/cookbook-transport-security.md +++ b/docs/TheBook/src/main/markdown/cookbook-transport-security.md @@ -14,7 +14,9 @@ There is the option `webdav.redirect.allow-https` which is `false` by default. I Please note, that IT security is an ever changing field. Make sure you follow the latest developments. If security in dCache is a priority for you, be sure to follow the dCache user mailinglist. - +----- +[TOC bullet hierarchy] +----- ## Configuring a secure WebDAV door diff --git a/docs/TheBook/src/main/markdown/cookbook-writing-hsm-plugins.md b/docs/TheBook/src/main/markdown/cookbook-writing-hsm-plugins.md index 7062a5aa6b3..175f6d292ed 100644 --- a/docs/TheBook/src/main/markdown/cookbook-writing-hsm-plugins.md +++ b/docs/TheBook/src/main/markdown/cookbook-writing-hsm-plugins.md @@ -16,6 +16,10 @@ recalled at a time. dCache has a third party accessible plugin system. One of the pluggable components is the nearline storage interface. +----- +[TOC bullet hierarchy] +----- + ## Nearline Requests A nearline storage drive must support three types of requests: flush to the diff --git a/docs/TheBook/src/main/markdown/intouch.md b/docs/TheBook/src/main/markdown/intouch.md index ab50318b249..5eeb44e7878 100644 --- a/docs/TheBook/src/main/markdown/intouch.md +++ b/docs/TheBook/src/main/markdown/intouch.md @@ -1,38 +1,11 @@ Chapter 3. Getting to know dCache ================================= -## Table of Contents - -* [Checking the Functionality](#checking-the-functionality) - - - [dCache without mounted namespace](#dcache-without-mounted-namespace) - - [WebDAV](#webdav) - - [dCap](#dcap) - - -* [The Web Interface for Monitoring dCache](#the-web-interface-for-monitoring-dcache) -* [The Admin Interface](#the-admin-interface) - - - [First steps](#first-steps) - - [Access with ssh](#access-with-ssh2) - - [How to use the Admin Interface](#how-to-use-the-admin-interface) - - [Create a new user](#create-a-new-user) - - [Use of the ssh Admin Interface by scripts](#use-of-the-ssh-admin-interface-by-scripts) - -* [Authentication and Authorization in dCache](#authentication-and-authorization-in-dcache) -* [How to work with secured dCache](#how-to-work-with-secured-dCache) - - - [GSIdCap](#gsidcap) - - [SRM](#srm) - - [WebDAV with certificates](#webdav-with-certificates) - - * [Files](#files) - - - This section is a guide for exploring a newly installed dCache system. The confidence obtained by this exploration will prove very helpful when encountering problems in the running system. This forms the basis for the more detailed stuff in the later parts of this book. The starting point is a fresh installation according to [the section called “Installing a dCache instance”.](install.md#installing-a-dcache-instance) - +----- +[TOC bullet hierarchy] +----- ## CHECKING THE FUNCTIONALITY diff --git a/docs/TheBook/src/main/markdown/intro.md b/docs/TheBook/src/main/markdown/intro.md index 6bbb9ba5021..16449ae15b5 100644 --- a/docs/TheBook/src/main/markdown/intro.md +++ b/docs/TheBook/src/main/markdown/intro.md @@ -1,13 +1,9 @@ Chapter 1. Introduction ======================= -## Table of Contents - -* [dCache Architecture](#architecture) -* [Cells and Domains](#cells-and-domains) -* [Cells Communication](config-message-passing.md) -* [Protocols Supported by dCache](#protocols-supported-by-dcache) -* [Logging](#logging) +----- +[TOC bullet hierarchy] +----- dCache is a distributed storage system providing location independent access to data. The data are stored across multiple data servers as complete files presented to end-user via a single rooted namespace. diff --git a/docs/TheBook/src/main/markdown/kafkaproducer.md b/docs/TheBook/src/main/markdown/kafkaproducer.md index a92e2234d96..0d7c2d72b9b 100644 --- a/docs/TheBook/src/main/markdown/kafkaproducer.md +++ b/docs/TheBook/src/main/markdown/kafkaproducer.md @@ -2,12 +2,11 @@ Apache-Kafka with to dCache, or how to push events outside of dCache. ======================= - - - Starting from version 4.1 we introduced a new approach to external messaging. With the introduction of Apache-Kafka (kafka.apache.org/documentation/) as a message transport, for now in Billing, we start switching to a more modern and efficient messaging system. The goal for the Billing service is easier inclusion with systems like Elasticsearch: Instead of adding messages to a textual log and having an external component parse that log, a Kafka-aware ingester can import them directly into the remote system. - +----- +[TOC bullet hierarchy] +----- ## What do you need to download to enable Kafka in dCache? diff --git a/docs/TheBook/src/main/markdown/preface.md b/docs/TheBook/src/main/markdown/preface.md index 67f912142bf..94ce570b52e 100644 --- a/docs/TheBook/src/main/markdown/preface.md +++ b/docs/TheBook/src/main/markdown/preface.md @@ -1,11 +1,6 @@ Preface ======= -## Table of Contents - -[Minimum System Requirements](#minimum-system-requirements) -[Structure of this book](#structure-of-this-book) - ## Welcome to dCache. dCache is a distributed storage system proven to scale to hundreds of Petabytes.