diff --git a/docs/cvmfs.md b/docs/cvmfs.md new file mode 100644 index 0000000..8fcfe8a --- /dev/null +++ b/docs/cvmfs.md @@ -0,0 +1,41 @@ +# The ESCAPE CVMFS repository + +The [CERN Virtual Machine File System](https://cernvm.cern.ch/fs/) (CVMFS) is a read-only, HTTP-based file system optimized for distributing scientific software at scale. +Originally developed at CERN for the Worldwide LHC Computing Grid (WLCG), CVMFS is now widely adopted across many research infrastructures.
+Among other capabilities, CVMFS provides the following functionalities: + +- **Global software delivery system**: Software is published once and instantly becomes available across all connected clients and compute sites. + +- **HTTP and caching-based**: CVMFS fetches files over standard HTTP and relies on [caching proxies](https://www.squid-cache.org/) to reduce load and latency. + +- **FUSE-mounted**: Software appears as a regular local file system on the client machine. + +Find more at [the official CVMFS documentation](https://cvmfs.readthedocs.io/en/stable/). +### Components +The ESCAPE CVMFS infrastructure consists of three main components: +1. The single source, called the Stratum 0 Repository Server +2. Public mirrors, called Stratum 1 Replica Servers +3. Caches, usually managed by Squid Proxy servers + +One protected read/write Stratum 0 instance, called `sw.escape.eu`, is feeding up the public, distributed mirror servers.
+Then, a distributed hierarchy of proxy servers fetches content from the closest public mirror server. + +Find more details about `sw.escape.eu` and how to setup CVMFS in [the developer documentation](tech-docs/services/cvmfs). + +A diagram of the infrastructure described is shown below. + +![image](../static/img/CVMFS-diagram.png) + +## CVMFS the ESCAPE VRE + +In the ESCAPE VRE, we use CVMFS to: + +- Distribute pre-built scientific software and tools in a consistent, portable way across cloud, HPC, and local environments. + +- Package domain-specific applications, such as Rucio clients or analysis environments, as tarballs that are extracted into CVMFS repositories. + +- Ensure reproducibility by version-controlling software environments and decoupling runtime dependencies from local installations. + +- Facilitate onboarding of new communities by removing the burden of local software setup and environment configuration. + +This CVMFS-based software delivery system allows ESCAPE to scale up access to research tools, reduce setup time, and ensure consistency across diverse computing resources. \ No newline at end of file diff --git a/docs/tech-docs/services/cvmfs.md b/docs/tech-docs/services/cvmfs.md new file mode 100644 index 0000000..70b0143 --- /dev/null +++ b/docs/tech-docs/services/cvmfs.md @@ -0,0 +1,68 @@ +# CVMFS +:::warning +These instructions are highly CERN-dependent.
+Please refer to the [Official CernVM-FS documentation](https://cvmfs.readthedocs.io/en/stable) for detailed instructions on how to start from scratch. +::: +## Setting up a Repository + +At CERN, the repository and release manager creation are provided by the Storage and Data Management group, through the [CVMS Repository creation request](https://cern.service-now.com/service-portal?id=sc_cat_item&name=CVMS-repository&se=cvmfs). + +Elsewhere, please follow the steps at [Creating a Repository (Stratum 0)](https://cvmfs.readthedocs.io/en/stable/cpt-repo.html). + +A set of useful information to know before submitting the request (or autonomously creating a repository): +1. What is the quota for the repository and expected growth of the over time? +2. Does the repository need (periodical) garbage collection? +3. Should the visibility of the repository be restricted (e.g., due to licensed software)? +4. Should the repository be replicated on other Stratum 1 servers worldwide? + +:::tip[Repository naming] +*The repository name resembles a DNS scheme, but it does not need to reflect any real server name. It is supposed to be a globally unique name that indicates where/who the publishing of content takes place. A repository name must only contain alphanumeric characters plus -, , or ., and it is limited to a length of 60 characters.* +::: + +The release manager is usually a VM where operators can upload, modify and delete the content of the repository. + +## Publishing content + +As described [here](https://cvmfs.readthedocs.io/en/stable/cpt-repo.html#content-publishing), there are four main steps needed to publish content: +1. Initiate the transation via `cvmfs_server transaction ` +2. Install content into `/cvmfs/` +3. (optional) Create nested catalogs at proper locations +4. Finalise the transaction via `cvmfs_server publish ` + +#### Install content + +At startup, a CERN-hosted release manager will look like this: +```sh +* ******************************************************************** +* Welcome to xxxxx.cern.ch, AlmaLinux release 9.5 (Teal Serval) +* Archive of news is available in /etc/motd-archive +* Reminder: you have agreed to the CERN +* computing rules, in particular OC5. CERN implements +* the measures necessary to ensure compliance. +* https://cern.ch/ComputingRules +* Puppet environment: production, Roger state: build +* Foreman hostgroup: xxxxxxx +* ####################################################### +The CVMFS Stratum 0 for repo xxxxxx. +Access is controlled via the e-group xxxxxxx. +Shared local unix user for account xxxxxxx. + +To become the shared user execute: + sudo -i -u xxxxxxx +To start a transaction: + cvmfs_server transaction sw.escape.eu +To publish a transaction + cvmfs_server publish sw.escape.eu + +* A cvmfs installation host. +* ******************************************************************** +``` + +In order to publish content, it is necessary to switch to a dedicated user, as suggested: `sudo -i -u xxxxxxx`. + +How to provide packages and proper versioning is left to the operators. + +:::danger[Pro Tip] +The VRE strategy for content publishing relies on the creation of a tarball containing the necessary software and versions, followed by a setup script to be executed at need.
+More details can be found in the **[ESCAPE CVMFS GitHub Repository](https://github.com/vre-hub/escape-cvmfs)**. +::: \ No newline at end of file diff --git a/docusaurus.config.js b/docusaurus.config.js index 4888891..20b06ce 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -101,8 +101,13 @@ const config = { }, { type: 'doc', // This is a preset regarding the type of document, please update if needed - docId: 'rucio', // The ID of the corresponding document. This should match with what you wrote in the document header. - label: 'Rucio', // The title that will appear on the menu + docId: 'cvmfs', // The ID of the corresponding document. This should match with what you wrote in the document header. + label: 'CVMFS', // The title that will appear on the menu + }, + { + type: 'doc', // This is a preset regarding the type of document, please update if needed + docId: 'notebook', // The ID of the corresponding document. This should match with what you wrote in the document header. + label: 'JupyterHub', // The title that will appear on the menu }, { type: 'doc', // This is a preset regarding the type of document, please update if needed @@ -111,8 +116,8 @@ const config = { }, { type: 'doc', // This is a preset regarding the type of document, please update if needed - docId: 'notebook', // The ID of the corresponding document. This should match with what you wrote in the document header. - label: 'JupyterHub', // The title that will appear on the menu + docId: 'rucio', // The ID of the corresponding document. This should match with what you wrote in the document header. + label: 'Rucio', // The title that will appear on the menu }, ], }, diff --git a/sidebars.js b/sidebars.js index a92dea0..234cbd5 100644 --- a/sidebars.js +++ b/sidebars.js @@ -159,6 +159,7 @@ const sidebars = { label: 'Services', items: [ 'tech-docs/services/aai', + 'tech-docs/services/cvmfs', 'tech-docs/services/data-management', 'tech-docs/services/jupyterhub', 'tech-docs/services/computing-resources', diff --git a/src/pages/LatestNews.jsx b/src/pages/LatestNews.jsx index 3d26465..b02b6f7 100644 --- a/src/pages/LatestNews.jsx +++ b/src/pages/LatestNews.jsx @@ -48,7 +48,7 @@ const newsItems = [ title: 'The Virtual Research Environment: A multi-science analysis platform', authors: 'E. Gazzarrini, E. Garcia Garcia, D. Gosein, and X. Espinal', journal: 'EPJ Web of Conferences 295, 08023 (2024)', - links: [{ text: 'CHEP 2023 proceedings', url: 'https://doi.org/10.1051/epjconf/202429508023/' }], + links: [{ text: 'CHEP 2023 proceedings', url: 'https://doi.org/10.1051/epjconf/202429508023' }], iconType: 'book-open', type: 'proceedings', }, diff --git a/src/pages/index.mdx b/src/pages/index.mdx index 31a6677..cf79bb2 100644 --- a/src/pages/index.mdx +++ b/src/pages/index.mdx @@ -22,9 +22,10 @@ The Virtual Research Environment’s main components are: 1. **[AAI](/docs/auth)**: A federated and reliable **Authentication and Authorization** layer -2. **[The Rucio Data Lake](/docs/rucio)**: A **federated distributed storage** solution (the ESCAPE Data Lake), providing functionalities for data injection and replication through a Data Management framework (Rucio) +2. **[The Rucio Data Lake](/docs/rucio)**: A **federated distributed storage** solution, providing functionalities for data injection and replication through a Data Management framework (Rucio) 3. **[Reana](/docs/reana)**: A **computing** cluster supplying the processing power to run full analyses with Reana, a re-analysis software -4. **[JupyterHub](/docs/notebook)**: An enhanced **notebook interface** with containerised environments to hide the infrastructure’s complexity from the user. +4. **[CVMFS](/docs/cvmfs)**: A read-only file system designed to distribute software, and more. +5. **[JupyterHub](/docs/notebook)**: An enhanced **notebook interface** with containerised environments to hide the infrastructure’s complexity from the user. ![image](../../static/img/VRE-diagram.png) diff --git a/static/img/CVMFS-diagram.png b/static/img/CVMFS-diagram.png new file mode 100644 index 0000000..061029c Binary files /dev/null and b/static/img/CVMFS-diagram.png differ