diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..d8fe4fa --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +/.project diff --git a/docs/appendix/index.rst b/docs/appendix/index.rst deleted file mode 100644 index fd80c3d..0000000 --- a/docs/appendix/index.rst +++ /dev/null @@ -1,10 +0,0 @@ -======== -Appendix -======== - -This appendix contains additional pages of information about the LOCKSS system. - -.. toctree:: - :hidden: - - ports diff --git a/docs/appendix/ports.rst b/docs/appendix/ports.rst deleted file mode 100644 index ab412aa..0000000 --- a/docs/appendix/ports.rst +++ /dev/null @@ -1,33 +0,0 @@ -============= -Network Ports -============= - -This page describes the default network ports used by the LOCKSS system. - -Unless otherwise noted, all ports are **TCP**. - -All ports in the 24600-24699 range should be considered reserved. The LCAP (LOCKSS polling and repair) port retains its historical value of 9729. - -* 9729: LCAP (LOCKSS polling and repair) -* 24600: *reserved* (currently LOCKSS Configuration Service UI) -* 24602: PostgreSQL -* 24603: Solr -* 24606: ActiveMQ -* 24610: LOCKSS Repository Service - REST port -* 24619: *reserved* (HDFS FS port) -* 24620: LOCKSS Configuration Service - Rest port -* 24621: LOCKSS Configuration Service - UI port -* 24630: LOCKSS Poller Service - REST port -* 24631: LOCKSS Poller Service - UI port -* 24640: LOCKSS Metadata Extraction Service - REST port -* 24641: LOCKSS Metadata Extraction Service - UI port -* 24650: LOCKSS Metadata Service - REST port -* 24651: LOCKSS Metadata Service - UI port -* 24670: LOCKSS Proxy -* 24671: *reserved* -* 24672: LOCKSS Audit Proxy -* 24673: *reserved* -* 24674: ICP server **(UDP)** -* 24680: LOCKSS Content Server (ServeContent) -* 24681: Pywb replay engine -* 24682: *reserved* OpenWayback replay engine diff --git a/docs/conf.py b/docs/conf.py index 9b23b84..722b37a 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -33,6 +33,7 @@ extensions = [ # See https://github.com/readthedocs/sphinx_rtd_theme 'sphinx_rtd_theme', + 'sphinx.ext.autosectionlabel', 'sphinx.ext.intersphinx' ] diff --git a/docs/configuring.rst b/docs/configuring.rst deleted file mode 100644 index 75c6b9b..0000000 --- a/docs/configuring.rst +++ /dev/null @@ -1,65 +0,0 @@ -============================= -Configuring the LOCKSS System -============================= - -After :doc:`installing/index` and :doc:`installing/lockss-installer`, configure the system with the configure script: - -.. code-block:: shell - - scripts/configure-lockss - -(If you have experience with classic LOCKSS daemon version 1.x, this is the equivalent of ``hostconfig``.) - -The questions asked by the script often come with a suggested value, displayed in square brackets; hit :kbd:`Enter` to accept the suggested value, or type the correct value and hit Enter. Questions include: - -1. ``Fully qualified hostname (FQDN) of this machine:`` Enter the machine's hostname (e.g. ``locksstest.myuniversity.edu``). - -2. ``IP address of this machine:`` The publicly routable IP address of the machine, or if it is not publicly routable but will be accessible via network address translation (NAT), its IP address on the internal network. - -3. ``Is this machine behind NAT?:`` Enter :kbd:`Y` if the machine is not publicly routable but will be accessible via network address translation (NAT), or :kbd:`N` otherwise. - - 1. ``External IP address for NAT:`` If you answered :kbd:`Y` to the previous question, enter the publicly routable IP address of the machine. - -4. ``Initial subnet for admin UI access:`` Enter a semicolon-separated list of subnets in CIDR notation that should have access to the Web user interfaces of the system. - -5. ``LCAP V3 protocol port:`` Enter the port on the publicly routable IP address that will be used to receive LCAP (LOCKSS polling and repair) traffic. Historically, most LOCKSS nodes use 9729. - -6. ``PROXY port:`` Not yet re-enabled in 2.0-alpha; ignore. - -7. ``Mail relay for this machine:`` Hostname for this machine’s outgoing mail server. - -9. ``Does mail relay need user & password``: Enter :kbd:`Y` if the outgoing mail server requires password authentication, :kbd:`N` otherwise. - - 1. ``User for :`` If you answered :kbd:`Y` to the outgoing mail server password authentication question, enter the username for the mail server. - - 2. ``Password for @:`` Enter the password for the given username. - - 3. ``Again:`` Re-enter the mail server password (if the two passwords do not match, the password will be asked again). - -9. ``E-mail address for administrator:`` Enter the e-mail address of the person or team who will administer the LOCKSS system on this machine. - -10. ``Configuration URL:`` Enter the URL of the LOCKSS network configuration file. If you are not running your own LOCKSS network, use ``http://props.lockss.org:8001/demo/lockss.xml``, the configuration file for a demo network set up for LOCKSS 2.0 pre-release testing. - -11. ``Configuration proxy (host:port):`` enter a host:port combination for the proxy server needed to reach the network configuration file (or simply hit Enter if none is needed). - -12. ``Preservation group(s):`` Enter a semicolon-separated list of preservation network identifiers. If you are not joining an existing network or running your own, enter demo, the network identifier for the demo network set up for LOCKSS 2.0 pre-release testing. - -13. ``Content data storage directory:`` Enter the path of a directory that is the root of the main storage area of the LOCKSS system. If you are used to the classic LOCKSS daemon (1.x), this would be the equivalent of :file:`/cache0`. - -14. ``Service logs directory:`` Enter the path of a directory that is the root of the storage area for LOCKSS-related log files (historically :file:`/var/log/lockss`). - -15. ``Temporary storage directory:`` not actively used in LOCKSS 2.0-alpha2; ignore. - -16. ``User name for web UI administration:`` Enter the username for an administrative user in the LOCKSS system’s Web user interfaces. - -17. ``Password for web UI administration user :`` Enter the password for the given administrative user in the LOCKSS system’s Web user interfaces. - -18. ``Password for web UI administration user (again):`` Re-enter the password for the given administrative user in the LOCKSS system’s Web user interfaces (if the two passwords do not match, the password will be asked again). - -19. ``Password for database:`` Enter the password for the embedded Postgres database included in LOCKSS 2.0-alpha. *Future versions will allow you to use an existing Postgres database and enter credentials accordingly.* - -20. ``Password for database (again):`` Re-enter the password for the embedded Postgres database (if the two passwords do not match, the password will be asked again). - -21. ``OK to store this configuration:`` confirm with :kbd:`Y` that the summarized configuration data is correct and that you are ready to write it to a file. - -If prompted to generate files, accept (or run :file:`scripts/generate-lockss` immediately after). diff --git a/docs/images/lockss-2.0-alpha1_200.png b/docs/images/lockss-2.0-alpha1_200.png deleted file mode 100644 index 5493926..0000000 Binary files a/docs/images/lockss-2.0-alpha1_200.png and /dev/null differ diff --git a/docs/images/lockss-2.0-alpha1_800.png b/docs/images/lockss-2.0-alpha1_800.png deleted file mode 100644 index 440f92f..0000000 Binary files a/docs/images/lockss-2.0-alpha1_800.png and /dev/null differ diff --git a/docs/index.rst b/docs/index.rst index cf2877b..5b6c002 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,25 +1,266 @@ -.. image:: /images/lockss-2.0-alpha1_200.png - :alt: LOCKSS 2.0-alpha1 logo - :align: right +=================================== +LOCKSS 2.0-alpha Technology Preview +=================================== -==================== -LOCKSS System Manual -==================== +*Released: 2019-04-05. Last modified: 2021-04-12.* -**Welcome to the LOCKSS 2.0-alpha1 System Manual.** +At the `2019 LOCKSS Alliance Meeting `_, the LOCKSS engineering team presented a Technology Preview of the upcoming LOCKSS 2.0-alpha1 release. **The LOCKSS 2.0-alpha Technology Preview and the upcoming LOCKSS 2.0-alpha1 release are for evaluation and testing purposes.** -Use the popup at the bottom of the menu to navigate to other versions and download offline copies of the manual. +The LOCKSS 2.0-alpha1 release will be the first public release of the next generation of LOCKSS software. It represents a major evolution of the classic `LOCKSS daemon `_ (1.x) into a suite of specialized software components. These components will interoperate via REST APIs and run as Docker components as part of a Docker stack. + +Unlike the production LOCKSS 2.0 release, the LOCKSS 2.0-alpha Technology Preview consists of a fixed set of LOCKSS components, and the supporting components are provided and managed by the LOCKSS system: + +* Provided Postgres database container + +* Provided Solr database container + +* Provided HDFS (Hadoop File System) container + +* LOCKSS Repository Service + +* LOCKSS Configuration Service + +* LOCKSS Metadata Extraction Service + +* LOCKSS Metadata Service + +* LOCKSS Poller Service + +* Provided Pywb container + +------------------------ +Overview and Quick Start +------------------------ + +To run the LOCKSS 2.0-alpha Technology Preview, you will need a Linux host with Docker running in Swarm mode with the Local-Persist Docker volume plugin installed. + +You will also need to install a couple of Python modules and download a small project from GitHub. + +Quick Start: + +* Install `Docker `_ (18.09 or better, in Swarm mode), the :ref:`Local-Persist` Docker volume plugin, the setuptools and :ref:`Pystache` Python modules, and :ref:`Git`. + +* ``git clone --branch develop https://github.com/lockss/lockss-installer`` + +* ``cd lockss-installer`` + +* ``scripts/configure-lockss`` + +* ``scripts/assemble-lockss`` + +* ``scripts/deploy-lockss`` + +* Use the system + +* Shut down the system with ``script/shutdown-lockss`` + +--------------------- +System Pre-Requisites +--------------------- + +Machine and operating system: + +* 64-bit Linux flavor running ``systemd`` + +* 4 cores (8 or more recommended) + +* 8GB of memory (16GB or more recommended) + +Disk space: + +* A few gigabytes of storage space to process a dozen archival units as part of the provided demo network (a few LOCKSS plugins for open access journals) + +* Proportionately more storage space if you process more archival units + +----------------------- +Software Pre-Requisites +----------------------- + +The software pre-requisites are: + +* `Docker `_ (18.09 or better) + +* :ref:`Local-Persist` (see below) + +* :ref:`Pystache` (see below) + +* :ref:`Git` (see below) + +Local-Persist +============= + +`Local-Persist `_ is a plugin for Docker. + +* *(recommended)* Direct installation with Curl: ``curl -fsSL https://raw.githubusercontent.com/CWSpear/local-persist/master/scripts/install.sh | sudo bash`` + +* Direct installation with Wget: ``wget -qO - https://raw.githubusercontent.com/CWSpear/local-persist/master/scripts/install.sh | sudo bash`` + +* Manual installation: https://github.com/CWSpear/local-persist#manual-way + +To verify that the Local-Persist Docker plugin is working, run ``docker info`` and check that the ``Volume`` category under ``Plugins`` contains ``local-persist``. + +Pystache +======== + +`Pystache `_ is a Python implementation of the `Mustache `_ templating language. + +* Via pip: ``sudo pip install pystache`` + +* Via a system package: + + * Arch Linux: ``python-pystache`` from the AUR (https://aur.archlinux.org/packages/python-pystache/) + + * CentOS 7: ``sudo yum install pystache`` + + * Debian 9 (Stretch) or better: ``sudo apt-get install python-pystache`` + + * Fedora 28 or better: ``sudo yum install python2-pystache`` + + * Ubuntu 16.04 LTS (Xenial) or better: ``sudo apt-get install python-pystache`` + +To verify that Pystache is installed, run ``pystache --help`` and check that you get a help message rather than an error message. Alternatively, run ``pystache-test`` to run Pystache's internal test suite. + +Git +=== + +`Git `_ is a version control system. + +If Git is not installed already (run ``git --help`` and see if you get a help message or an error message), it can be installed via a system package: + +* Arch Linux: ``sudo pacman -S git`` + +* CentOS 7: ``sudo apt-get install git`` + +* Debian 9 (Stretch) or better: ``sudo yum install git`` + +* Fedora 28 or better: ``sudo yum install git`` + +* Ubuntu 16.04 LTS (Xenial) or better: ``sudo apt-get install git`` + +------------------------------- +Installing the LOCKSS Installer +------------------------------- + +Clone the ``develop`` branch of the ``lockss-installer`` GitHub project: + +.. code-block:: shell + + git clone --branch develop https://github.com/lockss/lockss-installer + +and change into the newly created directory: + +.. code-block:: shell + + cd lockss-installer + +In this document, this directory is referred to as ``${INSTALLER_HOME}``. + +---------------------- +Configuring the System +---------------------- + +To configure the system in ``${INSTALLER_HOME}``, run: + +.. code-block:: shell + + scripts/configure-lockss + +The gist of this script will be familiar to users of the classic LOCKSS daemon (1.x) installation script ``hostconfig``. + +The questions asked by the script often come with a suggested value, displayed in square brackets; hit Enter to accept the suggested value, or type the correct value and hit Enter. Questions include: + +1. ``Fully qualified hostname (FQDN) of this machine:`` Enter the machine's hostname (e.g. ``locksstest.myuniversity.edu``). + +2. ``IP address of this machine:`` The publicly routable IP address of the machine, or if it is not publicly routable but will be accessible via network address translation (NAT), its IP address on the internal network. + +3. ``Is this machine behind NAT?:`` Enter :kbd:`Y` if the machine is not publicly routable but will be accessible via network address translation (NAT), or :kbd:`N` otherwise. + + 1. ``External IP address for NAT:`` If you answered :kbd:`Y` to the previous question, enter the publicly routable IP address of the machine. + +4. ``Initial subnet for admin UI access:`` Enter a semicolon-separated list of subnets in CIDR notation that should have access to the Web user interfaces of the system. + +5. ``LCAP V3 protocol port:`` Enter the port on the publicly routable IP address that will be used to receive LCAP (LOCKSS polling and repair) traffic. Historically, most LOCKSS nodes use 9729. + +6. ``PROXY port:`` Not yet re-enabled in 2.0-alpha; ignore. + +7. ``Mail relay for this machine:`` Hostname for this machine’s outgoing mail server. + +8. ``Does mail relay need user & password``: Enter :kbd:`Y` if the outgoing mail server requires password authentication, :kbd:`N` otherwise. + + 1. ``User for :`` If you answered :kbd:`Y` to the outgoing mail server password authentication question, enter the username for the mail server. + + 2. ``Password for @:`` Enter the password for the given username. + + 3. ``Again:`` Re-enter the mail server password (if the two passwords do not match, the password will be asked again). + +9. ``E-mail address for administrator:`` Enter the e-mail address of the person or team who will administer the LOCKSS system on this machine. + +10. ``Configuration URL:`` Enter the URL of the LOCKSS network configuration file. If you are not running your own LOCKSS network, use ``http://props.lockss.org:8001/demo/lockss.xml``, the configuration file for a demo network set up for LOCKSS 2.0 pre-release testing. + +11. ``Configuration proxy (host:port):`` enter a host:port combination for the proxy server needed to reach the network configuration file (or simply hit Enter if none is needed). + +12. ``Preservation group(s):`` Enter a semicolon-separated list of preservation network identifiers. If you are not joining an existing network or running your own, enter demo, the network identifier for the demo network set up for LOCKSS 2.0 pre-release testing. + +13. ``Content data storage directory:`` Enter the path of a directory that is the root of the main storage area of the LOCKSS system. If you are used to the classic LOCKSS daemon (1.x), this would be the equivalent of :file:`/cache0`. + +14. ``Service logs directory:`` Enter the path of a directory that is the root of the storage area for LOCKSS-related log files (historically :file:`/var/log/lockss`). + +15. ``Temporary storage directory:`` not actively used in LOCKSS 2.0-alpha; ignore. + +16. ``User name for web UI administration:`` Enter the username for an administrative user in the LOCKSS system’s Web user interfaces. + +17. ``Password for web UI administration user :`` Enter the password for the given administrative user in the LOCKSS system’s Web user interfaces. + +18. ``Password for web UI administration user (again):`` Re-enter the password for the given administrative user in the LOCKSS system’s Web user interfaces (if the two passwords do not match, the password will be asked again). + +19. ``Password for database:`` Enter the password for the embedded Postgres database included in LOCKSS 2.0-alpha. *Future versions will allow you to use an existing Postgres database and enter credentials accordingly.* + +20. ``Password for database (again):`` Re-enter the password for the embedded Postgres database (if the two passwords do not match, the password will be asked again). + +21. ``OK to store this configuration:`` confirm with :kbd:`Y` that the summarized configuration data is correct and that you are ready to write it to a file. + +If prompted to generate files, accept (or run ``scripts/generate-lockss`` immediately after). + +--------------------------- +Preparing to Run the System +--------------------------- + +Run the following script from ``${INSTALLER_HOME}``: + +.. code-block:: shell + + scripts/assemble-lockss + +This script sets up Docker-related infrastructure, configuration files and configuration data. + +------------------ +Running the System +------------------ + +Run the following script from ``${INSTALLER_HOME}``: + +.. code-block:: shell + + script/deploy-lockss + +This has the effect of creating a Docker stack (an orchestrated group of Docker services) by calling ``docker stack create ... lockss-stack1`` with a generated Docker Compose file parameterized appropriately and flanked by necessary infrastructure (Docker configs, Docker secrets, Docker volumes, etc.) + +------------------------ +Shutting Down the System +------------------------ + +Run the following script from ``${INSTALLER_HOME}``: + +.. code-block:: shell + + script/shutdown-lockss + +This has the effect of calling ``docker stack rm lockss-stack1``. Note that it takes a moment for all service containers to properly shut down. .. toctree:: - :caption: LOCKSS System Manual - :maxdepth: 1 - :numbered: + :hidden: - installing/index - configuring - running - using/index - appendix/index + self .. toctree:: :caption: LOCKSS Documentation Portal diff --git a/docs/installing/docker.rst b/docs/installing/docker.rst deleted file mode 100644 index 16c96ca..0000000 --- a/docs/installing/docker.rst +++ /dev/null @@ -1,260 +0,0 @@ -================= -Installing Docker -================= - -`Docker `_ is a container runtime and orchestration engine. This page will walk you through the initial installation of the Docker engine and the `Local-Persist `_ Docker volume plugin. - --------- -Overview --------- - -To install Docker for the purposes of running the LOCKSS system: - -* Install Docker -* Check the System Group -* Start Docker -* Enable Docker at Startup -* Check the Storage Driver -* Initialize Swarm Mode -* Install Local-Persist -* Reconfiguring Docker (if required along the way) - -------------------------- -Install the Docker Engine -------------------------- - -The LOCKSS system requires **Docker 18.09 or better**. - -Docker on Arch Linux -==================== - -Simply use Arch’s official software repositories to install Docker with Pacman: - -.. code-block:: shell - - sudo pacman -S docker - -Docker on CentOS -================ - -.. caution:: - - CentOS 7 is required. - -Use Docker’s official software repositories to install Docker with Yum: https://docs.docker.com/install/linux/docker-ce/centos/ (the version of Docker available through the standard CentOS software repositories is not suitably recent). - -Docker on Debian -================ - -.. caution:: - - Debian 9 (Stretch) is required. - -Use Docker’s official software repositories to install Docker with Apt: https://docs.docker.com/install/linux/docker-ce/debian/ (the docker package available through the standard Debian software repositories is for an unrelated system tray application). - -Docker on Fedora -================ - -.. caution:: - - Fedora 28 or better is required. - -Use Docker’s official software repositories to install Docker with Dnf: https://docs.docker.com/install/linux/docker-ce/fedora/ (the version of Docker available through the standard Fedora software repositories is not suitably recent). - -Docker on Oracle Linux -====================== - -.. caution:: - - Oracle Linux 7 is required. - -Use Oracle’s official software repositories to install Docker with Yum: https://docs.oracle.com/cd/E52668_01/E87205/html/section_install_upgrade_yum_docker.html. - -Docker on Ubuntu -================ - -.. caution:: - - Ubuntu 16.04 LTS (Xenial) or better is required. - -Use Docker’s official software repositories to install Docker with Apt: https://docs.docker.com/install/linux/docker-ce/ubuntu/ (the docker package available via the Ubuntu Universe software repository is for an unrelated desktop system tray application). - ----------------------- -Check the System Group ----------------------- - -Installing Docker creates a new system group typically named ``docker``. The command: - -.. code-block:: shell - - groups lockss - -will display the list of groups the ``lockss`` user is a member of, which should already include the ``lockss`` group. If the list of groups does not include the ``docker`` group, add the ``lockss`` user to the ``docker`` group with the following command: - -.. code-block:: shell - - sudo usermod -G docker -a lockss - -or with a similar user management command or tool. - ------------- -Start Docker ------------- - -Start Docker with ``systemd``: - -.. code-block:: shell - - sudo systemctl start docker - -Verify that Docker is running: - -.. code-block:: shell - - sudo systemctl is-active docker - -The output should say ``active``. - ------------------------- -Enable Docker at Startup ------------------------- - -Unless you are only trying out the LOCKSS system on a machine that will not be running it or Docker routinely, enable Docker to launch at startup with ``systemd``: - -.. code-block:: shell - - sudo systemctl enable docker - -Verify that the operation succeeded with: - -.. code-block:: shell - - sudo systemctl is-enabled docker - -The output should say ``enabled``. - ------------------------- -Check the Storage Driver ------------------------- - -Verify that Docker is using the OverlayFS (``overlay2``) storage driver: - -.. code-block:: shell - - sudo -u lockss docker info | grep 'Storage Driver:' - -If the output is: - -.. code-block:: text - - Storage Driver: overlay2 - -then Docker is running with the OverlayFS storage driver and you can move on to the next section. - -If the output lists another storage driver than ``overlay2`` (for example ``devicemapper``), see the Reconfiguring Docker section below, add the key-value pair ``"storage-driver":"overlay2"`` to :file:`/etc/docker/daemon.json`, and restart the Docker daemon. - ---------------------- -Initialize Swarm Mode ---------------------- - -Verify that Docker is using Swarm mode: - -.. code-block:: shell - - sudo -u lockss docker info | grep 'Swarm:' - -If the output is: - -.. code-block:: text - - Swarm: active - -then Docker is running in Swarm mode correctly and you can move on to the next section. - -If the output is empty or if the Swarm is not listed as active, initialize Swarm mode with this command: - -.. code-block:: shell - - sudo -u lockss docker swarm init - -If the output looks like this: - -.. code-block:: text - - Swarm initialized: current node (bvz81updecsj6wjz393c09vti) is now a manager. - To add a worker to this swarm, run the following command: - docker swarm join \ - --token SWMTKN-1-3pu6hszjas19xyp7ghgosyx9k8atbfcr8p2is99znpy26u2lkl-1awxwuwd3z9j1z3puu7rcgdbx \ - xx.xx.xx.xx:2377 - To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. - -where ``xx.xx.xx.xx`` is the IP address of the machine, then the Swarm initialization was successful and you can move on to the next section. - -If the output contains an error message that looks like this: - -.. code-block:: text - - Error response from daemon: could not choose an IP address to advertise since this system has multiple addresses on interface eth0 (xx.xx.xx.xx and yy.yy.yy.yy) - specify one with --advertise-addr - -or like this: - -.. code-block:: text - - Error response from daemon: could not choose an IP address to advertise since this system has multiple addresses on different interfaces (xx.xx.xx.xx on eth0 and yy.yy.yy.yy on eth1) - specify one with --advertise-addr - -then Docker was not able to automatically select an IP address among the several IP addresses it discovered. Identify the desired IP address of the machine, for example ``xx.xx.xx.xx``, and enter the modified command: - -.. code-block:: shell - - sudo -u lockss docker swarm init --advertise-addr xx.xx.xx.xx - ---------------------- -Install Local-Persist ---------------------- - -The LOCKSS system uses the `Local-Persist `_ Docker volume plugin. - -Run the `Local-Persist installation script `_ which will download and install the necessary files and set up the necessary ``systemd`` infrastructure: - -.. code-block:: shell - - curl -fsSL https://raw.githubusercontent.com/MatchbookLab/local-persist/master/scripts/install.sh | sudo bash - -If you prefer, you can follow the `Local-Persist manual installation instructions `_. - -Verify that Local-Persist is running and enabled at startup: - -* ``sudo systemctl is-active docker-volume-local-persist`` should say ``active`` -* ``sudo systemctl is-enabled docker-volume-local-persist`` should say ``enabled`` - -Also verify that Local-Persist is registered with Docker: - -* ``sudo -u lockss docker info`` should have a ``Plugins`` section with lists of volume, network and log plugins. The ``Volume`` list under the ``Plugins`` section should contain ``local-persist``. Here is an excerpt of ``docker info`` output showing that Local-Persist is correctly registered as a Docker volume plugin: - -.. code-block:: text - - Plugins: - Volume: local local-persist - Network: bridge host macvlan null overlay - Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog - --------------------- -Reconfiguring Docker --------------------- - -**This section describes what to do when Docker needs to be reconfigured. You do not need to do anything unless one of the sections above sends you here.** - -Edit or create the :file:`/etc/docker/daemon.json` configuration file and input the required key-value pairs in a JSON object, separated by commas, typically one per line for clarity. Example: - -.. code-block:: text - - { - "storage-driver": "overlay2", - "iptables": true - } - -After editing and saving the configuration file, restart the Docker daemon with systemd: - -.. code-block:: shell - - sudo systemctl restart docker diff --git a/docs/installing/git.rst b/docs/installing/git.rst deleted file mode 100644 index 60b0f86..0000000 --- a/docs/installing/git.rst +++ /dev/null @@ -1,103 +0,0 @@ -============== -Installing Git -============== - -`Git `_ is a version control system, used to interact with code repositories. - -The LOCKSS Installer is available from `GitHub `_, and you will need a Git client to download it. - -Your operating system may already be equipped with a Git client. Type: - -.. code-block:: shell - - git --version - -If the output is a version number, for example: - -.. code-block:: text - - git version 2.21.0 - -then Git is already installed and you do not need to take any further action. - -Otherwise, Git can be installed from your operating system’s software repositories. - ------------------ -Git on Arch Linux ------------------ - -Use Pacman to install Git on Arch Linux: - -.. code-block:: shell - - sudo pacman -S git - -------------- -Git on CentOS -------------- - -.. caution:: - - CentOS 7 is required. - -Use Yum to install Git on CentOS: - -.. code-block:: shell - - sudo yum install git - -------------- -Git on Debian -------------- - -.. caution:: - - Debian 9 (Stretch) is required. - -Use Apt to install Git on Debian: - -.. code-block:: shell - - sudo apt-get install git - -------------- -Git on Fedora -------------- - -.. caution:: - - Fedora 28 or better is required. - -Use Dnf to install Git on Fedora: - -.. code-block:: shell - - sudo dnf install git - -------------------- -Git on Oracle Linux -------------------- - -.. caution:: - - Oracle Linux 7 is required. - -Use Yum to install Git on Oracle Linux: - -.. code-block:: shell - - sudo yum install git - -------------- -Git on Ubuntu -------------- - -.. caution:: - - Ubuntu 16.04 LTS (Xenial) or better is required. - -Use Apt to install Git on Ubuntu: - -.. code-block:: shell - - sudo apt-get install git diff --git a/docs/installing/index.rst b/docs/installing/index.rst deleted file mode 100644 index 6d02274..0000000 --- a/docs/installing/index.rst +++ /dev/null @@ -1,14 +0,0 @@ -============================ -Installing the LOCKSS System -============================ - -This section describes how to install the LOCKSS system. - -.. toctree:: - :hidden: - - prerequisites - docker - pystache - git - lockss-installer diff --git a/docs/installing/lockss-installer.rst b/docs/installing/lockss-installer.rst deleted file mode 100644 index 9be07ea..0000000 --- a/docs/installing/lockss-installer.rst +++ /dev/null @@ -1,15 +0,0 @@ -================================ -Downloading the LOCKSS Installer -================================ - -You can download the LOCKSS Installer from GitHub using a :doc:`Git ` command as the ``lockss`` user created in :doc:`prerequisites`: - -.. code-block:: shell - - git clone https://github.com/lockss/lockss-installer - -You can then enter the :file:`lockss-installer` directory: - -.. code-block:: shell - - cd lockss-installer diff --git a/docs/installing/prerequisites.rst b/docs/installing/prerequisites.rst deleted file mode 100644 index 142677a..0000000 --- a/docs/installing/prerequisites.rst +++ /dev/null @@ -1,30 +0,0 @@ -==================== -System Prerequisites -==================== - -------- -Machine -------- - -The LOCKSS system runs on a **64-bit Linux** host (physical or virtual), with **4 cores** (8 or more preferable) and **8 GB of memory** (16 GB or more preferable). - ----------------- -Operating System ----------------- - -The LOCKSS system requires a **64-bit Linux** host compatible with `Systemd `_ and `Docker `_ **18.09 or better**. - -Many Linux distributions have systemd and can run Docker 18.09 or better. To name a few that are commonly used with the LOCKSS system: - -* `Arch Linux `_ -* `CentOS `_ 7 -* `Debian `_ 9 (Stretch) -* `Fedora `_ 28 or better -* `Oracle Linux `_ 7 -* `Ubuntu `_ 16.04 LTS (Xenial) or better - ----- -User ----- - -The LOCKSS system runs under a system user named ``lockss`` under a group named ``lockss``, which you will need to create. diff --git a/docs/installing/pystache.rst b/docs/installing/pystache.rst deleted file mode 100644 index a238fec..0000000 --- a/docs/installing/pystache.rst +++ /dev/null @@ -1,32 +0,0 @@ -=================== -Installing Pystache -=================== - -`Pystache `_ is the reference implementation of the `Mustache `_ templating language in Python. This page will walk you through the simple process of installing Pystache. The Pystache Python module depends on the Setuptools Python module. - -You can install Pystache from `PyPI ` using Pip: - -.. code-block:: shell - - sudo pip install setuptools pystache - -Verify that Pystache is installed correctly: - -.. code-block:: shell - - pystache --help - -You should see a help message about Pystache, for example: - -.. code-block:: text - - Usage: pystache [-h] template context - - Render a mustache template with the given context. - - positional arguments: - template A filename or template string. - context A filename or JSON string. - - Options: - -h, --help show this help message and exit diff --git a/docs/running.rst b/docs/running.rst deleted file mode 100644 index 2506f4e..0000000 --- a/docs/running.rst +++ /dev/null @@ -1,11 +0,0 @@ -========================= -Running the LOCKSS System -========================= - -After :doc:`configuring` and anytime after updating the LOCKSS Installer to a new version and stopping your LOCKSS stack: - -* Run :file:`scripts/generate-lockss`. This script takes your configuration data and turns into into a set of configuration files containing the right values. - -* Run :file:`scripts/assemble-lockss`. This script puts the configuration files and puts them in the right places, and ensures that all storage volumes are ready for use (creating them if necessary). - -* Run :file:`scripts/deploy-lockss`. This script deploys your LOCKSS stack by invoking Docker. diff --git a/docs/using/configuration.rst b/docs/using/configuration.rst deleted file mode 100644 index 40b654d..0000000 --- a/docs/using/configuration.rst +++ /dev/null @@ -1,49 +0,0 @@ -====================================== -Using the LOCKSS Configuration Service -====================================== - -.. note:: - - This page is under construction. - --------------------------------- -Accessing the Web User Interface --------------------------------- - -If you are already connected to the Web user interface (UI) of another component of the LOCKSS System, click :guilabel:`Config Service` in the top-left menu. - -Alternatively, if your primary hostname is :samp:`{}`, you can use your browser to connect to the LOCKSS Configuration Service Web user interface (UI) at :samp:`http://{}:24621`. - -Enter your Web UI username and password to login if prompted. - ---------------------- -Adding Archival Units ---------------------- - -To add AUs to the system for preservation: - -1. In the top-right menu, click :guilabel:`Journal Configuration`. -2. In the center menu, click :guilabel:`Add AUs`. -3. Select one or more collections of AUs by selecting the checkbox next to the appropriate collection. -4. Click the :guilabel:`Select AUs` button. It may take a bit of time (60+ seconds) for the next screen to appear, while the list of AUs is built. -5. Select one or more AUs from the AU list. You may click :guilabel:`Select All` if you would like to select all AUs. If you choose to use select all AUs, please note that the next step may take some time to load. -6. Click the :guilabel:`Add Selected AUs` button. The time it takes for the page to refresh depends on the number of AUs added. Give the LOCKSS system some time to load the AUs and reload the page before moving on. -7. A screen will show a list of added AUs. Crawling of these new AUs will start automatically -- no further action is necessary unless prompted by a footnote next to an AU's name. - -------------------------- -Configuring a Crawl Proxy -------------------------- - -If Web crawls must be routed through a Web proxy: - -1. In the top-right menu, click :guilabel:`Content Access Options`. -2. In the center menu, click :guilabel:`Proxy Client Options`. -3. Select the :guilabel:`Proxy crawls` checkbox. -4. Enter the hostname and port of the Web proxy in the :guilabel:`HTTP Proxy host` and :guilabel:`Port` text areas, respectively. -5. Click the :guilabel:`Update Proxy Client` button. - ------------------------------------------- -Managing Access to the Web User Interfaces ------------------------------------------- - -*This section is under construction.* diff --git a/docs/using/crawler.rst b/docs/using/crawler.rst deleted file mode 100644 index f694980..0000000 --- a/docs/using/crawler.rst +++ /dev/null @@ -1,93 +0,0 @@ -================================ -Using the LOCKSS Crawler Service -================================ - -.. note:: - - This page is under construction. - --------------------------------- -Accessing the Web User Interface --------------------------------- - -.. note:: - - Currently the crawler service is run as part of the poller service. - -If you are already connected to the Web user interface (UI) of another component of the LOCKSS System, click :guilabel:`Crawler Service` in the top-left menu. - -Alternatively, if your primary hostname is :samp:`{}`, you can use your browser to connect to the LOCKSS Configuration Service Web user interface (UI) at :samp:`http://{}:24631`. - -Enter your Web UI username and password to login if prompted. - -------------------------------------- -Monitoring Crawl Status in the System -------------------------------------- - -The Crawl status of all configured AUs is available in the Archival Unit table - -1. In the top-right menu, click :guilabel:`Daemon Status`. - -2. Open the control in the middle of the screen that says :guilabel:`Overview` and select :guilabel:`Archival Units:guilabel:` from the drop down menu. - - * If prompted, enter your Username and Password again. - - * It will take a bit of time for the next screen to appear while the AU list is being built. - -3. The Archival Units screen lists statistics for each configured AU - - * the :guilabel:`Last Successful Crawl` column provides a timestamp of the most recent sucessful crawl. - - * the :guilabel:`Last Crawl Start` column provides a timestamp of the last attempted crawl. - - * the :guilabel:`Last Crawl Result` column provides the exit status of the last attempted crawl. - ---------------------------------- -Causing an Archival Unit to Crawl ---------------------------------- - -Archival units (AUs) that have been added to the system for preservation crawl periodically, but you can cause an AU to crawl on demand: - -1. In the top-right menu, click :guilabel:`Debug Panel`. - -2. Select an AU in the :guilabel:`AU Actions: select AU` drop-down list. - -3. Click the :guilabel:`Start Crawl` button. - -4. If the AU has crawled recently, you will be prompted to confirm that you wish to override the usual recrawl interval by clicking on the :guilabel:`Force Start Crawl` button. - -------------------- -Crawl Status Screen -------------------- - -To inspect the state of crawls, access the :guilabel:`Crawl Status` screen: - -1. In the top-right menu, click :guilabel:`Daemon Status`. - -2. In the center drop-down list, select :guilabel:`Crawl Status`. Alternatively, in the center overview, click on the second line, which says "*N* active crawls". - -Top-Level Crawl Information -=========================== - -The top left of the Crawl Status table contains the number of active, successful or failed crawls, and a countdown until the next time the system will look at the AUs being preserved and pick some that are ready to crawl or recrawl. - -Crawl Status Entry -================== - -Each line in the Crawl Status table contains: - -* The name of the AU -* The type of crawl -* The start time of the crawl -* The duration of a finished or in-progress crawl -* The status of the crawl -* The number of bytes fetched over the network as part of the crawl -* The number of URLs fetched as part of the crawl -* The number of URLs parsed for more links -* The number of URLs remaining to be fetched as part of this crawl -* The number of URLs encountered as part of this crawl but excluded from being fetched -* The number of URLs fetched as part of the crawl, that received an HTTP Not Modified response -* The number of URLs that caused errors as part of this crawl -* The number of different content types encountered as part of the crawl - -Most of these values can be clicked to see a list of the corresponding objects. diff --git a/docs/using/index.rst b/docs/using/index.rst deleted file mode 100644 index ce8031d..0000000 --- a/docs/using/index.rst +++ /dev/null @@ -1,15 +0,0 @@ -======================= -Using the LOCKSS System -======================= - -This section describes how to use the LOCKSS system. - -.. toctree:: - :hidden: - - configuration - metadata-extraction - metadata - poller - crawler - pywb diff --git a/docs/using/metadata-extraction.rst b/docs/using/metadata-extraction.rst deleted file mode 100644 index 05f0e79..0000000 --- a/docs/using/metadata-extraction.rst +++ /dev/null @@ -1,23 +0,0 @@ -============================================ -Using the LOCKSS Metadata Extraction Service -============================================ - -.. note:: - - This page is under construction. - --------------------------------- -Accessing the Web User Interface --------------------------------- - -If you are already connected to the Web user interface (UI) of another component of the LOCKSS System, click :guilabel:`Metadata Extraction Service` in the top-left menu. - -Alternatively, if your primary hostname is :samp:`{}`, you can use your browser to connect to the LOCKSS Configuration Service Web user interface (UI) at :samp:`http://{}:24641`. - -Enter your Web UI username and password to login if prompted. - ------------------------------- -Requesting Metadata Extraction ------------------------------- - -*This section is under construction.* diff --git a/docs/using/metadata.rst b/docs/using/metadata.rst deleted file mode 100644 index b12fba2..0000000 --- a/docs/using/metadata.rst +++ /dev/null @@ -1,23 +0,0 @@ -================================= -Using the LOCKSS Metadata Service -================================= - -.. note:: - - This page is under construction. - --------------------------------- -Accessing the Web User Interface --------------------------------- - -If you are already connected to the Web user interface (UI) of another component of the LOCKSS System, click :guilabel:`Metadata Service` in the top-left menu. - -Alternatively, if your primary hostname is :samp:`{}`, you can use your browser to connect to the LOCKSS Configuration Service Web user interface (UI) at :samp:`http://{}:24651`. - -Enter your Web UI username and password to login if prompted. - -------------------------------- -Requesting Metadata Information -------------------------------- - -*This section is under construction.* diff --git a/docs/using/poller.rst b/docs/using/poller.rst deleted file mode 100644 index ed74a66..0000000 --- a/docs/using/poller.rst +++ /dev/null @@ -1,29 +0,0 @@ -=============================== -Using the LOCKSS Poller Service -=============================== - -.. note:: - - This page is under construction. - --------------------------------- -Accessing the Web User Interface --------------------------------- - -If you are already connected to the Web user interface (UI) of another component of the LOCKSS System, click :guilabel:`Poller Service` in the top-left menu. - -Alternatively, if your primary hostname is :samp:`{}`, you can use your browser to connect to the LOCKSS Configuration Service Web user interface (UI) at :samp:`http://{}:24631`. - -Enter your Web UI username and password to login if prompted. - ----------------- -Requesting Polls ----------------- - -*This section is under construction.* - ------------------------------ -Monitoring Polling and Voting ------------------------------ - -*This section is under construction.* diff --git a/docs/using/pywb.rst b/docs/using/pywb.rst deleted file mode 100644 index c00f7a7..0000000 --- a/docs/using/pywb.rst +++ /dev/null @@ -1,51 +0,0 @@ -=============================== -Replaying Web Content with Pywb -=============================== - ---------------------------------- -Accessing the Pywb User Interface ---------------------------------- - -Given that your primary hostname is samp:`{}`, you can use your browser to connect to the Pywb user interface (UI) at :samp:`http://{}:24681`. - ---------------- -Replaying a URL ---------------- - -To view a URL from Pywb: - -1. The Pywb screen provides a list of links to available collections. Click on the top-most collection which should be ``/lockss``. - -2. Enter the URL you want to replay in the URL search box. - -3. Click the :guilabel:`Search` button. - -4. Replay the most recent URL by clicking on the topmost entry of the third column. - ----------------------------------- -Finding a URL From an AU to Replay ----------------------------------- - -There are multiple ways to discover URLs belonging to an AU in the Configuration Service UI: - -1. Obtaining a URL by clicking on "pages fetched" inside of crawl status - - * In the top-right menu, click :guilabel:`Daemon Status`. - - * Open the control in the middle of the screen that says *Overview* and select :guilabel:`Crawl Status` from the drop down menu. - - * Picking an AU from the active crawls, click on the number associated with :guilabel:`Pages Fetched` to bring up a list of URLs that have been crawled. - - * Copy one of the URLs and paste it in the Pywb interface as described previously. - -2. Obtaining a Substance URL - - * In the top-right menu, click :guilabel:`Daemon Status`. - - * Open the control in the middle of the screen that says :guilabel:`Overview` and select :guilabel:`Archival Units` from the drop down menu. If prompted, enter your Username and Password again. It will take a bit of time for the next screen to appear while the AU list is being built. - - * Select an AU by clicking on the AU title in the first column. - - * Open the :guilabel:`Substance URLs` link - - * Copy one of the URLs and paste it in the Pywb interface as described previously.