Merge remote-tracking branch 'origin/release/4.3' into bugfix/install…

…-plantuml-dependencies

docs/algorithm_store/configure.rst

@@ -0,0 +1,113 @@
+.. _algorithm-store-configure:
+
+Configure
+---------
+
+The algorithm store requires a configuration file to run. This is a
+``yaml`` file with a specific format.
+
+The next sections describes how to configure the algorithm store. It first provides a few
+quick answers on setting up your store, then shows an example of all
+configuration file options, and finally explains where your configuration files
+are stored.
+
+How to create a configuration file
+""""""""""""""""""""""""""""""""""
+
+The easiest way to create an initial configuration file is via:
+``v6 algorithm-store new``. This allows you to configure the
+basic settings. For more advanced configuration options, which are listed below,
+you can view the :ref:`example configuration file <algorithm-store-config-file-structure>`.
+
+
+Where is my configuration file?
+"""""""""""""""""""""""""""""""
+
+To see where your configuration file is located, you can use the following
+command
+
+.. code:: bash
+
+    v6 algorithm-store files
+
+.. warning::
+    This command will only work for if the algorithm store has been deployed
+    using the ``v6`` commands.
+
+    Also, note that on local deployments you may need to specify the
+    ``--user`` flag if you put your configuration file in the
+    :ref:`user folder <algorithm-store-configure-location>`.
+
+You can also create and edit this file manually.
+
+.. _algorithm-store-config-file-structure:
+
+All configuration options
+"""""""""""""""""""""""""
+
+The following configuration file is an example that intends to list all possible
+configuration options.
+
+You can download this file here: :download:`algorithm_store_config.yaml <yaml/algorithm_store_config.yaml>`
+
+.. _algorithm-store-configuration-file:
+
+.. literalinclude :: yaml/algorithm_store_config.yaml
+    :language: yaml
+
+.. todo this section is close duplicate of docs/node/configure -- merge?
+
+.. _algorithm-store-configure-location:
+
+Configuration file location
+"""""""""""""""""""""""""""
+
+The directory where to store the configuration file depends on your
+operating system (OS). It is possible to store the configuration file at
+**system** or at **user** level. At the user level, configuration files are only
+available for your user. By default, algorithm store configuration files are
+stored at **system** level.
+
+The default directories per OS are as follows:
+
++---------+----------------------------+------------------------------------+
+| **OS**  | **System**                 | **User**                           |
++=========+============================+====================================+
+| Windows | |win_sys|                  | |win_usr|                          |
++---------+----------------------------+------------------------------------+
+| MacOS   | |mac_sys|                  | |mac_usr|                          |
++---------+----------------------------+------------------------------------+
+| Linux   | |lin_sys|                  | |lin_usr|                          |
++---------+----------------------------+------------------------------------+
+
+.. |win_sys| replace:: ``C:\ProgramData\vantage\algorithm-store``
+.. |win_usr| replace:: ``C:\Users\<user>\AppData\Local\vantage\algorithm-store``
+.. |mac_sys| replace:: ``/Library/Application/Support/vantage6/algorithm-store``
+.. |mac_usr| replace:: ``/Users/<user>/Library/Application Support/vantage6/algorithm-store``
+.. |lin_sys| replace:: ``/etc/xdg/vantage6/algorithm-store/``
+.. |lin_usr| replace:: ``/home/<user>/.config/vantage6/algorithm-store/``
+
+.. warning::
+    The command ``v6 algorithm-store`` looks in certain directories by default. It is
+    possible to use any directory and specify the location with the ``--config``
+    flag. However, note that using a different directory requires you to specify
+    the ``--config`` flag every time!
+
+    Similarly, you can put your algorithm store configuration file in the user folder
+    by using the ``--user`` flag. Note that in that case, you have to specify
+    the ``--user`` flag for all ``v6 algorithm-store`` commands.
+
+.. _algorithm-store-logging:
+
+Logging
+"""""""
+
+Logging is enabled by default. To configure the logger, look at the ``logging``
+section in the example configuration in :ref:`algorithm-store-config-file-structure`.
+
+Useful commands:
+
+1. ``v6 algorithm-store files``: shows you where the log file is stored
+2. ``v6 algorithm-store attach``: show live logs of a running store in your
+   current console. This can also be achieved when starting the store
+   with ``v6 algorithm-store start --attach``

docs/algorithm_store/deploy.rst

@@ -0,0 +1,47 @@
+
+.. _algorithm-store-deployment:
+
+Deploy
+^^^^^^
+
+The deployment of the algorithm store is highly similar to the deployment of
+the vantage6 server. Both are Flask applications that are structured very
+similarly.
+
+The algorithm store's deployment is a bit simpler because it does not use
+socketIO. This means that you don't have to take into account that the websocket
+channels should be open, and makes it easier to horizontally scale the
+application.
+
+.. _deploy-algostore-nginx:
+
+NGINX
+"""""
+
+The algorithm store can be deployed with a similar nginx script as detailed
+for the :ref:`server <deploy-nginx>`.
+
+One note is that for the algorithm store, the subpath is fixed at `/api`, so
+be sure to set that in the subpath block.
+
+.. _deploy-docker-compose:
+
+Docker compose
+""""""""""""""
+
+The algorithm store can be started with ``v6 algorithm-store start``, but in
+most deployment scenarios a docker-compose file is used. Below is an example
+of a docker-compose file that can be used to deploy the algorithm store.
+
+.. code:: yaml
+
+    services:
+      vantage6-algorithm-store:
+        image: harbor2.vantage6.ai/infrastructure/algorithm-store:cotopaxi
+        ports:
+        - "8000:5000"
+        volumes:
+        - /path/to/my/server.yaml:/mnt/config.yaml
+        command: ["/bin/bash", "-c", "/vantage6/vantage6-algorithm-store/server.sh"]
+
+.. TODO How to deploy on Azure app service

docs/algorithm_store/index.rst

@@ -0,0 +1,21 @@
+.. include:: <isonum.txt>
+
+.. _algorithm-store-admin-guide:
+
+Algorithm store admin guide
+===========================
+
+This section shows you how you can set up your own vantage6 algorithm store.
+First, we discuss the requirements for hosting this server, then guide you
+through the installation process. Finally, we explain how to configure and
+start your algorithm store.
+
+.. toctree::
+    :maxdepth: 3
+
+    introduction
+    requirements
+    install
+    deploy
+    use
+    configure

docs/algorithm_store/install.rst

@@ -0,0 +1,9 @@
+.. _install-algostore:
+
+.. |instance-type| replace:: algorithm store
+.. |requirements-link| replace:: :ref:`requirements <algorithm-store-requirements>`
+.. |image| replace:: ``harbor2.vantage6.ai/infrastructure/algorithm-store``
+.. |image-old| replace:: ``harbor2.vantage6.ai/infrastructure/algorithm-store:<VERSION>``
+.. |deployment-link| replace:: :ref:`deployment <algorithm-store-deployment>`
+
+.. include:: ../common/install.rst

docs/algorithm_store/introduction.rst

@@ -0,0 +1,33 @@
+.. _algorithm-store:
+
+Introduction
+------------
+
+What is an algorithm store?
+"""""""""""""""""""""""""""
+
+When using vantage6, it is important to know which algorithms are available
+to you. This is why vantage6 has algorithm stores. An algorithm store contains
+metadata about the algorithms so that you can easily find the algorithm you
+need, and know how to use it.
+
+There is a community algorithm store, which is by default available to all
+collaborations. This store is maintained by the vantage6 community. You can
+also create your own algorithm store. This allows you to create a private
+algorithm store, which is only available to your own collaborations.
+.. # TODO add link to creating algorithm store
+.. TODO add links to an architectural page where algorithm store is explained
+
+Linking algorithm stores
+""""""""""""""""""""""""
+
+Algorithm stores can be linked to a vantage6 server or to a specific
+collaboration on a server. If an algorithm store is linked to a server, the
+algorithms in the store are available to all collaborations on that server. If
+an algorithm store is linked to a collaboration, the algorithms in the store
+are only available to that collaboration.
+
+Users can link algorithm stores to a collaboration if they have permission to
+modify that collaboration. Algorithm stores can only be linked to a server by
+users that have permission to modify all collaborations on the server.
+

docs/algorithm_store/requirements.rst

@@ -0,0 +1,14 @@
+.. _algorithm-store-requirements:
+
+Requirements
+------------
+
+.. note::
+    This section is almost the same as the :ref:`node <node-requirements>` and
+    :ref:`server <server-requirements>` sections - their requirements are very
+    similar.
+
+.. |requirement-public-ip| replace:: *Note that the algorithm store's IP address should also be reachable by users and the central server. This will usually be a public IP address.*
+.. |requirement-OS| replace:: \
+
+.. include:: ../common/requirements-node-server.rst

docs/algorithm_store/use.rst

@@ -0,0 +1,61 @@
+.. _use-algorithm-store:
+
+Use
+---
+
+This section explains which commands are available to manage your algorithm
+store. These can be used to set up a test server locally. To deploy a server,
+see the :ref:`deployment <algorithm-store-deployment>` section.
+
+
+Quick start
+"""""""""""
+
+To create a new algorithm store, run the command below. A menu will be started
+that allows you to set up an algorithm store configuration file.
+
+::
+
+   v6 algorithm-store new
+
+For more details, check out the :ref:`algorithm-store-configure` section.
+
+To run an algorithm store, execute the command below. The ``--attach`` flag will
+copy log output to the console.
+
+::
+
+   v6 algorithm-store start --name <your_store> --attach
+
+Finally, a server can be stopped again with:
+
+::
+
+   v6 algorithm-store stop --name <your_store>
+
+Available commands
+""""""""""""""""""
+
+The following commands are available in your environment. To see all the
+options that are available per command use the ``--help`` flag,
+e.g. ``v6 server start --help``.
+
+.. list-table:: Available commands
+   :name: algorithm-store-commands
+   :widths: 30 70
+   :header-rows: 1
+
+   * - Command
+     - Description
+   * - ``v6 algorithm-store new``
+     - Create a new algorithm store configuration file
+   * - ``v6 algorithm-store start``
+     - Start an algorithm store
+   * - ``v6 algorithm-store stop``
+     - Stop an algorithm store
+   * - ``v6 algorithm-store files``
+     - List the files that an algorithm store is using
+   * - ``v6 algorithm-store attach``
+     - Show an algorithm store's logs in the current terminal
+   * - ``v6 algorithm-store list``
+     - List the available algorithm store instances

docs/algorithm_store/yaml/algorithm_store_config.yaml

@@ -0,0 +1,64 @@
+# Human readable description of the algorithm store instance. This is to help
+# your peers to identify the store
+description: Test
+
+# IP adress to which the algorithm store server binds. In case you specify 0.0.0.0
+# the server listens on all interfaces
+ip: 0.0.0.0
+
+# Port to which the algorithm store binds
+port: 5000
+
+# The URI to the algorithm store database. This should be a valid SQLAlchemy URI,
+# e.g. for an Sqlite database: sqlite:///database-name.sqlite,
+# or Postgres: postgresql://username:password@172.17.0.1/database).
+uri: sqlite:///test.sqlite
+
+# This should be set to false in production as this allows to completely
+# wipe the database in a single command. Useful to set to true when
+# testing/developing.
+allow_drop_all: True
+
+# Settings for the logger
+logging:
+  # Controls the logging output level. Could be one of the following
+  # levels: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET
+  level: DEBUG
+
+  # Filename of the log-file, used by RotatingFileHandler
+  file: test.log
+
+  # Whether the output is shown in the console or not
+  use_console: True
+
+  # The number of log files that are kept, used by RotatingFileHandler
+  backup_count: 5
+
+  # Size in kB of a single log file, used by RotatingFileHandler
+  max_size: 1024
+
+  # format: input for logging.Formatter,
+  format: "%(asctime)s - %(name)-14s - %(levelname)-8s - %(message)s"
+  datefmt: "%Y-%m-%d %H:%M:%S"
+
+  # (optional) set the individual log levels per logger name, for example
+  # mute some loggers that are too verbose.
+  loggers:
+    - name: urllib3
+      level: warning
+    - name: sqlalchemy.engine
+      level: warning
+
+# Additional debug flags
+debug:
+  # Set to `true` to enable debug mode in the Flask app
+  flask: false
+
+# development mode settings. Only use when running both the algorithm store and
+# the server that it communicates with locally
+dev:
+  # Specify the URI to the host. This is used to generate the correct URIs to
+  # communicate with the server. On Windows and Mac, you can use the special
+  # hostname `host.docker.internal` to refer to the host machine. On Linux, you
+  # should normally use http://172.17.0.1.
+  host_uri: http://host.docker.internal

docs/algorithms/index.rst

@@ -5,7 +5,7 @@ Algorithm Development
 
 This section helps you to develop MPC and FL algorithms that are compatible
 with vantage6. You are **not** going to find a list of algorithms here or help
-on how to use them. In the :ref:`components`, the basic
+on how to use them. In the :ref:`main-intro`, the basic
 concepts and interface between node and algorithm are explained. Then, in the
 :ref:`algo-dev-guide`, the algorithm development process is explained. Finally,
 in the :ref:`classic-algo-tutorial`, an FL algorithm is build from scratch.