Updated backends docs, caching and cookbook

docs/backends.rst

@@ -3,11 +3,13 @@
 Backends
 ========
 
-There are currently four different backends. They can also be initialized in few different ways, consistent with general settings.
-You can use the ``ARCA_BACKEND`` setting or you can pass a ``backend`` keyword directly to ``Arca``.
-This setting can be either a string, class or an instance. String is used to load the class and initialize the instance,
-class is used to initialize and a instance is used as is. All the initializations shown bellow are equivalent, but again,
-as mentioned above, the straight ``backend`` keyword cannot be overridden by settings or environ variables.
+There are currently four different backends. They can also be initialized in few different ways,
+consistent with general settings. You can use the ``ARCA_BACKEND`` setting
+or you can pass a ``backend`` keyword directly to ``Arca``.
+
+The backend setting can be either a string, class or an instance. All the initializations shown bellow are equivalent,
+but again, as mentioned in :ref:`configuring`, the ``backend`` keyword cannot be overridden by settings
+or environ variables.
 
 .. code-block:: python
 
@@ -21,8 +23,8 @@ as mentioned above, the straight ``backend`` keyword cannot be overridden by set
 
 
 Setting up backends is based on the same principle as setting up ``Arca``.
-You can either pass keyword arguments when initializing class
-or you can use settings with the prefix ``ARCA_BACKEND_``. For example these two calls are equivalent:
+You can either pass keyword arguments when initializing the backend class
+or you can use settings (described in more details in :ref:`configuring`. For example these two calls are equivalent:
 
 .. code-block:: python
 
@@ -34,12 +36,10 @@ or you can use settings with the prefix ``ARCA_BACKEND_``. For example these two
   })
   Arca(backend=DockerBackend(python_version="3.6.4"))
 
+As mentioned in :ref:`options`, there are two options common for all backends. (See that section for more details.)
 
-There are two options common for all backends:
-
-* **requirements_location**: Tells backends where they should look for requirements in the repositories,
-  the default is ``requirements.txt``.
-* **cwd**: Tells backends in which folder they should launch the code, default is the root folder of the repository.
+* **requirements_location**
+* **cwd**
 
 .. _backends_cur:
 
@@ -48,21 +48,27 @@ Current Environment
 
 *arca.backend.CurrentEnvironmentBackend*
 
-This backend is the default option, it runs the tasks with the same Python that's used to run arca in a subprocess.
+This backend is the default option, it runs the tasks with the same Python that's used to run Arca, in a subprocess.
 There are two settings for this backend, to determine how the backend should treat requirements in the repositories.
 
 * **current_environment_requirements**: a path to the requirements of the current environment,
   the default is ``requirements.txt``.
   ``None`` would indicate there are no requirements for the current environment.
 * **requirements_strategy**: Which approach the backend should take. There are three, the default being ``raise``.
 
+(possible settings prefixes: ``ARCA_CURRENT_ENVIRONMENT_BACKEND_`` and ``ARCA_BACKEND_``)
+
 Requirements strategies:
+++++++++++++++++++++++++
 
-* ``raise``: Raise an ``arca.exceptions.RequirementsMismatch`` if there are any extra requirements
-  in the target repository.
-* ``ignore``: Ignore any extra requirements.
-* ``install_extra``: Install the requirements that are extra in the target repository as
-  opposed to the current environment.
+The strategies are defined in a enum, ``arca.RequirementsStrategy``. Its values or the string representation can be
+used in settings.
+
+* ``raise``, ``RequirementsStrategy.RAISE``:
+  Raise an ``arca.exceptions.RequirementsMismatch`` if there are any extra requirements in the target repository.
+* ``ignore``, ``RequirementsStrategy.IGNORE``: Ignore any extra requirements.
+* ``install_extra``, ``RequirementsStrategy.INSTALL_EXTRA``:
+  Install the requirements that are extra in the target repository as opposed to the current environment.
 
 .. _backends_vir:
 
@@ -76,6 +82,8 @@ used to run Arca and they are shared between repositories that have the same exa
 The virtual environments are stored in folder ``venv`` in folder
 determined by the ``Arca`` ``base_dir`` setting, usually ``.arca``.
 
+(possible settings prefixes: ``ARCA_VENV_BACKEND_`` and ``ARCA_BACKEND_``)
+
 .. _backends_doc:
 
 Docker
@@ -121,6 +129,8 @@ Settings:
 * **push_to_registry_name**: Pushes all built images with installed requirements and dependencies to
   docker registry with this name, tries to pull image from the registry before building it locally to save time.
 
+(possible settings prefixes: ``ARCA_DOCKER_BACKEND_`` and ``ARCA_BACKEND_``)
+
 .. _backends_vag:
 
 Vagrant
@@ -150,6 +160,8 @@ for ``keep_container_running`` and has these extra settings:
 * **destroy**: Destroy the VM right after the task is finished if ``True`` (default).
   If ``False`` is set, the VM is only halted.
 
+(possible settings prefixes: ``ARCA_VAGRANT_BACKEND_`` and ``ARCA_BACKEND_``)
+
 Your own
 --------
 

docs/caching.rst

@@ -0,0 +1,32 @@
+.. _caching:
+
+Caching
+=======
+
+Arca can cache results of the tasks using `dogpile.cache <https://dogpilecache.readthedocs.io/en/latest/>`_.
+The default cache backend is ``dogpile.cache.null``, so no caching.
+
+You can setup caching with the backend setting ``ARCA_CACHE_BACKEND`` and all the arguments needed for setup can be set
+using ``ARCA_CACHE_BACKEND_ARGUMENTS`` which can either be a dict or a json string.
+
+Example setup:
+
+.. code-block:: python
+
+    arca = Arca(settings={
+        "ARCA_CACHE_BACKEND": "dogpile.cache.redis",
+        "ARCA_CACHE_BACKEND_ARGUMENTS": {
+            "host": "localhost",
+            "port": 6379,
+            "db": 0,
+        }
+    })
+
+To see all available cache backends and their settings,
+please visit the ``dogpile.cache`` `documentation <https://dogpilecache.readthedocs.io/en/latest/>`_.
+Some of the other backends might have other python dependencies.
+
+When ``Arca`` is being initialized, a check is made if the cache backend is writable and readable,
+which raises an ``arca.exceptions.ArcaMisconfigured`` if it's not.
+If the cache requires some python dependency :class:`ModuleNotFoundError` will be raised.
+If you wish to ignore these errors, ``ignore_cache_errors`` setting can be used.

docs/cookbook.rst

@@ -3,53 +3,76 @@
 Cookbook
 ========
 
-Static files
-------------
+I need some files from the repositories
+---------------------------------------
 
-With method ``static_filename`` you can request a file from a repository.
-The method accepts a relative path (can be a ``pathlib.Path`` or ``str``) to the file in the repository
-and it returns an absolute path to the file is currently stored. The method raises ``arca.exceptions.FileOutOfRangeError``
-when the relative path goes outside the scope of the repository and ``FileNotFoundError`` if the file doesn't exist in the
-repository.
+Let's say there are some files in the repository that you need for something,
+for example a image ``images\example.png`` from the branch ``master`` of ``https://example.com/hello_word.git``.
 
-Singe pull
-----------
+With the :class:`Arca <arca.Arca>` method ``static_filename`` you can get the absolute path to that file, to where Arca cloned it.
+The method accepts a relative path (can be a ``pathlib.Path`` or ``str``) to the file in the repository.
+
+Example call:
+
+.. code-block:: python
+
+  arca = Arca()
 
-You might not want the repositories to be pulled everytime ``run`` is called.
-You can specify that you want each repository and branch combination should be pulled only once
-per initialization of ``Arca`` with the ``ARCA_SINGLE_PULL`` setting or ``single_pull`` keyword argument.
+  path_to_file = arca.static_filename("https://example.com/hello_word.git",
+                                      "master",
+                                      "images/example.png")
 
-You can tell ``Arca`` to pull again by calling the method ``pull_again``.
+``path_to_file`` will be an absolute :class:`Path <pathlib.Path>`.
 
-Caching
--------
+If the file is not in the repository, :class:`FileNotFoundError` will be raised. If the provided relative path
+leads out of the repo, :class:`FileOutOfRangeError <arca.exceptions.FileOutOfRangeError>` will be raised.
+
+I will be running a lot of tasks from the same repository
+---------------------------------------------------------
+
+Same as the previous recipe, this might be useful for example if you're building a static page from the repository and
+you want to speed things up. Arca has some overhead for each launched task, but it can be sped up.
+
+Singe pull
+++++++++++
 
-Arca can cache results of the tasks using ``dogpile.cache``. The default cache backend is ``dogpile.cache.null``, so no caching.
-You can setup caching with the backend setting ``ARCA_CACHE_BACKEND`` and all the arguments needed for setup can be set
-using ``ARCA_CACHE_BACKEND_ARGUMENTS`` which can either be a dict or or a json string.
+This option ensures that each branch is only cloned/pulled once per initialization of :class:`Arca <arca.Arca>`.
+You can set it up with the :class:`Arca <arca.Arca>` ``single_pull`` option (``ARCA_SINGLE_PULL`` setting).
 
-Example setup:
+You can tell :class:`Arca <arca.Arca>` to pull again (if a task from that repo/branch is called again)
+by calling the method :meth:`Arca.pull_again <arca.Arca.pull_again>`.
 
 .. code-block:: python
 
-    arca = Arca(settings={
-        "ARCA_CACHE_BACKEND": "dogpile.cache.redis",
-        "ARCA_CACHE_BACKEND_ARGUMENTS": {
-            "host": "localhost",
-            "port": 6379,
-            "db": 0,
-        }
-    })
+  arca = Arca()
 
-To see all available cache backends and their settings,
-please visit the ``dogpile.cache`` `documentation <https://dogpilecache.readthedocs.io/en/latest/>`_.
+  ...
+
+  # only this specific branch will be pulled again
+  arca.pull_again(repo="https://example.com/hello_word.git", branch="master")
+
+  # all branches from this repo will be pulled again
+  arca.pull_again(repo="https://example.com/hello_word.git")
+
+  # everything will be pulled again
+  arca.pull_again()
+
+
+Running container
++++++++++++++++++
+
+If you're using the :ref:`Docker <backends_doc>` backend, you can speed up things by keeping the containers
+for running the tasks running. Since a container for each repository is launched, this can speed up things considerably,
+because starting up, copyting files and shutting down containers takes time.
+
+This can be enabled by setting the ``keep_container_running`` option to ``True``.
+When you're done with running the tasks you can kill the containers by calling the method
+:meth:`DockerBackend.stop_containers <arca.DockerBackend.stop_containers>`:
+
+.. code-block:: python
 
-When ``Arca`` is being initialized, a check is made if the cache backend is writable and readable,
-which raises an ``arca.exceptions.ArcaMisconfigured`` if it's not.
-If you wish to ignore this error, set ``ARCA_IGNORE_CACHE_ERRORS`` to True or
-initialize ``Arca``  with ``ignore_cache_errors`` keyword argument.
+  arca = Arca(backend=DockerBackend())
 
-Logging
--------
+  ...
 
-Arca logs debug information via standard Python logging. The logger is called ``arca``.
+  arca.backend.stop_containers()

docs/index.rst

@@ -29,6 +29,7 @@ Arca can also cache the results of these scripts using `dogpile.cache <https://d
    install
    settings
    tasks
+   caching
    backends
    cookbook
    reference

docs/settings.rst

@@ -1,6 +1,8 @@
 Settings
 ========
 
+.. _configuring:
+
 Configuring Arca
 ----------------