Docs refactoring. 100% healthy build.

sosw · Jul 19, 2019 · aa99ad2 · aa99ad2
1 parent f2d5bf3
commit aa99ad2
Show file tree

Hide file tree

Showing 26 changed files with 99 additions and 92 deletions.
diff --git a/Pipfile b/Pipfile
@@ -5,6 +5,7 @@ verify_ssl = true
 
 [dev-packages]
 sphinx = "*"
+sphinx_rtd_theme = "*"
 pytest = "*"
 
 [packages]

diff --git a/Pipfile.lock b/Pipfile.lock
diff --git a/README.md b/README.md
@@ -1,13 +1,14 @@
-# Serverless Orchestrator of Serverless Workers (SOSW)
+# Serverless Orchestrator of Serverless Workers (``sosw``)
 [![Build Status](https://travis-ci.org/bimpression/sosw.svg?branch=master)](https://travis-ci.org/bimpression/sosw)
+[![Documentation Status](https://readthedocs.org/projects/sosw/badge/?version=latest)](https://docs.sosw.app/en/latest/?badge=latest)
 
 **sosw** is a set of tools for orchestrating asynchronous invocations of AWS Lambda Workers.
 
 ## Documentation
 [Read The Docs: sosw.readthedocs.io](https://sosw.readthedocs.io/en/latest/)
 
 ## Essential Workflows
-![Essential sosw Workflow Schema](https://raw.githubusercontent.com/bimpression/sosw/docme/docs/images/simple-sosw.png)
+![Essential sosw Workflow Schema](https://raw.githubusercontent.com/bimpression/sosw/docme/docs/_static/images/simple-sosw.png)
 
 ## Dependencies
 - Python 3.6, 3.7

diff --git a/docs/images/greenfield-timeline.png → docs/_static/images/greenfield-timeline.png b/docs/images/greenfield-timeline.png → docs/_static/images/greenfield-timeline.png
diff --git a/docs/images/orchestrator-invocation.png → ...static/images/orchestrator-invocation.png b/docs/images/orchestrator-invocation.png → ...static/images/orchestrator-invocation.png
diff --git a/docs/images/scheduler.png → docs/_static/images/scheduler.png b/docs/images/scheduler.png → docs/_static/images/scheduler.png
diff --git a/docs/images/simple-sosw.png → docs/_static/images/simple-sosw.png b/docs/images/simple-sosw.png → docs/_static/images/simple-sosw.png
diff --git a/docs/components.rst b/docs/components.rst
diff --git a/docs/components/config.rst b/docs/components/config.rst
@@ -1,3 +1,5 @@
+.. _Config_Sourse:
+
 Config Source
 -------------
 

diff --git a/docs/components/index.rst b/docs/components/index.rst
@@ -0,0 +1,14 @@
+==========
+Components
+==========
+
+.. toctree::
+   :titlesonly:
+   :caption: Components:
+
+   config
+   dynamo_db
+   helpers
+   siblings
+   sns
+   tasks_api_client_for_workers
diff --git a/docs/convention.rst → docs/contribution/convention.rst b/docs/convention.rst → docs/contribution/convention.rst
diff --git a/docs/contribution.rst → docs/contribution/index.rst b/docs/contribution.rst → docs/contribution/index.rst
@@ -4,6 +4,13 @@
 Contribution Guidelines
 =======================
 
+
+..  toctree::
+    :caption: See Also
+
+    Documentation Convention <convention>
+    Sprinting PyCon US 2019 <pycon-us-2019>
+
 Release cycle
 -------------
 
@@ -28,17 +35,17 @@ Follow the PEP8_ but both classes and functions are padded with 2 empty lines.
 Initialization
 --------------
 
-* Fork the repository: https://github.com/bimpression/sosw_
+* Fork the repository: `https://github.com/bimpression/sosw`_
 
 * Register Account in AWS: `SignUp`_
 
 * Run `pipenv sync --dev` to setup your virtual environment and download the required dependencies
 
-* Create DynamoDB Tables:
-  * You can find the CloudFormation template for the databases in `the example`_.
-  * If you are not familiar with CloudFormation, we highly recommend at least learning the basics from `the tutorial`_.
+* If you are not familiar with CloudFormation, we highly recommend at least learning the basics from `the tutorial`_.
+
+* Follow the :ref:`Installation Guidelines` to setup your environment.
 
-* Create Sandbox Lambda with Scheduler
+* Create Sandbox Lambda.
 
 * Play with it.
 

diff --git a/docs/pycon.rst → docs/contribution/pycon-us-2019.rst b/docs/pycon.rst → docs/contribution/pycon-us-2019.rst
diff --git a/docs/essentials/index.rst b/docs/essentials/index.rst
@@ -0,0 +1,14 @@
+========
+Managers
+========
+
+.. toctree::
+   :titlesonly:
+   :caption: Managers:
+
+   worker
+   orchestrator
+   scheduler
+   scavenger
+   worker_assistant
+   processor
diff --git a/docs/orchestrator.rst → docs/essentials/orchestrator.rst b/docs/orchestrator.rst → docs/essentials/orchestrator.rst
@@ -10,7 +10,7 @@ supported ways in :ref:`Config <components-config>`
 
 The following diagram represents the basic Task Workflow initiated by the Orchestrator.
 
-.. figure:: images/orchestrator-invocation.png
+.. figure:: ../_static/images/orchestrator-invocation.png
    :alt: Invocation Process
    :align: center
 

diff --git a/docs/processor.rst → docs/essentials/processor.rst b/docs/processor.rst → docs/essentials/processor.rst
diff --git a/docs/scavenger.rst → docs/essentials/scavenger.rst b/docs/scavenger.rst → docs/essentials/scavenger.rst
diff --git a/docs/scheduler.rst → docs/essentials/scheduler.rst b/docs/scheduler.rst → docs/essentials/scheduler.rst
@@ -3,11 +3,11 @@
 Scheduler
 ---------
 
-Scheduler is the public interface of SOSW for any applications who want to invoke some orchestrated Lambdas.
+Scheduler is the public interface of ``sosw`` for any applications who want to invoke some orchestrated Lambdas.
 It's main role is to transform some business job to the actual payload of Lambda invocations. It respects the
 configurable rules for chunking specific for different workers.
 
-.. figure:: images/scheduler.png
+.. figure:: ../_static/images/scheduler.png
    :alt: sosw Scheduler Workflow
    :align: center
 

diff --git a/docs/worker.rst → docs/essentials/worker.rst b/docs/worker.rst → docs/essentials/worker.rst
diff --git a/docs/worker_assistant.rst → docs/essentials/worker_assistant.rst b/docs/worker_assistant.rst → docs/essentials/worker_assistant.rst
diff --git a/docs/greenfield.rst b/docs/greenfield.rst
@@ -20,7 +20,7 @@ Possible states:
 
 The following diagram represents different states.
 
-.. figure:: images/greenfield-timeline.png
+.. figure:: _static/images/greenfield-timeline.png
    :alt: Greenfield Timeline
    :align: center
 

diff --git a/docs/index.rst b/docs/index.rst
@@ -10,7 +10,7 @@ Essential components of **`sosw`** are implemented as AWS Lambda functions thems
 Essential Workflow Schema
 -------------------------
 
-.. figure:: images/simple-sosw.png
+.. figure:: _static/images/simple-sosw.png
    :alt: Simple sosw workflows
    :align: center
 
@@ -64,59 +64,40 @@ The :ref:`Scavenger` is called automatically every minute by `Scheduled Events`_
 It collects the tasks marked as ``completed`` by the Workers and archives them.
 
 If the task did not successfully accomplish it tries to re-invoke it with configurable exponential delay.
-In case the task completely fails after several invocations, the Scavenger marks it is `dead` and removes
+In case the task completely fails after several invocations, the Scavenger marks it is ``dead`` and removes
 from the queue to avoid infinite retries. In this case some external alarm system: SNS or Lambda
 may be triggered as well.
 
 Read more: :ref:`Scavenger`
 
 Installation
 ------------
-**`sosw`** requires you to implement/deploy several Lambda functions (Essentials) using the following core classes:
 
-.. toctree::
-   :titlesonly:
-
-   orchestrator
-   scheduler
-   scavenger
+:ref:`Installation Guidelines`
 
-Optionally you  may deploy the :ref:`Worker Assistant`.
+**`sosw`** requires you to implement/deploy several Lambda functions (Essentials) using the appropriate core classes.
+The deployment is described in details, but assumes that you are familiar with basic AWS Serverless products.
 
-Another deployment requirement is to create several `DynamoDB` tables:
-
-- ``sosw_tasks``
-- ``sosw_retry_tasks``
-- ``sosw_closed_tasks``
-
-The more detailed guide for initial setup can be found in the :ref:`Installation`.
+Another deployment requirement is to create several `DynamoDB` tables.
 
 | You can find the Cloudformation template for the databases in `the example`_.
 | If you are not familiar with CloudFormation, we highly recommend at least learning the basics from `the tutorial`_.
 
+Once again, the detailed guide for initial setup can be found in the :ref:`Installation Guidelines`.
 
-Other
------
 
 .. toctree::
    :titlesonly:
    :caption: Contents:
+   :maxdepth: 2
 
    installation
+   essentials/index
+   components/index
+   managers/index
 
-   worker
-   worker_assistant
-   orchestrator
-   scheduler
-   scavenger
-   processor
-
-   components
-   managers
+   contribution/index
 
-   contribution
-   pycon
-   convention
 
 Indices and tables
 ==================

diff --git a/docs/installation.rst b/docs/installation.rst
@@ -1,29 +1,29 @@
+.. _Installation Guidelines:
+
+
+============
 Installation
 ============
 
+
 Steps
 -----
-
-.. toctree::
-:numbered:
-
-    #. Setup AWS Account
-    #. Provision Required AWS Resources
-    #. Provision Lambda Functions for Essentials
-    #. Upload Essentials Configurations
-    #. Create Scheduled Rules
+#. `Setup AWS Account`_
+#. `Provision Required AWS Resources`_
+#. `Provision Lambda Functions for Essentials`_
+#. `Upload Essentials Configurations`_
+#. `Create Scheduled Rules`_
 
 
 Setup AWS Account
 -----------------
-
 As an AWS Lambda Serverless implementation deployment should be done in an AWS account. To setup a new account, follow
 the `AWS Documentation <https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/>`_
 
+
 Provision Required AWS Resources
 --------------------------------
-
-There are three tables required to run SOSW. Nobody should touch these tables except sosw essentials.
+There are three tables required to run ``sosw``. Nobody should touch these tables except sosw essentials.
 
 - close_tasks
 - retry_tasks
@@ -34,7 +34,8 @@ AWS SSM Parameter Store has pretty low IO limits and having hundreds of parallel
 `sosw` introduces similar mechanism for accessing configurations and parameters using DynamoDB as a storage.
 All functions inheriting from core `sosw.Processor` will fetch their config automatically.
 
-These can be setup with the provided example :download:`CloudFormation template </yaml/sosw-shared-dynamodb.yaml>`
+These can be setup with the provided example
+:download:`CloudFormation template <../examples/yaml/initial/sosw-dev-shared-dynamodb.yaml>`
 easily and includes both a testing set of tables along with a production set.
 
 The following Guide assumes that you are running these comands from some machine using either Key or Role
@@ -126,17 +127,11 @@ not to panic if there are no changes.
 
 Provision Lambda Functions for Essentials
 -----------------------------------------
-
 In this tutorial we were first going to use AWS SAM for provisioning Lambdas, but eventually gave it up.
 Too many black magic is required and you eventually loose control over the Lambda. The example of deploying Essentials
 uses raw bash scripts, AWS CLI and CloudFormation templates. If you want to contribute providing examples
 with SAM - welcome. Some sandbox can be found in `examples/sam/` in the repository.
 
-Unfortunately the tutorial is not yet ready, but the result should have four Lambdas all importing ``sosw`` from PyPI.
-Example code for Orchestrator is in :download:`/sam/orchestrator/app.py`.
-The only dependency in requirements.txt for SAM is ``sosw`` package.
-
-
 .. warning:: This is still unfinished tutorial. Use wizely.
 
 Non-hipster way just manually building the package and creating CF stack with raw CloudFormation.
@@ -217,21 +212,19 @@ If you change anything in the code or simply want to redeploy the code use the f
 
 Upload Essentials Configurations
 --------------------------------
-
 sosw-managed Lambdas will automatically try to read their configuration from the DynamoDB table ``config``.
 Each Lambda looks for the document with hash_key ``config_name = 'LAMBDA_NAME_config'``.
 e.g. ``'sosw_orchestrator_config'``
 
 The ``config_value`` should contain JSON-ified dictionary that will be recursively merged to the ``DEFAULT_CONFIG``
 of each Lambda.
 
-Please take your time to read more about :ref:`Config` and find the examples in :ref:`Orchestrator`,
+Please take your time to read more about :ref:`Config Sourse<Config_Sourse>` and find the examples in :ref:`Orchestrator`,
 :ref:`Scavenger`, :ref:`Scheduler`., etc.
 
 
 Create Scheduled Rules
 ----------------------
-
 The usual implementation expects the ``Orchestrator`` and ``Scavenger`` to run every minute, while ``Scheduler``
 and ``WorkerAssistant`` are executed per request. ``Scheduler`` may have any number of cronned Business Tasks with any
 desired periodicity of course.

diff --git a/docs/managers.rst → docs/managers/index.rst b/docs/managers.rst → docs/managers/index.rst
@@ -6,7 +6,7 @@ Managers
    :titlesonly:
    :caption: Managers:
 
-   managers/task
-   managers/ecology
+   task
+   ecology
 
-   greenfield
+   ../greenfield
diff --git a/sosw/managers/task.py b/sosw/managers/task.py
@@ -29,8 +29,8 @@
 class TaskManager(Processor):
     """.. _task:
 
-    TaskManager is the core class used by most SOSW Lambdas.
-    It handles all the operations with tasks thus the configuration of this Manager is essential during your SOSW implementation.
+    TaskManager is the core class used by most ``sosw`` Lambdas.
+    It handles all the operations with tasks thus the configuration of this Manager is essential during your ``sosw`` implementation.
 
     The default version of TaskManager works with DynamoDB tables to store and analyze the state of Tasks.
     This could be upgraded in future versions to work with other persistent storage or DBs.