docs

docs/conf.py

@@ -62,9 +62,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = u'1.0'
+version = u'0.7'
 # The full version, including alpha/beta/rc tags.
-release = u'1.0'
+release = u'0.7.7 (staging)'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/index.rst

@@ -1,25 +1,79 @@
+====
 sosw
 ====
 
-`sosw` = Serverless Orchestrator of Serverless Workers
+**`sosw`** = Serverless Orchestrator of Serverless Workers
+
+**`sosw`** is a set of tools for orchestrating **asynchronous** invocations of AWS Lambda Functions.
+Essential components of **`sosw`** are implemented as AWS Lambda functions themselves.
+
+Workers
+-------
+The **Worker** functions themselves are **your** functions that require Orchestration. ``sosw`` package
+has multiple components and tools that we encourage you to use in your Workers to make the code DRY and stable:
+statistic aggregation, components initialization, configuration automatic assembling and more...
+
+In order to be Orchestrated by ``sosw``, the functions should be able to mark tasks they receive as `completed`
+once the job is done. If you inherit the main classes of your Lambdas from :ref:`Worker` this will be handled
+automatically. The default behaviour of Workers is not to touch the queue, but to call a :ref:`Worker Assistant`
+lambda to mark tasks as completed.
+
+Read more: :ref:`Worker`
+
+------
+
+There could be defined three different Workflows.
+
+Scheduling
+----------
+
+Scheduling is transformation of your business jobs to specific tasks for Lambda :ref:`Worker`.
+One task = one invocation of a Worker. Scheduler provides an interface for your Business Applications, other Lambdas,
+Events (e.g. Scheduled Rules) to provide the Business Job.
+
+Chunking of the Payload happens following the rules that you pre-configure.
+One of the built-in dimensions for chunking is calendar.
+
+Scheduled tasks go to a stateful queue to be invoked when the time comes.
+
+Read more: :ref:`Scheduler`
+
+
+Orchestration
+-------------
+
+The :ref:`Orchestrator` is called automatically every minute by `Scheduled Events`_.
+It evaluates the status of Workers at the current moment, the health of external metrics
+the Workers may depend on (e.g CPU lod of some DB, IOPS, etc.) and invokes the appropriate
+amount of new parallel invocations.
+
+Read more: :ref:`Orchestrator`
+
+Scavenger
+---------
+
+The :ref:`Scavenger` is called automatically every minute by `Scheduled Events`_.
+It collects the tasks marked as ``completed`` by the Workers and archives them.
+
+If the task did not successfully accomplish it tries to re-invoke it with configurable exponential delay.
+In case the task completely fails after several invocations, the Scavenger marks it is `dead` and removes
+from the queue to avoid infinite retries. In this case some external alarm system: SNS or Lambda
+may be triggered as well.
 
-`sosw` is a set of tools for orchestrating **asynchronous** invocations of AWS Lambda Workers.
+Read more: :ref:`Scavenger`
 
-`sosw` requires you to implement/deploy several Lambda functions (Essentials) using the following core classes:
+Installation
+------------
+**`sosw`** requires you to implement/deploy several Lambda functions (Essentials) using the following core classes:
 
 .. toctree::
    :titlesonly:
 
-   installation
    orchestrator
    scheduler
    scavenger
-   worker_assistant
 
-The **Worker** functions themselves are expected to call the WorkerAssistant when completed the task
-from each invocation. If you inherit the :ref:`Worker` class in your function the ``__call__`` method
-does that automatically. And there are several other common features that Worker class provides
-(statistic aggregator, components initialization, configuration automatic assembling and more...)
+Optionally you  may deploy the :ref:`Worker Assistant`.
 
 Another deployment requirement is to create several `DynamoDB` tables:
 
@@ -30,29 +84,38 @@ Another deployment requirement is to create several `DynamoDB` tables:
 | You can find the Cloudformation template for the databases in `the example`_.
 | If you are not familiar with CloudFormation, we highly recommend at least learning the basics from `the tutorial`_.
 
-.. _the example: https://raw.githubusercontent.com/bimpression/sosw/docme/docs/yaml/sosw-shared-dynamodb.yaml
-.. _the tutorial: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/GettingStarted.Walkthrough.html
+
+Other
+-----
 
 .. toctree::
    :titlesonly:
    :caption: Contents:
 
-   contribution
+   installation
 
+   contribution
+   pycon
    convention
+
    components
    managers
 
-   processor
+   worker
+   worker_assistant
    orchestrator
    scheduler
    scavenger
-   worker
-
+   processor
 
 Indices and tables
 ==================
 
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
+
+.. _the example: https://raw.githubusercontent.com/bimpression/sosw/docme/docs/yaml/sosw-shared-dynamodb.yaml
+.. _the tutorial: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/GettingStarted.Walkthrough.html
+.. _Scheduled Events: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
+

docs/orchestrator.rst

@@ -1,3 +1,5 @@
+.. _Orchestrator:
+
 Orchestrator
 ------------
 

docs/scavenger.rst

@@ -1,3 +1,5 @@
+.. _Scavenger:
+
 Scavenger
 ---------
 

docs/worker_assistant.rst

@@ -1,3 +1,5 @@
+.. _Worker Assistant:
+
 Worker Assitant
 ---------------
 

sosw/scheduler.py

@@ -142,7 +142,7 @@ def parse_job_to_file(self, job: Dict):
     #     for task in data:
     #         self.task_client.create_task(labourer=labourer, payload=task)
 
-    def validate_list_of_vals(self, data: Union[List, Set, Tuple, Dict]) -> List:
+    def validate_list_of_vals(self, data: Union[list, set, tuple, Dict]) -> list:
         """
         Supported resulting values: str, int, float.