Merge pull request #207 from sosw/0_7_33

v0.7.33 - release - dynamo_db: get_by_scan supports scanning custom Indexes - meta_handler: Optional collector of all the events that were happening to the task while being orchestrated by sosw - scheduler: support for extra custom methods for creating a date list - essential: support shared essential config for all the sub-classes of Essential. Useful for labourers config. - worker_assistant: mark_task_as_failed. Support in both Worker & WorkerAssistant - helpers.recursive_update: merging lists of Dictionaries - Contribution guidelines update
sosw · Jul 2, 2020 · 5e7e450 · 5e7e450
2 parents 992aadf + f21eb67
commit 5e7e450
Show file tree

Hide file tree

Showing 40 changed files with 811 additions and 137 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -4,7 +4,7 @@ language: python
 python:
   - '3.6'
   - '3.7'
-  - '3.8-dev'
+  - '3.8'
 install:
   - pip install boto3
 script:

diff --git a/LICENSE b/LICENSE
@@ -1,5 +1,5 @@
 The MIT License (MIT)
-Copyright (C) 2019  sosw core contributors <info@sosw.app>:
+Copyright (C) 2020  sosw core contributors <info@sosw.app>:
     Nikolay Grishchenko
     Sophie Fogel
     Gil Halperin

diff --git a/NOTICE b/NOTICE
@@ -1,5 +1,5 @@
 The MIT License (MIT)
-Copyright (C) 2019  sosw core contributors <info@sosw.app>:
+Copyright (C) 2020  sosw core contributors <info@sosw.app>:
     Nikolay Grishchenko
     Sophie Fogel
     Gil Halperin

diff --git a/README.md b/README.md
@@ -1,30 +1,27 @@
 <img alt="sosw - Serverless Orchestrator of Serverless Workers" width="350" src="https://raw.githubusercontent.com/sosw/sosw/docme/docs/_static/images/logo/sosw_black.png">
 
 # Serverless Orchestrator of Serverless Workers
-[![Build Status](https://travis-ci.org/sosw/sosw.svg?branch=master)](https://travis-ci.org/sosw/sosw)
+[![Build Status](https://travis-ci.com/sosw/sosw.svg?branch=master)](https://travis-ci.com/sosw/sosw)
 [![Documentation Status](https://readthedocs.org/projects/sosw/badge/?version=latest)](https://docs.sosw.app/en/latest/?badge=latest)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/sosw?color=blue&label=pypi%20installs)](https://pypi.org/project/sosw/)
 [![PyPI - Licence](https://img.shields.io/pypi/l/sosw?color=blue)](https://github.com/sosw/sosw/blob/master/LICENSE)
 
 **sosw** is a set of serverless tools for orchestrating asynchronous invocations of AWS Lambda Functions (Workers).
 
----
- Please pronounce **sosw** correctly: _/ˈsɔː səʊ/_
-
 ---
 
 ## Documentation
-[https://docs.sosw.app](https://docs.sosw.app/en/latest/)
+[https://docs.sosw.app](https://docs.sosw.app/en/master/)
 
 ## Essential Workflows
 ![Essential sosw Workflow Schema](https://raw.githubusercontent.com/sosw/sosw/docme/docs/_static/images/simple-sosw.png)
 
 ## Dependencies
-- Python 3.6, 3.7
+- Python 3.6, 3.7, 3.8
 - [boto3](https://github.com/boto/boto3) (AWS SDK for Python)
 
 ## Installation
-See the [Installation Guidelines](https://docs.sosw.app/en/latest/installation.html) in the Documentation.
+See the [Installation Guidelines](https://docs.sosw.app/en/master/installation.html) in the Documentation.
 
 ## Development
 ### Getting Started
@@ -50,15 +47,23 @@ $ pytest ./sosw/test/suite_3_6_unit.py
 
 ### Contribution Guidelines
 
+The latest [Contribution Guidelines](https://docs.sosw.app/en/master/contribution/index.html) with examples are in the documentation.
+
 #### Release cycle
 - We follow both [Semantic Versioning](https://semver.org/) pattern
   and [PEP440](https://www.python.org/dev/peps/pep-0440/) recommendations where comply
 - Master branch commits (merges) are automatically packaged and published to PyPI.
 - Branches for planned staging versions follow the pattern: `X_Y_Z` (Major.Minor.Micro)
-- Make your pull requests to the latest staging branch (with highest number)
-- Latest documentation is compiled from branch `docme`.
-  It should be up to date with latest **staging** branch, not the master.
-  Make PRs with documentation change directly to `docme`.
+- Make your pull requests to the closest staging branch (with smallest after release number of either current or next Minor)
+- Make sure your branch is up to date with the branch you are making a PR to
+
+Example:
+  - Latest released version in PyPI `0.7.31`
+  - Closest staging Minor branch in sosw/sosw `0_7_33`
+  - Latest Minor staging branch in sosw/sosw `0_7_35`
+  - Closest Next Minor branch in sosw/sosw `0_9_1`
+
+Your PR should be to either `0_7_33` or `0_9_1` depending on the importance of changes. 
 
 #### Code formatting
 Follow [PEP8](https://www.python.org/dev/peps/pep-0008/), but:
@@ -77,20 +82,14 @@ Follow [PEP8](https://www.python.org/dev/peps/pep-0008/), but:
 7. Read the Documentation Convention.
 
 #### More
-See more guidelines for contribution [in the docs](https://docs.sosw.app/en/latest/contribution/index.html).
+See more guidelines for contribution [in the docs](https://docs.sosw.app/en/master/contribution/index.html).
 
 ### Building the docs
-Sphinx is used for building documentation. To build HTML documentation locally, use:
-
-```bash
-$ sphinx-build -ab html ./docs ./sosw-rtd
-```
-
-You can then use the built in Python web server to view the html version directly from `localhost` in your preferred browser.
+Sphinx is used for building documentation.
+You can build HTML documentation locally then use the built in Python web server to view the html version directly from `localhost` in your preferred browser.
 
 ```bash
-$ cd sosw-rtd
-$ python -m http.server
+$ sphinx-build -ab html ./docs ./sosw-rtd; (cd sosw-rtd && python -m http.server)
 ```
 
 ## Copyright
@@ -100,7 +99,7 @@ This document has been placed in the public domain.
     sosw - Serverless Orchestrator of Serverless Workers
 
     The MIT License (MIT)
-    Copyright (C) 2019  sosw core contributors <info@sosw.app>:
+    Copyright (C) 2020  sosw core contributors <info@sosw.app>:
         Nikolay Grishchenko
         Sophie Fogel
         Gil Halperin

diff --git a/docs/contribution/index.rst b/docs/contribution/index.rst
@@ -5,8 +5,9 @@ Contribution Guidelines
 =======================
 
 
+..  contents::
+
 ..  toctree::
-    :caption: See Also
 
     Documentation Convention <convention>
     Sprinting PyCon US 2019 <pycon-us-2019>
@@ -16,14 +17,25 @@ Release cycle
 
 - Master branch commits are automatically packaged and published to PyPI.
 
-- Branches for staging versions follow the pattern: ``X_X_X``
+- We follow both `Semantic Versioning`_ pattern and PEP440_ recommendations where comply
+
+- Branches for planned staging versions follow the pattern: ``X_Y_Z`` (Major.Minor.Micro)
+
+- Make your pull requests go to the closest staging branch (with smallest after release number of either current or next Minor)
+
+- Make sure your branch is up to date with the branch you are making a PR to
 
-- Make your pull requests to the staging branch with highest number
+Example:
 
-- Latest documentation is compiled from branch ``docme``.
-  It should be up to date with latest **staging** branch, not the master.
-  Make PRs with documentation change directly to ``docme``.
+  - Latest released version in PyPI ``0.7.31``
+  - Closest staging Minor branch in sosw/sosw ``0_7_33``
+  - Latest Minor staging branch in sosw/sosw ``0_7_35``
+  - Closest Next Minor branch in sosw/sosw ``0_9_1``
 
+Your PR should be to either ``0_7_33`` or ``0_9_1`` depending on the importance of changes.
+
+.. _`Semantic Versioning`: https://semver.org/
+.. _PEP440: https://www.python.org/dev/peps/pep-0440/
 
 Code formatting
 ---------------
@@ -51,8 +63,6 @@ Initialization
 
 * Read the :ref:`Documentation convention`
 
-
-.. _the example: https://raw.githubusercontent.com/sosw/sosw/docme/docs/yaml/sosw-shared-dynamodb.yaml
 .. _the tutorial: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/GettingStarted.Walkthrough.html
 
 Building the docs
@@ -66,3 +76,54 @@ in your preferred browser.
 .. code-block:: bash
 
  sphinx-build -ab html ./docs ./sosw-rtd; (cd sosw-rtd && python -m http.server)
+
+Pull Requests Checklist
+-----------------------
+
+Great that you are ready to contribute!
+
+* Make sure your fork is up to date with upstream
+
+..  code-block:: bash
+
+    # Clean everything
+    git reset --hard
+    git clean -fdx
+    git checkout master
+
+    # Fetch possible changes to YOUR master
+    git pull origin master
+
+    # Check if remote upstream is configured
+    git remote -v
+
+    # If missing upstream
+    git remote add upstream https://github.com/sosw/sosw
+
+    # Update your fork remote with the upstream changes
+    git pull upstream master
+    git push origin master
+
+* Make sure your code passes all the tests
+
+..  code-block:: bash
+
+    pytest sosw/test/suite_3_6_unit.py
+
+* Make sure the documentation builds correctly
+
+..  code-block:: bash
+
+    sphinx-build -ab html ./docs ./sosw-rtd; (cd sosw-rtd && python -m http.server)
+
+* Push the changes to your fork remote
+
+..  code-block:: bash
+
+    git push origin master
+
+* Make a PR to the upstream repository of sosw
+
+Some guidelines of how to do create PRs from forks can be found in `GitHub documentation`_.
+
+.. _GitHub documentation: https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork
diff --git a/docs/index.rst b/docs/index.rst
@@ -4,9 +4,9 @@
 Serverless Orchestrator of Serverless Workers
 =============================================
 
-..  image:: https://travis-ci.org/sosw/sosw.svg?branch=master
+..  image:: https://travis-ci.com/sosw/sosw.svg?branch=master
     :alt: Travis - Build
-    :target: https://travis-ci.org/sosw/sosw
+    :target: https://travis-ci.com/sosw/sosw
 .. image:: https://readthedocs.org/projects/sosw/badge/?version=latest
     :alt: Read the Docs - Build
     :target: https://docs.sosw.app/en/latest/?badge=latest

diff --git a/docs/managers/index.rst b/docs/managers/index.rst
@@ -8,5 +8,6 @@ Managers
 
    task
    ecology
+   meta_handler
 
    ../greenfield
diff --git a/docs/managers/meta_handler.rst b/docs/managers/meta_handler.rst
@@ -0,0 +1,13 @@
+MetaHandler
+-----------
+
+Meta Handler is an optional collector of all the events that were happening to the task while being orchestrated by
+``sosw``. In case you create a DynamoDB table ``sosw_tasks_meta`` in your environment, ``sosw`` will start collecting
+the meta data  automatically. The required schema for the table is described in
+`sosw/examples/yaml/initial/sosw-dev-shared-dynamodb.yaml
+<https://raw.githubusercontent.com/sosw/sosw/master/examples/yaml/initial/sosw-dev-shared-dynamodb.yaml>`_ .
+
+The setup script will create the table by default.
+
+.. automodule:: sosw.managers.meta_handler
+   :members:
diff --git a/examples/yaml/initial/sosw-dev-shared-dynamodb.yaml b/examples/yaml/initial/sosw-dev-shared-dynamodb.yaml
@@ -1,6 +1,6 @@
 ###################################################################################
 # DynamoDB Tables                                                                 #
-# This template shall create 3 main tables for `sosw` and the `autotest_` copies  #
+# This template shall create 4 main tables for `sosw` and the `autotest_` copies  #
 # for them in case you are running integration tests.                             #
 ###################################################################################
 
@@ -284,6 +284,34 @@ Resources:
 #          Key: 'Environment'
 #          Value: 'dev'
 
+  SoswTasksMetaDynamoTable:
+    Type: "AWS::DynamoDB::Table"
+    Properties:
+      TableName: "sosw_tasks_meta"
+      AttributeDefinitions:
+        -
+          AttributeName: 'task_id'
+          AttributeType: 'S'
+        -
+          AttributeName: 'created_at'
+          AttributeType: 'N'
+      KeySchema:
+        -
+          AttributeName: 'task_id'
+          KeyType: "HASH"
+        -
+          AttributeName: 'created_at'
+          KeyType: "RANGE"
+      ProvisionedThroughput:
+        ReadCapacityUnits: 3
+        WriteCapacityUnits: 2
+
+      Tags:
+        -
+          Key: 'Environment'
+          Value: 'sandbox'
+
+
   # Configuration Tables
   SoswConfigDynamoTable:
     Type: "AWS::DynamoDB::Table"
@@ -417,6 +445,12 @@ Outputs:
       Export:
         Name: "sosw-ddb-tasks-closed"
 
+  SoswTasksMetaDynamoTableName:
+      Description: "Sosw Tasks Meta Data table"
+      Value: !Ref SoswTasksMetaDynamoTable
+      Export:
+        Name: "sosw-ddb-tasks-meta"
+
   SoswRetryTasksDynamoTableName:
       Description: "Sosw Tasks retry table"
       Value: !Ref SoswRetryTasksDynamoTable

diff --git a/setup.py b/setup.py
@@ -5,7 +5,7 @@
     long_description = f.read()
 
 setup(name='sosw',
-      version='0.7.31',
+      version='0.7.33',
       description='Serverless Orchestrator of Serverless Workers',
       long_description=long_description,
       long_description_content_type="text/markdown",
@@ -19,6 +19,7 @@
           'License :: OSI Approved :: MIT License',
           'Programming Language :: Python :: 3.6',
           'Programming Language :: Python :: 3.7',
+          'Programming Language :: Python :: 3.8',
           'Topic :: Software Development'
       ],
       packages=find_packages(exclude=['docs', 'test', 'examples']),