initial setup process

docs/installation.rst

@@ -4,9 +4,14 @@ Installation
 Steps
 -----
 
+.. toctree::
+   :numbered:
+
 #. Setup AWS Account
-#. Provision AWS DynamoDB Tables
-#. Provision Lambda Functions for
+#. Provision Required AWS Resources
+#. Provision Lambda Functions for Essentials
+#. Upload Essentials Configurations
+#. Create Scheduled Rules
 
 - Scheduler
 - Orchestrator
@@ -17,18 +22,20 @@ Steps
 Setup AWS Account
 -----------------
 
-As an AWS Lambda Serverless implementation deployment should be done in an AWS account. To setup a new account, follow the `AWS Documentation <https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/>`_
+As an AWS Lambda Serverless implementation deployment should be done in an AWS account. To setup a new account, follow
+the `AWS Documentation <https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/>`_
 
-Provision AWS DynamoDB Tables
-------------------------------
+Provision Required AWS Resources
+--------------------------------
 
 There are three tables required to run SOSW
 
 - close_tasks
 - retry_tasks
 - tasks
 
-These can be setup with the provided example :download:`CloudFormation template </yaml/sosw-shared-dynamodb.yaml>` easily and includes both a testing set of tables along with a production set.
+These can be setup with the provided example :download:`CloudFormation template </yaml/sosw-shared-dynamodb.yaml>`
+easily and includes both a testing set of tables along with a production set.
 
 The following Guide assumes that you are running these comands from some machine using either Key or Role
 with permissions to control IAM, Lambda, CloudWatch, DynamoDB (and probably something else will come).
@@ -55,15 +62,46 @@ The following commands are tested on a fresh EC2 instance running on default Ama
    # Creating AWS CloudFormation stacks with required resources.
 
    # DynamoDB tables
-   aws cloudformation create-stack --stack-name=sosw-development-dynamodb-tables --template-body=file://docs/yaml/sosw-shared-dynamodb.yaml
+   aws cloudformation create-stack --stack-name=sosw-dev-dynamodb-tables \
+   --template-body=file://docs/yaml/sosw-shared-dynamodb.yaml
 
    # A bucket for artifacts
-   aws cloudformation create-stack --stack-name=autotest-bucket --template-body=file://docs/yaml/autotest-bucket.yaml
+   aws cloudformation create-stack --stack-name=sosw-dev-s3-bucket \
+   --template-body=file://docs/yaml/sosw-s3-bucket.yaml
+
+
+| Now take a break and wait for these resourced to be created.
+| You may enjoy the changes in CloudFormation GUI or make some coffee.
+
+
+Provision Lambda Functions for Essentials
+-----------------------------------------
+
+In this tutorial we use AWS SAM for provisioning Lambdas.
+
+Unfortunately the tutorial is not yet ready, but the result should have four Lambdas all importing ``sosw`` from PyPI.
+Example code for Orchestrator is in :download:`/sam/orchestrator/app.py`.
+The only dependency in requirements.txt for SAM is ``sosw`` package.
+
+
+Upload Essentials Configurations
+--------------------------------
+
+sosw-managed Lambdas will automatically try to read their configuration from the DynamoDB table ``config``.
+Each Lambda looks for the document with hash_key ``config_name = 'LAMBDA_NAME_config'``.
+e.g. ``'sosw_orchestrator_config'``
+
+The ``config_value`` should contain JSON-ified dictionary that will be recursively merged to the ``DEFAULT_CONFIG``
+of each Lambda.
 
+Please take your time to read more about :ref:`Config` and find the examples in :ref:`Orchestrator`,
+:ref:`Scavenger`, :ref:`Scheduler`., etc.
 
-   # Sosw Essentials (AWS Lambdas)
-   # Each Lambda has it's own role.
 
-   aws cloudformation create-stack --stack-name=sosw-development-orchestrator --template-body=file://docs/yaml/sosw-orchestrator.yaml --capabilities CAPABILITY_NAMED_IAM
+Create Scheduled Rules
+----------------------
 
+The usual implementation expects the ``Orchestrator`` and ``Scavenger`` to run every minute, while ``Scheduler``
+and ``WorkerAssistant`` are executed per request. ``Scheduler`` may have any number of cronned Business Tasks with any
+desired periodicity of course.
 

docs/sam/orchestrator/app.py

@@ -0,0 +1,18 @@
+"""Sosw Essential Orchestrator
+"""
+
+import logging
+
+from sosw.orchestrator import Orchestrator
+from sosw.app import LambdaGlobals, get_lambda_handler
+
+logger = logging.getLogger()
+logger.setLevel(logging.INFO)
+
+
+class Essential(Orchestrator):
+    pass
+
+
+global_vars = LambdaGlobals()
+lambda_handler = get_lambda_handler(Essential, global_vars)

docs/sam/orchestrator/requirements.txt

@@ -0,0 +1 @@
+sosw

docs/yaml/scheduled_rules.yaml

@@ -0,0 +1,20 @@
+AWSTemplateFormatVersion: "2010-09-09"
+Description: "CloudWatch ScheduledEvents for Orchestrator and Scavenger"
+
+Resources:
+  ScheduledRuleSoswOrchestrator:
+    Type: "AWS::Events::Rule"
+    Properties:
+      Description: "Scheduler for sosw_orchestrator /1 min"
+      ScheduleExpression: "rate(1 minute)"
+      State: "ENABLED"
+      Targets:
+        -
+          Arn:
+            - Fn::ImportValue: "sosw-lambda-orchestrator"
+          Id: "SoswOrchestrator"
+
+
+# Each Lambda to be triggered by ScheduledRules should provide permissions for that.
+# These Permissions are usually in some generic Policy, but for this example they are directly
+# in the YAML templates of functions.

docs/yaml/sosw-orchestrator.yaml

@@ -12,11 +12,6 @@ Parameters:
     Type: String
     Default: 'config'
 
-  PackagesSourceS3:
-    Description: "S3 bucket used for storing Lambda packages."
-    Type: String
-    Default: 'autotest-bucket'
-
 Resources:
 
   LambdaSoswOrchestratorRole:
@@ -48,15 +43,14 @@ Resources:
           - Effect: "Allow"
             Action: "dynamodb:*"
             Resource:
+#              - !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}/table/${Fn::ImportValue: 'sosw-ddb-tasks'}/*"
               - Fn::Join:
                 - ':'
-                - - "arn:aws:dynamodb"
-                  - !Ref AWS::Region
-                  - !Ref AWS::AccountId
+                - - !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}"
                   - Fn::Join:
                     - '/'
                     - - "table"
-                      - Fn::ImportValue: "ddb-sosw-tasks"
+                      - Fn::ImportValue: "sosw-ddb-tasks"
                       - "*"
               - Fn::Join:
                 - ':'
@@ -66,7 +60,7 @@ Resources:
                   - Fn::Join:
                     - '/'
                     - - "table"
-                      - Fn::ImportValue: "ddb-sosw-tasks"
+                      - Fn::ImportValue: "sosw-ddb-tasks"
 
 # Closed tasks table and indexes
           - Effect: "Allow"
@@ -84,7 +78,7 @@ Resources:
                   - Fn::Join:
                     - '/'
                     - - "table"
-                      - Fn::ImportValue: "ddb-sosw-tasks-closed"
+                      - Fn::ImportValue: "sosw-ddb-tasks-closed"
                       - "index"
                       - "*"
               - Fn::Join:
@@ -95,7 +89,7 @@ Resources:
                   - Fn::Join:
                     - '/'
                     - - "table"
-                      - Fn::ImportValue: "ddb-sosw-tasks-closed"
+                      - Fn::ImportValue: "sosw-ddb-tasks-closed"
 
           - Effect: "Allow"
             Action: "lambda:InvokeFunction"
@@ -162,23 +156,34 @@ Resources:
 
           - Effect: "Allow"
             Action: "s3:*"
-            Resource: !Sub "arn:aws:s3:::${PackagesSourceS3}/sosw/*"
+            Resource:
+              - Fn::Join:
+                  - ':'
+                  - - "arn:aws:s3::"
+                    - Fn::Join:
+                      - '/'
+                      - - Fn::ImportValue: 'sosw-s3-bucket'
+                        - "sosw/*"
 
       RoleName: "lambda_sosw_orchestrator"
 
+
+#################################
+# Lambda Function for Essential #
+#################################
   LambdaSoswOrchestrator:
     Type: "AWS::Lambda::Function"
     Properties:
       Code:
-        S3Bucket: !Ref PackagesSourceS3
+        S3Bucket: !Sub "sosw-s3-${AWS::AccountId}"
         S3Key: "lambdas/sosw_orchestrator.zip"
       Description: "ABS. CloudFormation managed sosw Orchestrator."
       FunctionName: "sosw_orchestrator"
       Handler: "app.lambda_handler"
       MemorySize: 256
       Role: !GetAtt LambdaSoswOrchestratorRole.Arn
       Runtime: "python3.6"
-      Timeout: 300
+      Timeout: 60
 
       Tags:
        -
@@ -189,14 +194,27 @@ Resources:
          Value: 'empty'
 
 
-################################################
-## Permissions for CloudWatch ScheduledEvents ##
-## The actual Rules are configured in Julie   ##
-################################################
+##############################################
+# Permissions for CloudWatch ScheduledEvents #
+# The actual Rules are configured in the     #
+# scheduled-rules.yaml template.             #
+##############################################
   PermissionForEventsToInvokeLambdaSoswOrchestrator:
     Type: "AWS::Lambda::Permission"
     Properties:
       FunctionName:
         Ref: "LambdaSoswOrchestrator"
       Action: "lambda:InvokeFunction"
       Principal: "events.amazonaws.com"
+
+
+#############################################
+# Export Values to CloudFormation Namespace #
+#############################################
+Outputs:
+
+  LambdaSoswOrchestrator:
+    Description: "Sosw Orchestrator Essential"
+    Value: !Ref LambdaSoswOrchestrator
+    Export:
+      Name: "sosw-lambda-orchestrator"

docs/yaml/autotest-bucket.yaml → docs/yaml/sosw-s3-bucket.yaml

@@ -1,25 +1,25 @@
 AWSTemplateFormatVersion: "2010-09-09"
-Description: "autotest-bucket"
+Description: "sosw-s3-bucket"
 
 
 Parameters:
 
-  AutotestBucketName:
+  SoswBucketName:
     Description: "S3 bucket used for storing Lambda packages."
     Type: String
-    Default: 'autotest-bucket'
+    Default: 'sosw-s3'
 
 
 Resources:
-  AutotestBucket:
+  SoswBucket:
      Type: AWS::S3::Bucket
      Properties:
-       BucketName: !Sub "${AutotestBucketName}"
+       BucketName: !Sub "${SoswBucketName}-${AWS::AccountId}"
 
 
 Outputs:
-  AutotestBucket:
+  SoswBucket:
     Description: "Sample Bucket for Packages and Other Artifacts"
-    Value: !Ref AutotestBucket
+    Value: !Ref SoswBucket
     Export:
-      Name: "autotest-bucket"
+      Name: "sosw-s3-bucket"

docs/yaml/sosw-shared-dynamodb.yaml

@@ -367,29 +367,31 @@ Resources:
           Key: 'Environment'
           Value: 'dev'
 
-
+#############################################
+# Export Values to CloudFormation Namespace #
+#############################################
 Outputs:
 
   SoswTasksDynamoTableName:
     Description: "Sosw Tasks main table of unfinished tasks"
     Value: !Ref SoswTasksDynamoTable
     Export:
-      Name: "ddb-sosw-tasks"
+      Name: "sosw-ddb-tasks"
 
   SoswConfigDynamoTableName:
     Description: "Sosw Config Table"
     Value: !Ref SoswConfigDynamoTable
     Export:
-      Name: "ddb-sosw-config"
+      Name: "sosw-ddb-config"
 
   SoswClosedTasksDynamoTableName:
       Description: "Sosw Tasks archived table"
       Value: !Ref SoswClosedTasksDynamoTable
       Export:
-        Name: "ddb-sosw-tasks-closed"
+        Name: "sosw-ddb-tasks-closed"
 
   SoswRetryTasksDynamoTableName:
       Description: "Sosw Tasks retry table"
       Value: !Ref SoswRetryTasksDynamoTable
       Export:
-        Name: "ddb-sosw-tasks-retry"
+        Name: "sosw-ddb-tasks-retry"