Table of Contents

1. Introduction
2. Prerequisites
3. Installation
4. Unit tests
5. Integration tests
6. Usage
7. Roadmap
8. Contributing
9. License
10. Acknowledgements

Introduction

Barriers and Latches are synchronisation mechanisms that enable a dependent thread to wait upon independent events.

• C++: std::barrier, std::latch
• Java: CyclicBarrier, CountDownLatch

Lambda functions depend on independent events too.

This project implements generic Event Barriers for Lambda functions. The Lambda function waits until all independent events have arrived. AWS S3 is used for state management.

Along the way, we will implement Lambda function unit tests, integration tests, and load tests for over 100 Event Barriers.

Prerequisites

Python, an AWS account, and the AWS CLI are needed to run this project.

These AWS services are configured and deployed from the command line

• AWS Lambda
• AWS S3
• AWS IAM
• AWS CloudWatch

Installation

1. Clone the repo

   git clone https://github.com/press0/aws-eventbarrier.git

2. create a virtual environment

   python -m venv venv

3. install requirements

   python -m pip install -r requirements.txt

4. define event barriers in a configuration file.

   {
        "event-barrier-0": [
            "0/0",
            "0/1"
        ],
        "event-barrier-1": [
            "1/0",
            "1/1"
        ],
        "event-barrier-2": [
            "2/0",
            "2/1",
            "2/2"
        ]
   }
   

5. build the lambda function zip file

   zip function.zip eventbarrier.py config/eventbarrier.json 

6. create the lambda function. Replace the 12 hash characters with your AWS account number.

   aws lambda create-function --function-name eventbarrier \
         --runtime python3.8 \
         --zip-file fileb://function.zip \
         --handler eventbarrier.lambda_handler \
         --role arn:aws:iam::############:role/eventbarrier 

7. update lambda function code as needed

   aws lambda update-function-code \
         --function-name eventbarrier \
         --zip-file fileb://function.zip

8. create an S3 bucket and a prefix

   aws s3 rb s3://eventbarrier
   
   

9. create an IAM policy with minimum required permissions

   {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:s3:::eventbarrier/*",
                "arn:aws:logs:*:*:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::eventbarrier"
        }
    ]
   }

10. create an IAM role

    aws iam create-role \
       --role-name eventbarrier 
       --assume-role-policy-document file://config/eventbarrier-policy.json
    

11. create an event notification in one of two ways:

    • manually; s3 console > bucket properties tab > create event notification
    • automated; aws cli

    aws s3api put-bucket-notification-configuration \
        --bucket eventbarrier 
        --notification-configuration file://config/notification.json

with notification.json like:

    {
      "TopicConfigurations": [
         {
            "TopicArn": "arn:aws:sns:us-west-2:123456789012:s3-notification-topic",
            "Events": [
               "s3:ObjectCreated:*"
            ]
         }
      ]
   }

Unit tests

Verify lambda function logic and validate your custom event barrier configuration in eventbarrier.json

python eventbarrier_test.py

Integration tests

Upload files to the respective prefix of each event barrier. AWS CloudWatch verifies the event barrier conditions.

Usage

Roadmap

See the open issues for a list of proposed features (and known issues).

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Acknowledgements