Skip to content
Plugin to use S3 Batch with Serverless
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
examples/lambda-comprehend Add example May 4, 2019
.gitignore Initial commit May 4, 2019
README.md
index.js Initial commit May 4, 2019
package-lock.json Initial commit May 4, 2019
package.json Initial commit May 4, 2019
s3-batch-execution-role.json Initial commit May 4, 2019

README.md

Serverless S3 Batch plugin

The serverless-s3-batch plugin is designed to make it easy to work with S3 Batch operations.

If you're running an S3 Batch operation that invokes a Lambda function, you may be using the Serverless Framework to deploy your function anyway. Using the serverless-s3-batch plugin also assists with:

  • Managing the IAM role for your S3 Batch job

  • Launching your S3 Batch job

For a deep dive and walkthrough on S3 Batch and this plugin, check out this blog post.

Contents:

Installation

Install using Serverless plugin manager

serverless plugin install --name serverless-s3-batch

Install using npm

Install the module using npm:

npm install serverless-s3-batch --save-dev

Add serverless-s3-batch to the plugin list of your serverless.yml file:

plugins:
  - serverless-s3-batch

Quickstart

  1. Configure a function and basic usage of the plugin your serverless.yml:

    service: s3-batch
    
    plugins:
      - serverless-s3-batch
    
    custom:
      s3batch:
        manifest: s3://s3-batch-example/manifest.txt
        report: s3://s3-batch-example/reports
        operation: detectSentiment
    
    provider:
      name: aws
      runtime: python3.7
      stage: dev
      region: us-east-1
    
    functions:
      detectSentiment:
        handler: handler.detect_sentiment
  2. Deploy your service

    $ sls deploy
    

    This will deploy your function and create the IAM role to be used for your S3 Batch job.

  3. Create your S3 Batch job

    $ sls s3batch create
    
    Serverless: S3 Batch Job created. Job Id: 83e47ce1-0440-4b6c-b36b-284071fafe46
    Serverless:
    Serverless: View in browser: https://console.aws.amazon.com/s3/jobs/83e47ce1-0440-4b6c-b36b-284071fafe46
    

💥

Configuration

All configuration goes in the s3batch property in the custom block.

Example:

# serverless.yml
custom:
  s3batch:
    manifest: ...
    ...

Manifest

Shorthand version:

custom:
  s3batch:
    manifest: s3://s3-batch-example/manifest.txt

Advanced options:

custom:
  s3batch:
    manifest:
      format: csv # "csv" or "inventory". Default is "csv".
      location: s3://s3-batch-example/manifest.txt
      etag: 'ea42026f5e7aaa0addb4f0bb4131d4fb' # Optional. The plugin will fetch the latest ETag if you don't provide it.

Operation

Shorthand version:

custom:
  s3batch:
    operation: myFunc

The function name must match the name of a function in your serverless.yml and the function must be deployed.

Advanced options:

custom:
  s3batch:
    operation: 
      function: myFunc

I'd like to add support for other operations but haven't yet.

Report

Shorthand version:

custom:
  s3batch:
    report: s3://s3-batch-example/reports

Advanced options:

custom:
  s3batch:
    report:
      location: s3://s3-batch-example/reports
      enabled: true # true or false. Defaults to true if a location is provided.
      scope: all # "all" or "failed". Defaults to "all"
      

Role ARN

The plugin creates a role for use with your S3 Batch job.

By default, it creates a role named <service>-<stage>-<region>-s3BatchRole. You can specify your own name if you want:

custom:
  s3batch:
    role:
      name: myS3BatchRoleName

The role configures a policy with the following IAM statement:

{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:GetObjectVersion",
    "s3:PutObject",
    "lambda:InvokeFunction"
  ],
  "Resource": "*"
}

This will allow your role to read a manifest, invoke a Lambda function for the operation, and write the report at the end.

If you'd like to customize the IAM statement, you may do so with the following:

custom:
  s3batch:
    role:
      iamRoleStatements:
        - Effect: Allow
        - Action:
        		- s3:GetObject
        - Resource: "arn:aws:s3:::my_bucket"

Note that this will entirely replace the default IAM statement, so make sure you have all permissions you need for your job.

To disable IAM creation behavior and bring your own role, provide a Role ARN in the configuration:

custom:
  s3batch:
    role:
      arn: arn:aws:iam::123456789012:role/S3BatchRole

Future improvements:

  • Ability to run any S3 Batch operation, not just Lambda function operation.
You can’t perform that action at this time.