Add AwsCompileStreamEvents plugin

[pmuens]

What did you implement:

_Implementing Issue:_ #1441 and #1608

Adds support for the stream event definition (which covers DynamoDB and Kinesis streams). No need to set it up through custom resources anymore.

How did you implement it:

Introduced a new plugin which will compile the corresponding EventSourceMappings for CloudFormation. Furthermore the IamRoleStatements are updated so that the Lambda function can be called when something in the stream is happening.

How can we verify it:

First of all you need to create a new DynamoDB table and set up a new Stream for that table (In Overview --> Stream details --> Manage Stream / Kinesis stream . Copy and paste the ARN for the corresponding stream.

Next up use this serverless.yml file and paste the ARN you've just copied:

service: new-test-service

provider:
  name: aws
  runtime: nodejs4.3

functions:
  hello:
    handler: handler.hello
    events:
      - stream: arn:aws:dynamodb:us-east-1:XXXXXX:table/foo/stream/2016-10-01T21:14:36.344
      - stream:
          arn: arn:aws:dynamodb:us-east-1:XXXXXX:table/bar/stream/2016-10-01T21:14:31.803
          batchSize: 20
          startingPosition: TRIM_HORIZON
          enabled: true
      - stream: arn:aws:kinesis:us-east-1:XXXXXX:stream/baz
      - stream:
          arn: arn:aws:kinesis:us-east-1:XXXXXX:stream/qux
          batchSize: 50
          startingPosition: LATEST
          enabled: false

Todos:

• Write tests
• Write documentation
• Fix linting errors
• Make sure code coverage hasn't dropped
• Provide verification config/commands/resources
• Change ready for review message below

_Is this ready for review?:_ YES

[pmuens]

@eahefnawy would be great if you could review it

@mthenw @ac360 your feedback would be also very valuable!

[mthenw]

I'm personally not sure if it's not confusing that there is one event type (stream) for both DynamoDB and Kinesis streams. Looks like they share the same params so maybe it's OK.

Would be great if docs would also show how to create Kinesis stream via custom resources and pass ARN to event source config.

[eahefnawy]

@pmuens we shouldn't put the table name in the logical ID of the event mapping resource. I tried using a table named test-stream and it failed because it's not alpha numeric.

Other than that, it's working great. 👍 for handling the permissions as well, it's interesting that this is the first event that actually requires the lambda function to have permission via roles, not the other way around by adding a lambda:permission resource.

[pmuens]

@eahefnawy @mthenw thanks for the feedback.

I just refactored the resource logical Id name generation so that all non-alphanumerics are removed (so e.g. users-table will be userstable) and added a test for that as well.

We need those names inside of the resource logical id so that they are unique (and also easier to overwrite via custom resources). I've tried to do it with a counter variable (like it's done in other plugins) but this will break when you re-arrange or update the streams.

[eahefnawy]

@pmuens yeah you're right ... I was about to suggest removing the dynamodb label as well since they're all streams after all, but I realized that it might conflict with a kinesis stream of the same name. Thanks for clarification.

G2M from my side 👍

[pmuens]

Awesome! Thanks for the review @eahefnawy

@flomotlik could you take a look at it and merge it if it's appropriate? Thanks!

[screamish]

Is this a typo? bathSize -> batchSize ?

[pmuens]

Damn! Good catch @screamish

Just pushed an update which will fix this!

[eahefnawy]

"bathSize" 😂

[pmuens]

😹 #documentationGems

[flomotlik]

Good stuff. A few changes in the docs to make sure that users are aware that we're not actually creating the Kinesis or DynamoDB ourselves

[flomotlik]

I don't think we need this README. Its basically just a copy of the documentation so the whole file can be removed.

[flomotlik]

Mention that we do not create the DynamoDB, but it only connects to a previously created one.

[flomotlik]

Add 2 different stream configs, one with an ARN and one that uses the Ref syntax and mention that you can reference a dynamodb or kinesis stream created in the same service through Ref

[flomotlik]

Same as above, mention that you're not creating the Kinesis Stream

[pmuens]

@flomotlik just update the PR with the proposed changes.

Unfortunately we cannot use Ref as it will only return the name of the stream for a Kinesis stream so I've used Fn::GetAtt to get the ARN.

Furthermore I've added a commented out example to all the serverless.yml files.