Permalink
Browse files

Update eventsource docs

  • Loading branch information...
mweagle committed Jan 31, 2019
1 parent 59aa6f9 commit 24876a56ec0d9a3de6f74f813e40aa88410868b7
@@ -13,7 +13,7 @@ Assume that we're supposed to write a simple "HelloWorld" CloudWatch event funct
* Run every *5 minutes* to provide a heartbeat notification to our alerting system via a logfile entry
* Log *EC2-related* events for later processing

# Getting Started
## Getting Started

The lambda function is relatively small:
```go
@@ -28,7 +28,7 @@ return event, nil
```
Our lambda function doesn't need to do much with the event other than log and return it.

# Sparta Integration {#spartaIntegration}
## Sparta Integration {#spartaIntegration}

With `echoCloudWatchEvent()` implemented, the next step is to integrate the **go** function with Sparta. This is done by the `appendCloudWatchEventHandler` in the SpartaApplication [application.go](https://github.com/mweagle/SpartaApplication/blob/master/application.go) source.

@@ -51,7 +51,7 @@ cloudWatchEventsPermission.Rules = make(map[string]sparta.CloudWatchEventsRule,
Our two rules will be inserted into the `Rules` map in the next steps.
## Cron Expression
### Cron Expression
Our first requirement is that the lambda function write a heartbeat to the logfile every 5 mins. This can be configured by adding a scheduled event:
```go
@@ -61,7 +61,7 @@ cloudWatchEventsPermission.Rules["Rate5Mins"] = sparta.CloudWatchEventsRule{
```
The `ScheduleExpression` value can either be a _rate_ or a _cron_ [expression](http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/ScheduledEvents.html). The map keyname is used when adding the [rule](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatchEvents.html#putRule-property) during stack provisioning.
## Event Pattern
### Event Pattern
The other requirement is that our lambda function be notified when matching EC2 events are created. To support this, we'll add a second `Rule`:
@@ -81,8 +81,7 @@ The EC2 event pattern is the **go** JSON-compatible representation of the event
Sparta does <b>NOT</b> attempt to validate either <code>ScheduleExpression</code> or <code>EventPattern</code> values prior to calling CloudFormation. Syntax errors in either value will be detected during provisioning when the Sparta CloudFormation CustomResource calls <a href="http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatchEvents.html#putRule-property">putRule</a> to add the lambda target. This error will cause the CloudFormation operation to fail. Any API errors will be logged & are viewable in the <a href="https://blogs.aws.amazon.com/application-management/post/TxPYD8JT4CB5UY/View-CloudFormation-Logs-in-the-Console">CloudFormation Logs Console</a>.
{{% /notice %}}
# Add Permission
## Add Permission
With the two rules configured, the final step is to add the `sparta.CloudWatchPermission` to our `sparta.LambdaAWSInfo` value:
@@ -117,7 +116,6 @@ func appendCloudWatchEventHandler(api *sparta.API,
}
```
# Wrapping Up
With the `lambdaFn` fully defined, we can provide it to `sparta.Main()` and deploy our service. The workflow below is shared by all CloudWatch Events-triggered lambda functions:
@@ -132,7 +130,7 @@ With the `lambdaFn` fully defined, we can provide it to `sparta.Main()` and depl
* Append the `CloudWatchEventsPermission` value to the lambda function's `Permissions` slice.
* Include the reference in the call to `sparta.Main()`.
# Other Resources
## Other Resources
* Introduction to [CloudWatch Events](https://aws.amazon.com/blogs/aws/new-cloudwatch-events-track-and-respond-to-changes-to-your-aws-resources/)
* Tim Bray's [Cloud Eventing](https://www.tbray.org/ongoing/When/201x/2016/01/11/CloudWatch-Events) writeup
@@ -9,7 +9,7 @@ In this section we'll walkthrough how to trigger your lambda function in respons

Assume that we're supposed to write a simple "HelloWorld" CloudWatch Logs function that should be triggered in response to any log message issued to a specific Log Group.

# Getting Started
## Getting Started

Our lambda function is relatively short:

@@ -29,7 +29,7 @@ func echoCloudWatchLogsEvent(ctx context.Context, cwlEvent awsLambdaEvents.Cloud

Our lambda function doesn't need to do much with the log message other than log and return it.

# Sparta Integration
## Sparta Integration

With `echoCloudWatchLogsEvent()` implemented, the next step is to integrate the **go** function with Sparta. This is done by the `appendCloudWatchLogsLambda` in the SpartaApplication [application.go](https://github.com/mweagle/SpartaApplication/blob/master/application.go) source.

@@ -55,7 +55,7 @@ cloudWatchLogsPermission.Filters["MyFilter"] = sparta.CloudWatchLogsSubscription
The `sparta.CloudWatchLogsPermission` struct provides fields for both the LogGroupName and optional Filter expression (not shown here) to use when calling [putSubscriptionFilter](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudWatchLogs.html#putSubscriptionFilter-property).
# Add Permission
## Add Permission
With the subscription information configured, the final step is to add the `sparta.CloudWatchLogsPermission` to our `sparta.LambdaAWSInfo` value:
@@ -98,4 +98,4 @@ With the `lambdaFn` fully defined, we can provide it to `sparta.Main()` and depl
* Append the `CloudWatchLogsPermission` value to the lambda function's `Permissions` slice.
* Include the reference in the call to `sparta.Main()`.
# Other Resources
## Other Resources
@@ -0,0 +1,147 @@
---
date: 2019-01-31 05:44:32
title: CodeCommit
weight: 10
---

In this section we'll walkthrough how to trigger your lambda function in response to [CodeCommit Events](https://docs.aws.amazon.com/codecommit/latest/userguide/how-to-notify-lambda.html/).

# Goal

Assume that we're supposed to write a Lambda function that is triggered in response to any event emitted by a CodeCommit repository.

## Getting Started

Our lambda function is relatively short:

```go
import (
awsLambdaEvents "github.com/aws/aws-lambda-go/events"
)
func echoCodeCommit(ctx context.Context, event awsLambdaEvents.CodeCommitEvent) (interface{}, error) {
logger, _ := ctx.Value(sparta.ContextKeyRequestLogger).(*logrus.Entry)
logger.WithFields(logrus.Fields{
"Event": event,
}).Info("Event received")
return &event, nil
}
```

Our lambda function doesn't need to do much with the repository message other than log and return it.

## Sparta Integration

With `echoCodeCommit()` defined, the next step is to integrate the **go** function with your application.

Our lambda function only needs logfile write privileges, and since these are enabled by default, we can use an empty `sparta.IAMRoleDefinition` value:

```go
func appendCloudWatchLogsHandler(api *sparta.API,
lambdaFunctions []*sparta.LambdaAWSInfo) []*sparta.LambdaAWSInfo {
lambdaFn, _ := sparta.NewAWSLambda(sparta.LambdaName(echoCodeCommit),
echoCodeCommit,
sparta.IAMRoleDefinition{})
```
The next step is to add a `CodeCommitPermission` value that represents the
[notification settings](https://docs.aws.amazon.com/codecommit/latest/userguide/how-to-notify.html).
```go
repositoryName := gocf.String("MyTestRepository")
codeCommitPermission := sparta.CodeCommitPermission{
BasePermission: sparta.BasePermission{
SourceArn: repositoryName,
},
RepositoryName: repositoryName.String(),
Branches: branches, // may be nil
Events: events, // may be nil
}
```
The `sparta.CodeCommitPermission` struct provides fields that proxy the
[RepositoryTrigger](https://docs.aws.amazon.com/codecommit/latest/APIReference/API_RepositoryTrigger.html)
values.
## Add Permission
With the subscription information configured, the final step is to
add the `sparta.CodeCommitPermission` to our `sparta.LambdaAWSInfo` value:
```go
lambdaFn.Permissions = append(lambdaFn.Permissions, codeCommitPermission)
```
The entire function is therefore:
```go
func appendCodeCommitHandler(api *sparta.API,
lambdaFunctions []*sparta.LambdaAWSInfo) []*sparta.LambdaAWSInfo {
lambdaFn, _ := sparta.NewAWSLambda(sparta.LambdaName(echoCodeCommit),
echoCodeCommit,
sparta.IAMRoleDefinition{})
repositoryName := gocf.String("MyTestRepository")
codeCommitPermission := sparta.CodeCommitPermission{
BasePermission: sparta.BasePermission{
SourceArn: repositoryName,
},
RepositoryName: repositoryName.String(),
}
lambdaFn.Permissions = append(lambdaFn.Permissions, codeCommitPermission)
return append(lambdaFunctions, lambdaFn)
}
```
# Wrapping Up
With the `lambdaFn` fully defined, we can provide it to `sparta.Main()` and
deploy our service. The workflow below is shared by all
CodeCmmit-triggered lambda functions:
* Define the lambda function (`echoCodeCommit`).
* If needed, create the required [IAMRoleDefinition](https://godoc.org/github.com/mweagle/Sparta*IAMRoleDefinition) with appropriate privileges.
* Provide the lambda function & IAMRoleDefinition to `sparta.NewAWSLambda()`
* Create a [CodeCommitPermission](https://godoc.org/github.com/mweagle/Sparta#CodeCommitPermission) value.
* Define the necessary permission fields.
* Append the `CodeCommitPermission` value to the lambda function's `Permissions` slice.
* Include the reference in the call to `sparta.Main()`.
## Other Resources
* Consider the [archectype](https://gosparta.io/reference/archetypes/codecommit/) package to encapsulate these steps.
* Use the AWS CLI to inspect the configured triggers:
```bash
$ aws codecommit get-repository-triggers --repository-name=TestCodeCommitRepo
{
"configurationId": "7dd7933a-a26c-4514-9ab8-ad8cc133f874",
"triggers": [
{
"name": "MyHelloWorldStack-mweagle_main_echoCodeCommit",
"destinationArn": "arn:aws:lambda:us-west-2:123412341234:function:MyHelloWorldStack-mweagle_main_echoCodeCommit",
"branches": [],
"events": [
"all"
]
}
]
}
```
* Use the AWS CLI to test the configured trigger:
```bash
$ aws codecommit test-repository-triggers --repository-name TestCodeCommitRepo --triggers name=MyHelloWorldStack-mweagle-MyHelloWorldStack-mweagle_main_echoCodeCommit,destinationArn=arn:aws:lambda:us-west-2:123412341234:function:MyHelloWorldStack-mweagle_main_echoCodeCommit,branches=mainline,preprod,events=all
{
"successfulExecutions": [
"MyHelloWorldStack-mweagle-MyHelloWorldStack-mweagle_main_echoCodeCommit"
],
"failedExecutions": []
}
```
@@ -10,7 +10,7 @@ In this section we'll walkthrough how to trigger your lambda function in respons

Assume that we're given a DynamoDB stream. See [below](http://localhost:1313/docs/eventsources/dynamodb/#creatingDynamoDBStream:d680e8a854a7cbad6d490c445cba2eba) for details on how to create the stream. We've been asked to write a lambda function that logs when operations are performed to the table so that we can perform offline analysis.

# Getting Started
## Getting Started

We'll start with an empty lambda function and build up the needed functionality.

@@ -30,7 +30,7 @@ func echoDynamoDBEvent(ctx context.Context, ddbEvent awsLambdaEvents.DynamoDBEve
Since the `echoDynamoDBEvent` is triggered by Dynamo events, we can leverage the AWS Go Lambda SDK [event types](https://godoc.org/github.com/aws/aws-lambda-go/events)
to access the record.

# <a href="{{< relref "#spartaIntegration" >}}">Sparta Integration</a>
## Sparta Integration

With the core of the `echoDynamoDBEvent` complete, the next step is to integrate the **go** function with Sparta. This is performed by the [appendDynamoDBLambda](https://github.com/mweagle/SpartaApplication/blob/master/application.go#L114) function. Since the `echoDynamoDBEvent` function doesn't access any additional services (Sparta enables CloudWatch Logs privileges by default), the integration is pretty straightforward:

@@ -41,7 +41,7 @@ lambdaFn, _ := sparta.NewAWSLambda(
sparta.IAMRoleDefinition{})
```

# Event Source Mappings
## Event Source Mappings

If we were to deploy this Sparta application, the `echoDynamoDBEvent` function would have the ability to log DynamoDB stream events, but would not be invoked in response to events published by the stream. To register for notifications, we need to configure the lambda's [EventSourceMappings](http://docs.aws.amazon.com/lambda/latest/dg/intro-core-components.html#intro-core-components-event-sources):

@@ -60,7 +60,7 @@ The `dynamoTestStream` param is the ARN of the Dynamo stream that that your lamb

The `EventSourceMappings` field is transformed into the appropriate [CloudFormation Resource](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-eventsourcemapping.html) which enables automatic polling of the DynamoDB stream.

# Wrapping Up
## Wrapping Up

With the `lambdaFn` fully defined, we can provide it to `sparta.Main()` and deploy our service. The workflow below is shared by all DynamoDB stream based lambda functions:

@@ -69,26 +69,26 @@ With the `lambdaFn` fully defined, we can provide it to `sparta.Main()` and depl
* Provide the lambda function & IAMRoleDefinition to `sparta.NewAWSLambda()`
* Add the necessary [EventSourceMappings](https://godoc.org/github.com/aws/aws-sdk-go/service/lambda#CreateEventSourceMappingInput) to the `LambdaAWSInfo` struct so that the lambda function is properly configured.

# Other Resources
## Other Resources

* [Using Triggers for Cross Region DynamoDB Replication](https://aws.amazon.com/blogs/aws/dynamodb-update-triggers-streams-lambda-cross-region-replication-app/)

<hr />
# Appendix

## Creating a DynamoDB Stream
### Creating a DynamoDB Stream

To create a DynamoDB stream for a given table, follow the steps below:

### Select Table
#### Select Table

![Select Table](/images/eventsources/dynamodb/DynamoDB_ManageStream.png)

### Enable Stream
#### Enable Stream

![Enable Stream](/images/eventsources/dynamodb/DynamoDB_Enable.png)

### Copy ARN
#### Copy ARN
![Copy ARN](/images/eventsources/dynamodb/DynamoDB_StreamARN.png)

The **Latest stream ARN** value is the value that should be provided as the `EventSourceArn` in to the [Event Source Mappings](http://localhost:1313/docs/eventsources/dynamodb/#eventSourceMapping:d680e8a854a7cbad6d490c445cba2eba).
@@ -10,7 +10,7 @@ In this section we'll walkthrough how to trigger your lambda function in respons

The goal of this example is to provision a Sparta lambda function that logs Amazon Kinesis events to CloudWatch logs.

# Getting Started
## Getting Started

We'll start with an empty lambda function and build up the needed functionality.

@@ -32,7 +32,7 @@ it, and return the value.

With the function defined let's register it with Sparta.

# Sparta Integration
## Sparta Integration

First we wrap the **go** function in a [LambdaAWSInfo](https://godoc.org/github.com/mweagle/Sparta#LambdaAWSInfo) struct:

@@ -44,7 +44,7 @@ lambdaFn, _ := sparta.NewAWSLambda(sparta.LambdaName(echoKinesisEvent),

Since our lambda function doesn't access any other AWS Services, we can use an empty IAMRoleDefinition (`sparta.IAMRoleDefinition{}`).

# Event Source Registration
## Event Source Registration

Then last step is to configure our AWS Lambda function with Kinesis as the [EventSource](http://docs.aws.amazon.com/lambda/latest/dg/intro-core-components.html)

@@ -11,7 +11,7 @@ In this section we'll walkthrough how to trigger your lambda function in respons

Assume we have an S3 bucket that stores images. You've been asked to write a service that creates a duplicate image that includes a characteristic stamp overlay and store it in the same S3 bucket.

# Getting Started
## Getting Started

We'll start with an empty lambda function and build up the needed functionality.

@@ -74,7 +74,7 @@ applying the stamp, and putting the transformed content back to S3 with a new na
items which have already been stamped & prevents an "event-storm" from being triggered. This simple approach is acceptable for an example,
but in production you should use a more durable approach.
# Sparta Integration
## Sparta Integration
With the core of the `transformImage` complete, the next step is to integrate the **go** function with Sparta. This is performed by the [imagerFunctions](https://github.com/mweagle/SpartaImager/blob/master/application.go#L200) source.
@@ -115,7 +115,7 @@ lambdaFn.Options = transformOptions
It typically takes more than 3 seconds to apply the transform, so we increase the execution timeout and provision a new
lambda function using the `iamRole` we defined earlier.
# Event Source Registration
## Event Source Registration
If we were to deploy this Sparta application, the `transformImage` function would have the ability to *Get* and *Put* back
to the `s3EventBroadcasterBucket`, but would not be invoked in response to events triggered by that bucket. To register
@@ -151,6 +151,6 @@ With the `lambdaFn` fully defined, we can provide it to `sparta.Main()` and depl
The [SpartaImager](https://github.com/mweagle/SpartaImager) repo contains the full code, and includes [API Gateway](/reference/apigateway) support that allows you to publicly fetch the stamped image via an expiring S3 URL.
# Other Resources
## Other Resources
* The AWS docs have an excellent [S3 event source](http://docs.aws.amazon.com/lambda/latest/dg/getting-started-amazons3-events.html) walkthrough.
Oops, something went wrong.

0 comments on commit 24876a5

Please sign in to comment.