Ability to add custom EventSource and lambda triggers via amplify add function, Kinesis support in analytics category

[ambientlight]

Issue #, if available:
#997
#2256

Description of changes:

1. Exposes streamArnOutputId, datasourceOutputId, tableNameOutputId from nested api category stack backing each @model dynamoDB table, so they would available inside root api category stack output and inside amplify-meta and can be references in other categories templates (graphql-dynamodb-transformer). It is needed in the context of this PR, so we can reference @model dynamoDB table streamArn in our custom eventSource.
2. A reasonable CLI support in amplify-category-function to confirm a lambda with a custom event source. DynamoDBStream / Kinesis event source types are supported. For DynamoDBStream, user is able to choose from api category resource @model dynamoDB tables, storage category existing dynamoDB table, or as last resort directly enter streamArn. For Kinesis, user is able to choose from analytics category resource kinesis stream or also directly enter streamArn`.
3. Base lambda-cloudformation-template.json.ejs template has been extended to optionally include eventSourceMapping and LambdaTriggerPolicy when available as props.triggerEventSourceMapping.

Motivation:

1. A need to have a lambda trigger attached to @model backed dynamoDB tables. Since graphql-transformer-core will update CloudFormation templates on each graphql update, it is not possible to modify the templates directly like in other category resources to output nested stacks outputs
2. Existing eventSource lambda trigger functionality available in storage category is quite unintuitive, slightly opinionated, but I would argue most users will expect triggers to be available as part of function category just as it is in lambda aws console.
3. If the changes are accepted this will also allow to avoid a set of shortcuts in amplify-category-storage that explicitly load and manipulate resources and parameters in function CloudFormation templates, if we would go with replacing a storage category trigger creation with function category built-in implementation presented here.

   amplify-cli/packages/amplify-category-storage/provider-utils/awscloudformation/service-walkthroughs/dynamoDb-walkthrough.js

   Lines 606 to 684 in cb8f6dd

   	if (fs.existsSync(functionCFNFilePath)) {
   	const functionCFNFile = context.amplify.readJsonFile(functionCFNFilePath);
   	// Update parameters block
   	functionCFNFile.Parameters[`storage${resourceName}Name`] = {
   	Type: 'String',
   	Default: `storage${resourceName}Name`,
   	};
   	functionCFNFile.Parameters[`storage${resourceName}Arn`] = {
   	Type: 'String',
   	Default: `storage${resourceName}Arn`,
   	};
   	functionCFNFile.Parameters[`storage${resourceName}StreamArn`] = {
   	Type: 'String',
   	Default: `storage${resourceName}Arn`,
   	};
   	// Update policies
   	functionCFNFile.Resources[`${resourceName}TriggerPolicy`] = {
   	DependsOn: [
   	'LambdaExecutionRole',
   	],
   	Type: 'AWS::IAM::Policy',
   	Properties: {
   	PolicyName: 'lambda-execution-policy',
   	Roles: [
   	{
   	Ref: 'LambdaExecutionRole',
   	},
   	],
   	PolicyDocument: {
   	Version: '2012-10-17',
   	Statement: [
   	{
   	Effect: 'Allow',
   	Action: [
   	'dynamodb:DescribeStream',
   	'dynamodb:GetRecords',
   	'dynamodb:GetShardIterator',
   	'dynamodb:ListStreams',
   	],
   	Resource: [
   	{
   	Ref: `storage${resourceName}StreamArn`,
   	},
   	],
   	},
   	],
   	},
   	},
   	};
   	// Add TriggerResource
   	functionCFNFile.Resources[`${resourceName}Trigger`] = {
   	Type: 'AWS::Lambda::EventSourceMapping',
   	DependsOn: [
   	`${resourceName}TriggerPolicy`,
   	],
   	Properties: {
   	BatchSize: 100,
   	Enabled: true,
   	EventSourceArn: {
   	Ref: `storage${resourceName}StreamArn`,
   	},
   	FunctionName: {
   	'Fn::GetAtt': [
   	'LambdaFunction',
   	'Arn',
   	],
   	},
   	StartingPosition: 'LATEST',
   	},
   	};

4. It would also allow users to create event sources for non-amplify managed resources.

Caveats:

To make CLI discover @model backed dynamoDB table outputs, user will need to push any update to api category to have CloudFormation templates regenerated with nested stack outputs.

Usage:

When using amplify add function new function template is now available:

$ amplify-dev add function
Using service: Lambda, provided by: awscloudformation
? Provide a friendly name for your resource to be used as a label for this category in the project: testtrigger
? Provide the AWS Lambda function name: testtrigger
? Choose the function template that you want to use: 
  Hello world function 
  CRUD function for Amazon DynamoDB table (Integration with Amazon API Gateway and Amazon DynamoDB) 
  Serverless express function (Integration with Amazon API Gateway) 
❯ Custom EventSourceMapping Lambda Trigger

Custom event source mapping will then present an ability to choose event source type:

? What event source do you want to associate with Lambda trigger (Use arrow keys)
❯ Amazon DynamoDB Stream 
  Amazon Kinesis Stream

Selecting dynamoDB stream will allow user either select @model backed dynamoDB table as event source, or storage category dynamoDB table. api category graphql resource is REQUIRED to be already deployed.

? Choose a DynamoDB event source option (Use arrow keys)
❯ Use API category graphql @model backed DynamoDB table configured(and deployed) in the current Amplify project 
  Use storage category DynamoDB table configured in the current Amplify project 
  Provide the ARN of DynamoDB stream directly 

Now if we go ahead with API category @model option, resource selection and then @model selection will follow.

? Please choose an API resource to associate with? coreapi
? Please choose a graphql @model Fixture
? Do you want to access other resources created in this project from your Lambda function? (Y/n)

Alternatively, if user would go on to select storage category dynamoDB table, he will be just prompted to select the resource. All other options will involve event source ARN input that will be validated by the following regex: "arn:(aws[a-zA-Z0-9-]*):([a-zA-Z0-9\\-])+:([a-z]{2}(-gov)?-[a-z]+-\\d{1})?:(\\d{12})?:(.*)"

Generated dynamoDB event source lambda function will have the same placeholder as the dynamoDB trigger function looks like right now:

exports.handler = function (event, context) { //eslint-disable-line
  console.log(JSON.stringify(event, null, 2));
  event.Records.forEach((record) => {
    console.log(record.eventID);
    console.log(record.eventName);
    console.log('DynamoDB Record: %j', record.dynamodb);
  });
  context.done(null, 'Successfully processed DynamoDB record'); // SUCCESS with message
};

If selecting AWS KInesis instead:

? Choose a Kinesis event source option (Use arrow keys)
❯ Use Analytics category kinesis stream configured(and deployed) in the current Amplify project 
  Provide the ARN of Kinesis stream directly 

then

? Please select an Analytics resource Kinesis stream to associate with? 
  someotherkinesisresource 
❯ testkinesis 

Kinesis will have a generic empty trigger function:

exports.handler = (event, context, callback) => {
  // insert code to be executed by your lambda trigger
  callback(null, event);
};

Update (2019-10-23): Kinesis support for analytics.

This PR also extended analytics category to have native cloudformation wrappers for kinesis streams.

prompt:

$: amplify add analytics
? Select an Analytics provider Amazon Kinesis Streams
? Enter a Stream name testkinesis
? Enter number of shards 1
Successfully added resource testkinesis locally

This is a useful, taking into account amplify has kinesis support on the client side in analytics:

import { Analytics, AWSKinesisProvider } from 'aws-amplify';
/// ...
const kinesisProvider = new AWSKinesisProvider({ region: 'eu-central-1' });
Analytics.addPluggable(kinesisProvider);
kinesisProvider.record({
  event: {
    data: {},
    partitionKey: '0', 
    streamName: 'testkinesis-dev'
  }
}).then(didSucceed => { 
  console.log(didSucceed) 
});

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[kaustavghosh06]

@ambientlight Does this also work with multiple environments? Specifically the @model triggers?

[ambientlight]

@kaustavghosh06: it does, an api category root stack reexports embedded DynamoDB stack output(streamArn) since actually streams are already enabled on these dynamo tables, so then we can use dependsOn and pass those outputs to lambda CFT as parameters

[undefobj]

@ambientlight Thanks so much for this PR. We reviewed this today and would really like to merge this, however we have some requests to have this fit within the current Amplify design philosophy.

For this message:

Custom EventSourceMapping Lambda Trigger

Can you change this to just say Lambda Trigger?

Next, for this part of the flow:

? What event source do you want to associate with Lambda trigger (Use arrow keys)
❯ Amazon DynamoDB Stream 
  Amazon Kinesis 
  Amazon Simple Queue Service 

We have two asks:

1. Could you remove SQS as an option? That doesn't fit into any current Amplify category and could add extra confusion and complexity for customers, where Amplify is an opinionated Framework on implementations. If there are use cases and asks in the future requiring queuing we can revisit this decision.
2. Kinesis is currently a part of the Analytics category for the client libraries, therefore we think this is a great addition. Would you be able to modify the amplify add analytics flow to add a Kinesis stream with the following flow:

$amplify add analytics
? Select an Analytics provider
❯ Amazon Pinpoint (Default) 
❯ Amazon Kinesis Streams

If the customer selects Pinpoint the flow continues as-is today, however if they select Kinesis they can enter the name and number of shards:

? Enter a Stream name
? Enter number of shards (default 1)

You will also need to modify amplify update analytics so that the customer can update the shard count.

At this point there will be a Kinesis stream in the project, so similar to your @model flow when running amplify add function the customer should then be able to select this stream.

From looking at your PR it appears you have understanding of the CLI infrastructure to perform these changes, however if you need some assistance please let us know.

[ambientlight]

@undefobj: oh yeah, totally forgot amplify has kineses integration. I will try to have these changes sorted out these days so we can keep the momentum here

[Jkap7788]

@ambientlight Love your work!
Correct me if I'm wrong but this PR provides the ability to automatically connect lambda functions to dynamoDB streams via the CLI.

However, it does not provide a way of giving access to lambda functions to read, write, update and delete @model generated tables.
Currently, the CLI provides this functionality but only for non @model generated tables.
It would be awesome if we could extend this to include @model tables as well as this is a pretty common use case.

Just wondering if there are any plans on implementing CRUD permissions for @model tables from lambda functions?

[ambientlight]

@Jkap7788: thanks for the feedback, If my memory is right, additional permissions can be added in the next question following trigger selection question, for additional permissions we are able to select API category graphql resource and that will bring the selection for CRUD permissions, I will make to sure to verify this as I finalize this PR

[matt-e-king]

@ambientlight correct, I can select CRUD operation against the API when adding a new function. This is what you see in the CLI:

? Do you want to access other resources created in this project from your Lambda function? Yes
? Select the category api
Api category has a resource called <apiname>
? Select the operations you want to permit for <apiname> create, read, update, delete

But that applies to the whole api, not a specific @model or table in Dynamo - so it's not as streamlines to lock down a specific function to operations on a specific table.

[kaustavghosh06]

@ambientlight Please let us know if you need any help from our side on this PR. Eager to merge this!

[justin-robinson]

why not use the callback model since this isn't an async function? https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html

[ambientlight]

absolutely, actually originally just copied this template from

amplify-cli/packages/amplify-category-storage/provider-utils/awscloudformation/triggers/dynamoDB/index.js

Lines 1 to 10 in 08c0c70

	exports.handler = function(event, context) {
	//eslint-disable-line
	console.log(JSON.stringify(event, null, 2));
	event.Records.forEach(record => {
	console.log(record.eventID);
	console.log(record.eventName);
	console.log('DynamoDB Record: %j', record.dynamodb);
	});
	context.done(null, 'Successfully processed DynamoDB record'); // SUCCESS with message
	};

[justin-robinson]

Not suggesting you change it. Just curious. If that's the template used then follow that.

[ambientlight]

it is a default lambda code template for @model trigger, thanks for bringing this up, it makes sense refining both templates. I mean, for now storage category trigger does not utilize function-category implementation, while actually having a very same logic for triggers, if this gets accepted, will be great to have another PR updating that utilizes function-category implementation in storage flow.

[ambientlight]

@kaustavghosh06: I think I should be fine with completing proposed changes, thus far I have the only question: I have noticed pinpoint policy for cognito identity pool is same for unauth / auth case, I have followed the same approach here, I guess it is indented so Analytics.record(...) will be working for both unauthed / authed user, can we consider this to be a safe practice?, this by itself seems to bring the need to have throttling in place for unauth case right?

[lgtm-com]

This pull request introduces 2 alerts when merging 1f79657 into ad3a137 - view on LGTM.com

new alerts:

• 2 for Assignment to constant

[kaustavghosh06]

@ambientlight Do you know why the circleci tests are failing? Is it just due to rebasing?

[kaustavghosh06]

@ambientlight You'd have to rebase your PR. We fixed this issue in this PR - https://github.com/aws-amplify/amplify-cli/pull/3457/files around a week back. That should hopefully unblock and make the CircleCI tests pass.

[kaustavghosh06]

Yes, if you look at the PR metnioned above, we added this to our config since there seems to be some issue with the docker image on CircleCI:

      - run:
          name: Update OS Packages
          command: sudo apt-get update

[ambientlight]

Would you be able to add E2E tests for the bugs which I came across in the flows above?

@kaustavghosh06:
I guess I have fulfilled the above request.

To summarize the above described bugs:

• invalid fn:sub reference in storage category appsync additional permissions
• storage(appsynch) additional permissions function has no envvars
• storage(appsync) additional permissions function update when no storage category would crash
• kinesis trigger flow would crash

were covered. Also a two other discovered bugs were fixed:

• infinitely looped There are no DynamoDB resources configured in your project currently in trigger flow when attaching to existing dynamoDB (when none).
• invalid dynamoDB ARN in additional permissions for appsync-@model backed dynamoDB table.

And analytics e2e tests have been updated to account for CLI flow changes.

[kaustavghosh06]

Thanks @ambientlight. I'm going through the flow end to end - hopefully one last time :) and reviewing your code as well. Once again, thanks for taking action on the feedback so promptly.

[andreialecu]

Awesome. Great to finally see this merged!

Is there any chance we can see an official article/tutorial soon about how to integrate something like sending push notifications when a dynamodb collection is changed? For things like chat messages for example.

[ambientlight]

@andreialecu: thanks, how are doing them as of now? you are thinking about something like a lambda trigger that will push to SNS topic wired up with your APNS / Firebase?

Existing amplify push notification support is through pinpoint, those are a bit different in purpose imho, for marketing campaigns rather then custom notifications.

[andreialecu]

Not doing them yet on this amplify app, it's still under development.

I was waiting for this PR to be merged before moving on to implementing push notifications. 🚀

I'm really not sure what the best course of action would be. It's the first app on Amplify for us. In previous apps I've been sending them manually from the backend, completely separate from AWS.

SNS Mobile Push seems like the right way to tackle this, but I'm completely open to suggestions. :)

[ambientlight]

@andreialecu: myself handling this in django (in lambdas via zappa) with django-push-notifications

[ldariozero12]

Hi,
How long does it take to release this feature?

Thx for the support

[ambientlight]

@ldariozero12: it's under canary for now, you can try it with:

npm i --global @aws-amplify/cli@canary

[ldariozero12]

Ok, thx.

There are an alternative to create a DynamoDB stream and related lambda trigger for a type @model without this functionality?

[ambientlight]

@ldariozero12: yes, you can write cloudformation template yourself (which would be kinda same as this pr's lambda trigger), please refer to #2463 (comment)

[BrittonWinterrose]

This is cool. Great work.

[isaiahtaylorhh]

Hello, I want to make sure I'm understanding the nature of this feature, since all of the bugs that seem to address my problem redirect here.

I'm wanting to create a custom field pointing to a lambda that sources from my @model created DynamoDB tables.

e.g.:

type Cookie
@model
{
  flavor: string @function(name: "MyFunctionIMadeWithAmplifyAddFunction");
}

Is there an amplify add function path for this?

[bob-lee]

Much appreciated for this timely PR. I was using 4.18.0 and needed to create a lambda trigger for one of @model backed table to handle some modify event which would take a while longer than a minute. Running amplify add function and following instructions allowed me to create a new lambda function that will be invoked on the table change and it even started to log on cloudwatch properly. I am now ready to make code change on default lambda function to do a real thing. Thanks.

[jagadish432]

when running amplify add function , it is showing the "Lambda trigger" option ONLY when NODEJS is selected as the lambda's runtime language, when Python is selected as the lambda's runtime language then amplify cli is not showing the "Lambda trigger" option.

How to add a dynamodb trigger to python lambda using amplify framework?

[kristianmandrup]

@jagadish432 Sadly, python doesn't seem to have received much love or attention in amplify yet. I think you either need to keep pushing or implement that (yourself?) in the Python community and make the necessary PRs.

[github-actions]

This pull request has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels for those types of questions.