[aws-lambda] Add "Topics" property to EventSourceMappingOptions

[dscpinheiro]

Lambda recently added support for MSK as an event source (https://aws.amazon.com/about-aws/whats-new/2020/08/aws-lambda-now-supports-amazon-managed-streaming-for-apache-kafka-as-an-event-source/), and there's now a "Topics" property on the CloudFormation resource definition (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-eventsourcemapping.html#cfn-lambda-eventsourcemapping-topics).

Use Case

Using Lambda simplifies reading data from Apache Kafka, as other options usually require customers to worry about infrastructure management.

Proposed Solution

Include topics as an optional parameter of EventSourceMappingOptions interface:

export interface EventSourceMappingOptions {
    /**
     * existing properties
     */

    /**
     * The name of the Kafka topic.
     * 
     * Valid range:
     * * Maximum value of 1 
     */
    readonly topics?: string[];
}

It'd then be used as:

lambdaFn.addEventSourceMapping('MSKTrigger', {
    eventSourceArn: 'arn:aws:kafka:us-east-1:123456789012:cluster/kafka-cluster/random-id',
    startingPosition: lambda.StartingPosition.LATEST,
    topics: ['my-topic']
});

Other

In the future, there could even be a MSKEventSource in the aws-lambda-event-sources package, but just adding the topics property should allow customers to get started.

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[dscpinheiro]

So it looks like CfnEventSourceMapping does not contain topics yet, but it's included in this PR (#9979).

@nija-at Is there an estimate when that PR will be merged?

[nija-at]

There is currently no higher level construct support for MSK in the CDK - https://github.com/aws/aws-cdk/tree/master/packages/%40aws-cdk/aws-msk

It will make sense for this to be added once we have some basic L2 support.

[dscpinheiro]

@nija-at This might be a dumb question, but wouldn't it still make sense to still add the topics property to the interface? The EventSourceMappingOptions interface would then match completely with the AWS::Lambda::EventSourceMapping resource type.

Edit: I just noticed that the cfnspec has been merged, so I can create a PR for this change.

[nija-at]

Yes, you are right. This can absolutely be done.

Usually, this is accompanied by an integration class in aws-lambda-event-sources package. This will have to wait until the higher level construct for MSK is in place.

[dscpinheiro]

If I created the PR just adding the topics property, would that be rejected? My concern is that if a customer is already using the CDK to manage their MSK clusters, and now wants to add an integration with Lambda, that's just not possible (and waiting for a higher level construct might make them take a different approach).

I did find a PR for a L2 construct for MSK, but it hasn't been reviewed yet (#9908).

[nija-at]

If I created the PR just adding the topics property, would that be rejected?

Definitely not. Feel free to post one.

[dscpinheiro]

@nija-at PR was created: #10445