resource/aws_lambda_event_source_mapping: Adding ParallelizationFactor, MaximumRecordAgeInSeconds, BisectBatchOnFunctionError, MaximumRetryAttempts, DestinationConfig

[kostas-theo]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

This PR implements the new functionality added in aws-sdk-go v1.25.42 for lambda event source mappings, specifically adding the optional arguments/configurations:

• ParallelizationFactor
• MaximumRecordAgeInSeconds
• BisectBatchOnFunctionError
• MaximumRetryAttempts

Related AWS Posts describing changes:

• https://aws.amazon.com/blogs/compute/new-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources/
• https://aws.amazon.com/blogs/compute/new-aws-lambda-controls-for-stream-processing-and-asynchronous-invocations/

Release note for CHANGELOG:

* resource/aws_lambda_event_source_mapping: Support `parallelization_factor` configuration
* resource/aws_lambda_event_source_mapping: Support `maximum_retry_attempts` configuration
* resource/aws_lambda_event_source_mapping: Support `maximum_record_age_in_seconds` configuration
* resource/aws_lambda_event_source_mapping: Support `bisect_batch_on_function_error` configuration
* resource/aws_lambda_event_source_mapping: Support `destination_config` configuration

Output from acceptance testing:

$ make testacc TEST=./aws TESTARGS='-run=TestAccAWSLambdaEventSourceMapping'
Checking for updates
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSLambdaEventSourceMapping -timeout 120m
=== RUN   TestAccAWSLambdaEventSourceMapping_kinesis_basic
=== PAUSE TestAccAWSLambdaEventSourceMapping_kinesis_basic
=== RUN   TestAccAWSLambdaEventSourceMapping_kinesis_removeBatchSize
=== PAUSE TestAccAWSLambdaEventSourceMapping_kinesis_removeBatchSize
=== RUN   TestAccAWSLambdaEventSourceMapping_sqs_basic
=== PAUSE TestAccAWSLambdaEventSourceMapping_sqs_basic
=== RUN   TestAccAWSLambdaEventSourceMapping_sqs_withFunctionName
=== PAUSE TestAccAWSLambdaEventSourceMapping_sqs_withFunctionName
=== RUN   TestAccAWSLambdaEventSourceMapping_kinesis_disappears
=== PAUSE TestAccAWSLambdaEventSourceMapping_kinesis_disappears
=== RUN   TestAccAWSLambdaEventSourceMapping_sqsDisappears
=== PAUSE TestAccAWSLambdaEventSourceMapping_sqsDisappears
=== RUN   TestAccAWSLambdaEventSourceMapping_changesInEnabledAreDetected
=== PAUSE TestAccAWSLambdaEventSourceMapping_changesInEnabledAreDetected
=== RUN   TestAccAWSLambdaEventSourceMapping_StartingPositionTimestamp
=== PAUSE TestAccAWSLambdaEventSourceMapping_StartingPositionTimestamp
=== RUN   TestAccAWSLambdaEventSourceMapping_BatchWindow
=== PAUSE TestAccAWSLambdaEventSourceMapping_BatchWindow
=== RUN   TestAccAWSLambdaEventSourceMapping_ParallelizationFactor
=== PAUSE TestAccAWSLambdaEventSourceMapping_ParallelizationFactor
=== RUN   TestAccAWSLambdaEventSourceMapping_MaximumRetryAttempts
=== PAUSE TestAccAWSLambdaEventSourceMapping_MaximumRetryAttempts
=== RUN   TestAccAWSLambdaEventSourceMapping_MaximumRecordAgeInSeconds
=== PAUSE TestAccAWSLambdaEventSourceMapping_MaximumRecordAgeInSeconds
=== RUN   TestAccAWSLambdaEventSourceMapping_BisectBatch
=== PAUSE TestAccAWSLambdaEventSourceMapping_BisectBatch
=== RUN   TestAccAWSLambdaEventSourceMapping_KinesisDestinationConfig
=== PAUSE TestAccAWSLambdaEventSourceMapping_KinesisDestinationConfig
=== CONT  TestAccAWSLambdaEventSourceMapping_kinesis_basic
=== CONT  TestAccAWSLambdaEventSourceMapping_BatchWindow
=== CONT  TestAccAWSLambdaEventSourceMapping_kinesis_disappears
=== CONT  TestAccAWSLambdaEventSourceMapping_ParallelizationFactor
=== CONT  TestAccAWSLambdaEventSourceMapping_sqs_basic
=== CONT  TestAccAWSLambdaEventSourceMapping_KinesisDestinationConfig
=== CONT  TestAccAWSLambdaEventSourceMapping_BisectBatch
=== CONT  TestAccAWSLambdaEventSourceMapping_MaximumRecordAgeInSeconds
=== CONT  TestAccAWSLambdaEventSourceMapping_MaximumRetryAttempts
=== CONT  TestAccAWSLambdaEventSourceMapping_changesInEnabledAreDetected
=== CONT  TestAccAWSLambdaEventSourceMapping_StartingPositionTimestamp
=== CONT  TestAccAWSLambdaEventSourceMapping_sqsDisappears
=== CONT  TestAccAWSLambdaEventSourceMapping_sqs_withFunctionName
=== CONT  TestAccAWSLambdaEventSourceMapping_kinesis_removeBatchSize
--- PASS: TestAccAWSLambdaEventSourceMapping_sqs_withFunctionName (78.29s)
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_disappears (93.28s)
--- PASS: TestAccAWSLambdaEventSourceMapping_ParallelizationFactor (101.79s)
--- PASS: TestAccAWSLambdaEventSourceMapping_BatchWindow (101.84s)
--- PASS: TestAccAWSLambdaEventSourceMapping_KinesisDestinationConfig (104.86s)
--- PASS: TestAccAWSLambdaEventSourceMapping_MaximumRetryAttempts (114.41s)
--- PASS: TestAccAWSLambdaEventSourceMapping_BisectBatch (122.58s)
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_basic (136.74s)
--- PASS: TestAccAWSLambdaEventSourceMapping_sqs_basic (152.49s)
--- PASS: TestAccAWSLambdaEventSourceMapping_StartingPositionTimestamp (168.06s)
--- PASS: TestAccAWSLambdaEventSourceMapping_changesInEnabledAreDetected (169.48s)
--- PASS: TestAccAWSLambdaEventSourceMapping_sqsDisappears (175.23s)
--- PASS: TestAccAWSLambdaEventSourceMapping_MaximumRecordAgeInSeconds (176.58s)
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_removeBatchSize (177.83s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	177.931s
...

[kostas-theo]

Hi @bflad, I wonder if this feature set could be added to your shortlist?

[hexionas]

Any ideas when this could be merged in?

[kostas-theo]

Updated this PR to remove the implementation of DestinationConfig which, since this PR was created, has been implemented in the new aws_lambda_function_event_invoke_config.

[soupmatt]

Updated this PR to remove the implementation of DestinationConfig which, since this PR was created, has been implemented in the new aws_lambda_function_event_invoke_config.

I believe you should put the destination_config back in. The aws_lambda_function_event_invoke_config applies to asynchronous invocation of the lambda function, but an aws_event_source_mapping with either a Kinesis Stream or a DynamoDB Stream invokes the function synchronously (see the the second paragraph here and the second paragraph here).

I made a test lambda and tried 3 different situations for DLQing with a kinesis aws_event_source_mapping.

• Putting a dead_letter_config directly on the aws_lambda_function.
• Adding a aws_lambda_function_event_invoke_config with an on failure destination.
• Adding a destination_config to the aws_event_source_mapping.

Only the destination_config on the aws_event_source_mapping resulted in failed events being DLQ'ed to my configured SQS queue.

[kostas-theo]

@soupmatt - Thanks for the review. I've re-added the destination_config block to this PR. The tests have also been re-run and results ammended in the description.

[bflad]

Hi @kostas-theo 👋 Thank you for submitting this. Please see the below initial feedback and let us know if you have questions or do not have time to implement them.

[kostas-theo]

@bflad - Thanks for the review. I believe if resolved all of your recommendations. Let me know if there is anything further required. Cheers!

[bflad]

Fantastic, thanks @kostas-theo 🚀

Output from acceptance testing:

--- PASS: TestAccAWSLambdaEventSourceMapping_sqs_withFunctionName (36.43s)
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_disappears (63.39s)
--- PASS: TestAccAWSLambdaEventSourceMapping_MaximumRecordAgeInSeconds (67.88s)
--- PASS: TestAccAWSLambdaEventSourceMapping_ParallelizationFactor (70.38s)
--- PASS: TestAccAWSLambdaEventSourceMapping_StartingPositionTimestamp (70.49s)
--- PASS: TestAccAWSLambdaEventSourceMapping_BatchWindow (70.76s)
--- PASS: TestAccAWSLambdaEventSourceMapping_BisectBatch (71.28s)
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_removeBatchSize (73.50s)
--- PASS: TestAccAWSLambdaEventSourceMapping_KinesisDestinationConfig (76.77s)
--- PASS: TestAccAWSLambdaEventSourceMapping_MaximumRetryAttempts (79.29s)
--- PASS: TestAccAWSLambdaEventSourceMapping_kinesis_basic (83.44s)
--- PASS: TestAccAWSLambdaEventSourceMapping_sqs_basic (86.55s)
--- PASS: TestAccAWSLambdaEventSourceMapping_sqsDisappears (112.88s)
--- PASS: TestAccAWSLambdaEventSourceMapping_changesInEnabledAreDetected (134.17s)

[ghost]

This has been released in version 2.51.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!