AT_TIMESTAMP value for StartingPosition not supported in CloudFormation

[jeremlvt]

Description:

I'm trying to deploy a SAM template that creates a Kinesis stream and a Lambda function that triggers everytime data goes through the stream. I use parameters, one of them being the StartingPosition that I want to use for the Kinesis stream. The SAM documentation for Kinesis events seems to say that AT_TIMESTAMP is a correct value to pass as StartingPosition, but the CloudFormation documentation for EventSourceMapping doesn't include AT_TIMESTAMP as a correct value.
This seems weird considering timestamps can be used as starting positions with the AWS CLI as well as the AWS API.
Here is the template I used :

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-aws-kinesis-lambda

Globals:
  Function:
    Timeout: 5
    Runtime: nodejs12.x
    MemorySize: 128
    Tracing: Active
    AutoPublishAlias: live

Parameters:
  RetentionPeriodHours:
    Description: The number of hours for the data records that are stored in shards to remain accessible.
    Type: Number
    Default: 24
    MinValue: 24
    MaxValue: 168
  ShardCount:
    Description: The number of shards that the stream uses.
    Type: Number
    Default: 1
    MinValue: 1
  BatchSize:
    Description: The number of records to send to the function in each batch.
    Type: Number
    Default: 500
    MinValue: 1
    MaxValue: 10000
  BisectBatchOnFunctionError:
    Description: If the function returns an error, split the batch in two and retry.
    Type: String
    Default: "true"
    AllowedValues:
      - "true"
      - "false"
  StartingPosition:
    Description: Process only new records, or all existing records.
    Type: String
    Default: TRIM_HORIZON
    AllowedValues:
      - TRIM_HORIZON
      - LATEST
      - AT_TIMESTAMP
  MaximumRetryAttempts:
    Description: The maximum number of times to retry when the function returns an error.
    Type: Number
    Default: 1000
    MinValue: 0
    MaxValue: 10000
  ParallelizationFactor:
    Description: The maximum number of times to retry when the function returns an error.
    Type: Number
    Default: 1
    MinValue: 1
    MaxValue: 10
  
Resources:
  KinesisTriggerFunction:
    Type: "AWS::Serverless::Function"
    Properties:
      CodeUri: functions/
      Handler: trigger.handler
      Description: >
        This function triggered when a file is uploaded in a stream (Kinesis)
      Events: 
        Stream:
          Type: Kinesis
          Properties:
            Stream: !GetAtt KinesisStream.Arn
            BatchSize: !Ref BatchSize
            StartingPosition: !Ref StartingPosition
            ParallelizationFactor: !Ref ParallelizationFactor
            MaximumRetryAttempts: !Ref MaximumRetryAttempts
            BisectBatchOnFunctionError: !Ref BisectBatchOnFunctionError
      Policies:
        - KinesisStreamReadPolicy:
            StreamName: !Ref KinesisStream

  KinesisStream: 
    Type: AWS::Kinesis::Stream 
    Properties: 
        Name: KinesisStream
        RetentionPeriodHours: !Ref RetentionPeriodHours 
        ShardCount: !Ref ShardCount
        StreamEncryption:
            EncryptionType: KMS 
            KeyId: alias/aws/kinesis

Steps to reproduce the issue :

Use sam build and sam deploy --guided with the previous template and a dummy hello world function in NodeJS

I am using SAM CLI version 1.2.0 for Windows 10

Observed result:

I am getting the following error on deployment :

Model validation failed (#/StartingPosition: failed validation constraint for keyword [pattern])

Expected result:

I should be able to use AT_TIMESTAMP as a StartingPosition for my Kinesis event in CloudFormation. Any help is appreciated, thank you very much !

[c2tarun]

Hi @jeremlvt ,

I am not able to reproduce your issue. I tried to deploy your template using sam build and sam deploy --guided and it did deploy successfully.

CloudFormation events from changeset
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                              ResourceType                                                LogicalResourceId                                           ResourceStatusReason
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS                                          AWS::Kinesis::Stream                                        KinesisStream                                               -
CREATE_IN_PROGRESS                                          AWS::Kinesis::Stream                                        KinesisStream                                               Resource creation Initiated
CREATE_COMPLETE                                             AWS::Kinesis::Stream                                        KinesisStream                                               -
CREATE_IN_PROGRESS                                          AWS::IAM::Role                                              KinesisTriggerFunctionRole                                  -
CREATE_IN_PROGRESS                                          AWS::IAM::Role                                              KinesisTriggerFunctionRole                                  Resource creation Initiated
CREATE_COMPLETE                                             AWS::IAM::Role                                              KinesisTriggerFunctionRole                                  -
CREATE_IN_PROGRESS                                          AWS::Lambda::Function                                       KinesisTriggerFunction                                      -
CREATE_IN_PROGRESS                                          AWS::Lambda::Function                                       KinesisTriggerFunction                                      Resource creation Initiated
CREATE_COMPLETE                                             AWS::Lambda::Function                                       KinesisTriggerFunction                                      -
CREATE_IN_PROGRESS                                          AWS::Lambda::Version                                        KinesisTriggerFunctionVersionc5a55e0c54                     -
CREATE_COMPLETE                                             AWS::Lambda::Version                                        KinesisTriggerFunctionVersionc5a55e0c54                     -
CREATE_IN_PROGRESS                                          AWS::Lambda::Version                                        KinesisTriggerFunctionVersionc5a55e0c54                     Resource creation Initiated
CREATE_COMPLETE                                             AWS::Lambda::Alias                                          KinesisTriggerFunctionAliaslive                             -
CREATE_IN_PROGRESS                                          AWS::Lambda::Alias                                          KinesisTriggerFunctionAliaslive                             Resource creation Initiated
CREATE_IN_PROGRESS                                          AWS::Lambda::Alias                                          KinesisTriggerFunctionAliaslive                             -
CREATE_IN_PROGRESS                                          AWS::Lambda::EventSourceMapping                             KinesisTriggerFunctionStream                                -
CREATE_COMPLETE                                             AWS::Lambda::EventSourceMapping                             KinesisTriggerFunctionStream                                -
CREATE_IN_PROGRESS                                          AWS::Lambda::EventSourceMapping                             KinesisTriggerFunctionStream                                Resource creation Initiated
CREATE_COMPLETE                                             AWS::CloudFormation::Stack                                  issue-1741                                                  -
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - xxxx-1741 in us-east-1

I am using SAM CLI 1.6.2 on Windows 10. Can you please retry after updating SAM?

Thanks

[jeremlvt]

Hi @c2tarun, thank you for your answer.

I have updated SAM to 1.6.2 but I still get the same error when I deploy the stack :

Model validation failed (#/StartingPosition: failed validation constraint for keyword [pattern])

Did you use AT_TIMESTAMP as starting position ?

[c2tarun]

Ohh I didn't realize that the Default value for StartingPosition is not AT_TIMESTAMP, thanks for pointing this out. I have simplified that template a bit to add context here:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-aws-kinesis-lambda

Globals:
  Function:
    Timeout: 5
    Runtime: nodejs12.x
    MemorySize: 128
    Tracing: Active
    AutoPublishAlias: live

Resources:
  KinesisTriggerFunction:
    Type: "AWS::Serverless::Function"
    Properties:
      CodeUri: hello-world/
      Handler: trigger.handler
      Description: >
        This function triggered when a file is uploaded in a stream (Kinesis)
      Events: 
        Stream:
          Type: Kinesis
          Properties:
            Stream: !GetAtt KinesisStream.Arn
            BatchSize: 500
            StartingPosition: AT_TIMESTAMP
            ParallelizationFactor: 1
            MaximumRetryAttempts: 1000
            BisectBatchOnFunctionError: true
      Policies:
        - KinesisStreamReadPolicy:
            StreamName: !Ref KinesisStream

  KinesisStream: 
    Type: AWS::Kinesis::Stream 
    Properties: 
        Name: KinesisStream
        RetentionPeriodHours: 24
        ShardCount: 1
        StreamEncryption:
            EncryptionType: KMS 
            KeyId: alias/aws/kinesis

When trying to deploy this template I get following error from CFN:

Properties validation failed for resource KinesisTriggerFunctionStream with message: #/StartingPosition: failed validation constraint for keyword [pattern]

On searching a bit I found this github issue where someone claims to have an internal ticket for the issue. There is no update on that since then.

Like OP mentioned, CFN does not support AT_TIMESTAMP for Kinesis here.

[c2tarun]

This issue should be resolved by either Cloudformation team or Kinesis team working with CFN team. I am not able to find any monitored issues queue from CFN team on Github.
I have created an internal issue with CFN team for this issue. Sadly there is nothing more we can do here to fix this issue. I am resolving this issue now. Please feel free to send a feedback to CFN team by internal AWS tickets.

[awsjeffg]

Additionally, I've filed a ticket to remove the reference to AT_TIMESTAMP in the sam docs, bringing them in line with what is supported in CloudFormation currently.