Property PublicAccessBlockConfiguration not defined

[yvele]

I'm using the very new AWS::S3::Bucket property PublicAccessBlockConfiguration

I also have the latest AWS SDK and aws-cli version: aws-cli/1.16.60 Python/3.7.1 Darwin/16.7.0 botocore/1.12.50

My template looks like:

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31

Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      PublicAccessBlockConfiguration:
        BlockPublicAcls       : true
        BlockPublicPolicy     : true
        IgnorePublicAcls      : true
        RestrictPublicBuckets : true

And I'm having the following error:

Failed to create the changeset:
Waiter ChangeSetCreateComplete failed:
Waiter encountered a terminal failure state Status: FAILED.
Reason: Transform AWS::Serverless-2016-10-31 failed with:
 Invalid Serverless Application Specification document.
 Number of errors found: 1.
 Resource with id [Bucket] is invalid.
 property PublicAccessBlockConfiguration not defined for resource of type AWS::S3::Bucket

This looks like an AWS::Serverless-2016-10-31 transform error? Or am I wrong? 🤔

Edit: I've updated many buckets with PublicAccessBlockConfigurationand everything looks fine except when the CloudFormation template uses Transform: AWS::Serverless-2016-10-31 and the S3 is a CloudFormation dependency 🤔

Note also than I'm doing aws cloudformation package prior to aws cloudformation deploy

[samallisonds]

FWIW... that template works fine for me in us-west-2

[yvele]

Thanks. I'm in eu-west-1 could it be possible that Europe is late?

[samallisonds]

Sorry, you're not going to like this... it works there for me as well!

I'm just creating the stack manually through cloudformation. I guess if you're doing something with the Serverless Application Repository thing that might be different

[yvele]

Are you using AWS::Serverless-2016-10-31 transformation?

[samallisonds]

Yes - I just pasted your template snippet above into the template designer

[yvele]

Ok then I'm closing this, and I will take a rest this week end 😉 I'll keep you updated here when I fix the problem

Thanks for your support

[yvele]

Ok I ran lot's of tests, and it looks like everything goes fine for newly created buckets, but I got an error when I'm trying to update PublicAccessBlockConfiguration on old buckets (created a month ago, before AWS announcement)

[yvele]

I've updated many buckets with PublicAccessBlockConfigurationand everything looks fine except when the CloudFormation template uses Transform: AWS::Serverless-2016-10-31 and the S3 is a CloudFormation dependency 🤔

Note also than I'm doing aws cloudformation package prior to aws cloudformation deploy

[keetonian]

@yvele The template you posted works fine for me both when I deploy it and package+deploy it. Is there another dependency or other complexity in the template that is failing to deploy?

[yvele]

@keetonian sure my real template is larger and has Lambda, Custom resource, a CloudFront distribution, Policies, etc.

The problem is with CloudFront distribution, because it takes 40 minutes to fully deploy and it make it harder to test by elimination.

My template, anonymised and with redundant things removed looks like this (click me)

AWSTemplateFormatVersion: 2010-09-09
Description: Description
Transform: AWS::Serverless-2016-10-31


Parameters:
  RootDomain:
    Description: Domain name
    Default: domain.com
    Type: String
  Subdomain:
    Description: Subdomain name
    Default: dist
    Type: String
  ContentTypeLabel:
    Description: Content type short label
    Type: String
  ApplicationCategory:
    Description: Application category (tag value for xx:application:category)
    Default: distribution
    Type: String
  Environment:
    Description: Environment
    Default: sandbox
    Type: String
    AllowedValues: [production, staging, test, development, sandbox]
  CloudFrontZoneId:
    Description: CloudFront Zone ID for Route 53 record set
    Default: Z2FDTNDATAQYW2 # This ID is always the same to all AWS accounts
    Type: String
  BundleVersion:
    Description: 'Bundle version (e.g "1.0.0", "1.2.3-beta.1", etc.)'
    Type: String

Mappings:
  Environment:
    production:
      LabelSuffix: ""
      RootDomainPrefix: ""
      CertificateArn: "arn:aws:acm:us-east-1:xxxxx:certificate/xxxxx"
      DistributionPriceClass: PriceClass_All
      S3OriginAccessIdentity: origin-access-identity/cloudfront/xxxxxxxxxxxx
      S3CanonicalUser: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    development:
      LabelSuffix: " (development)"
      RootDomainPrefix: "dev."
      CertificateArn: "arn:aws:acm:us-east-1:xxxxx:certificate/xxxxx"
      DistributionPriceClass: PriceClass_100
      S3OriginAccessIdentity: origin-access-identity/cloudfront/xxxxxxxxxxxx
      S3CanonicalUser: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx


Resources:

  Bucket:
    Type: AWS::S3::Bucket
    # Only Amazon S3 buckets that are empty can be deleted. Deletion will fail for buckets that have contents
    # Deletion will fail if bucket is not empty
    DeletionPolicy: Delete
    Properties:
      BucketName: !Sub
        - ${Subdomain}.${RootDomainPrefix}${RootDomain}
        - Subdomain         : !Ref Subdomain
          RootDomainPrefix  : !FindInMap [Environment, !Ref Environment, RootDomainPrefix]
          RootDomain        : !Ref RootDomain
      # Restrict access
      AccessControl: Private
      PublicAccessBlockConfiguration:
        BlockPublicAcls       : true
        BlockPublicPolicy     : true
        IgnorePublicAcls      : true
        RestrictPublicBuckets : true
      # CORS pass through
      CorsConfiguration:
        CorsRules:
          - Id: BucketCORSRules
            AllowedMethods: [HEAD, GET] # OPTIONS is implicit
            AllowedOrigins: ["*"]
            MaxAge: 3600
      # Keep deleted objects 30 days to allow reverting accidental deletion
      VersioningConfiguration:
        Status: Enabled
      LifecycleConfiguration:
        Rules:
          - Id: PermanentDeletion
            Status: Enabled
            NoncurrentVersionExpirationInDays: 30


  BucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      PolicyDocument:
        Id: BucketPolicy
        Version: 2012-10-17
        Statement:
          - Sid: ReadAccess
            Effect: Allow
            Principal:
              CanonicalUser: !FindInMap [Environment, !Ref Environment, S3CanonicalUser]
            Action:
              - s3:GetObject
            Resource: !Sub
              - arn:aws:s3:::${Domain}/*
              - Domain: !Ref Bucket
      Bucket: !Ref Bucket


  FunctionCompress:
    Type: AWS::Serverless::Function
    Properties:
      ReservedConcurrentExecutions: 25
      AutoPublishAlias: live
      Description: !Sub
        - s3-compression v${BundleVersion}
        - BundleVersion: !Ref BundleVersion
      Handler     : main.handle
      Runtime     : nodejs8.10
      MemorySize  : 2048 # Mo
      Timeout     : 300 # 5 minutes
      CodeUri     : ../packages/lambda-s3-compression/dist
      Tracing     : Active # Enable X-Ray tracing
      Policies:
        - Version: 2012-10-17
          Statement:
            - Effect  : Allow
              Action  :
                - s3:GetObject
                - s3:PutObject
                - s3:DeleteObject
              # NOTE: Cannot directly reference Bucket because we get CloudFormation circular dependencies error
              Resource: !Sub
                - arn:aws:s3:::${Subdomain}.${RootDomainPrefix}${RootDomain}/*
                - Subdomain         : !Ref Subdomain
                  RootDomainPrefix  : !FindInMap [Environment, !Ref Environment, RootDomainPrefix]
                  RootDomain        : !Ref RootDomain
      Events:
        S3RawCss  :
          Type    : S3
          Properties:
            Bucket    : !Ref Bucket
            Events    : ["s3:ObjectCreated:*", "s3:ObjectRemoved:*"]
            Filter    :
              S3Key   :
                Rules :
                  - { Name: prefix, Value: "prefix/" }
                  - { Name: suffix, Value: "suffix" }



  ##############################################################################
  ###                                                                        ###
  ###                       CloudFront Distribution                          ###
  ###                                                                        ###
  ##############################################################################

  Distribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Enabled: true
        # CNAMEs (alternate domain names)
        # http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/CNAMEs.html
        Aliases: [!Ref Bucket]
        Origins:
          # The DNS name of the Amazon Simple Storage Service (S3) bucket or the HTTP server from which you want CloudFront to get objects for this origin.
          - DomainName: !GetAtt Bucket.DomainName
            Id: Bucket # An identifier for the origin. The value of Id must be unique within the distribution
            S3OriginConfig:
              OriginAccessIdentity: !FindInMap [Environment, !Ref Environment, S3OriginAccessIdentity]
        Comment: !Sub
          - ${ContentTypeLabel}${Suffix}
          - ContentTypeLabel  : !Ref ContentTypeLabel
            Suffix            : !FindInMap [Environment, !Ref Environment, LabelSuffix]
        DefaultRootObject: index.html
        CustomErrorResponses:
          - ErrorCode: 400 # Bad Request
            ResponseCode: 400
            ResponsePagePath: /errors/400.html
            ErrorCachingMinTTL: 157680000 # 5 years
          - ErrorCode: 403 # Forbidden
            ResponseCode: 403
            ResponsePagePath: /errors/403.html
            ErrorCachingMinTTL: 157680000 # 5 years
          - ErrorCode: 404 # Not Found
            ResponseCode: 404
            ResponsePagePath: /errors/404.html
            ErrorCachingMinTTL: 157680000 # 5 years
          - ErrorCode: 405 # Method Not Allowed
            ResponseCode: 404 # Not Found
            ResponsePagePath: /errors/404.html
            ErrorCachingMinTTL: 157680000 # 5 years
          - ErrorCode: 414 # URI Too Long
            ResponseCode: 404 # Not Found
            ResponsePagePath: /errors/404.html
            ErrorCachingMinTTL: 157680000 # 5 years
          - ErrorCode: 500 # Bad Request
            ResponseCode: 500
            ResponsePagePath: /errors/500.html
            ErrorCachingMinTTL: 300 # 5 minutes
          - ErrorCode: 501 # Not Implemented
            ResponseCode: 500 # Bad Request
            ResponsePagePath: /errors/500.html
            ErrorCachingMinTTL: 300 # 5 minutes
          - ErrorCode: 502 # Bad Gateway
            ResponseCode: 502
            ResponsePagePath: /errors/502.html
            ErrorCachingMinTTL: 300 # 5 minutes
          - ErrorCode: 503 # Service Unavailable
            ResponseCode: 503
            ResponsePagePath: /errors/503.html
            ErrorCachingMinTTL: 300 # 5 minutes
          - ErrorCode: 504 # Gateway Time-out
            ResponseCode: 504
            ResponsePagePath: /errors/504.html
            ErrorCachingMinTTL: 300 # 5 minutes
        DefaultCacheBehavior:
          TargetOriginId  : Bucket    # Uniquely distinguishes this origin from other origins in this distribution
          MinTTL          : 157680000 # 5 years
          MaxTTL          : 157680000 # 5 years
          DefaultTTL      : 157680000 # 5 years
          Compress        : false     # Compression is already managed at origin (using S3 Lambda events)
          AllowedMethods  : [OPTIONS, HEAD, GET]
          CachedMethods   : [OPTIONS, HEAD, GET]
          ViewerProtocolPolicy : redirect-to-https # Valid values: allow-all, https-only, redirect-to-https
          ForwardedValues : # Specifies how CloudFront handles query strings or cookies
            QueryString : false
            Cookies     : { Forward: none }
            Headers     :
              - Origin          # Usefull for CORS
              - Accept-Encoding # Usefull for compression
          # Lambda@Edge configuration requires a function version not alias
          LambdaFunctionAssociations:
            - EventType: origin-request
              LambdaFunctionARN: !GetAtt EdgeOriginRequest.FunctionVersion

        PriceClass: !FindInMap [Environment, !Ref Environment, DistributionPriceClass]
        HttpVersion: http2
        IPV6Enabled: true
        ViewerCertificate:
          # Note: Currently, you can specify only certificates that are in the US East (N. Virginia) region !!!!
          AcmCertificateArn: !FindInMap [Environment, !Ref Environment, CertificateArn]
          # WARNING: VIP means "All clients" and AWS charge a fixed monthly fee of $600
          SslSupportMethod: sni-only # sni-only | vip ($600!)
        Logging:
          IncludeCookies: false
          Bucket: !Sub
            - ${Bucket}.s3.amazonaws.com
            - Bucket: !Ref BucketLogs
          Prefix: cloudfront

  RecordSetGroup:
    Type: AWS::Route53::RecordSetGroup
    Properties:
      Comment: !Sub
        - ${ContentTypeLabel} dist${Suffix}
        - ContentTypeLabel  : !Ref ContentTypeLabel
          Suffix            : !FindInMap [Environment, !Ref Environment, LabelSuffix]
      HostedZoneId: !ImportValue MyHostedZoneId
      RecordSets:
        # IPv4
        - Type  : A
          Name  : !Ref Bucket
          AliasTarget: # Specific to AWS
            HostedZoneId  : !Ref CloudFrontZoneId
            DNSName       : !GetAtt Distribution.DomainName # CloudFront distribution domain name
        # IPv6
        - Type  : AAAA
          Name  : !Ref Bucket
          AliasTarget: # Specific to AWS
            HostedZoneId  : !Ref CloudFrontZoneId
            DNSName       : !GetAtt Distribution.DomainName # CloudFront distribution domain name


  # Origin/Request - AWS Lambda Function
  EdgeOriginRequestSource:
    Type: AWS::Serverless::Function
    Properties:
      Description: !Sub
        - edge-origin-request v${BundleVersion}
        - BundleVersion: !Ref BundleVersion
      Handler     : main.handle
      Runtime     : nodejs8.10
      MemorySize  : 2048 # Mo
      CodeUri     : ../packages/lambda-edge-origin-request/dist
      Tracing     : Active # Enable X-Ray tracing
      AutoPublishAlias: live # Required to get `Version` parameter

  # Origin/Request - Custom resource to "copy" the Lambda in the standard region (us-east-1)
  EdgeOriginRequest:
    Type: Custom::LambdaEdge
    Properties:
      ServiceToken: !ImportValue CustomResourceLambdaEdgeServiceToken
      Parameters:
        LambdaSourceArn : !Ref EdgeOriginRequestSource.Version

  ##############################################################################
  ###                                                                        ###
  ###                                Logs                                    ###
  ###                                                                        ###
  ##############################################################################

  BucketLogs:
    Type: AWS::S3::Bucket
    DeletionPolicy: Delete # Deletion will fail if bucket is not empty
    Properties:
      BucketName: !Sub
        - logs.${Subdomain}.${RootDomainPrefix}${RootDomain}
        - Subdomain         : !Ref Subdomain
          RootDomainPrefix  : !FindInMap [Environment, !Ref Environment, RootDomainPrefix]
          RootDomain        : !Ref RootDomain
      # Restrict access
      AccessControl: Private
      PublicAccessBlockConfiguration:
        BlockPublicAcls       : true
        BlockPublicPolicy     : true
        IgnorePublicAcls      : true
        RestrictPublicBuckets : true
      # Lifecycle
      VersioningConfiguration:
        Status: Enabled
      LifecycleConfiguration:
        Rules:
          - Id: Expiration
            Status: Enabled
            ExpirationInDays: 60 # After 60 days logs are deleted forever
            NoncurrentVersionExpirationInDays: 10 # 10 days to allow reverting accidental deletion
            AbortIncompleteMultipartUpload:
              DaysAfterInitiation: 1

  BucketLogsPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref BucketLogs
      PolicyDocument:
        Id: BucketPolicy
        Version: 2012-10-17
        Statement:
          - Sid: FullControl
            Effect: Allow
            Principal:
              # This is awsdatafeeds account
              # See https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html#AccessLogsBucketAndFileOwnership
              CanonicalUser: c4c1ede66af53448b93c283ce9448c4ba468c9432aa01d700d3878632f77d2d0
            Action:
              - s3:*
            Resource: !Sub
              - arn:aws:s3:::${Bucket}/*
              - Bucket: !Ref BucketLogs

  # AWS Lambda Function that put cloudfront logs into cloudwatch
  FunctionLogs:
    Type: AWS::Serverless::Function
    Properties:
      ReservedConcurrentExecutions: 10
      AutoPublishAlias: live
      Description: !Sub
        - cloudfront-logs v${BundleVersion}
        - BundleVersion: !Ref BundleVersion
      Handler     : main.handle
      Runtime     : nodejs8.10
      MemorySize  : 2048 # Mo
      CodeUri     : ../packages/lambda-cloudfront-logs/dist
      Tracing     : Active # Enable X-Ray tracing
      Policies:
        - Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action: cloudwatch:PutMetricData
              Resource: "*"
            - Effect: Allow
              Action:
                - s3:GetObject
              # NOTE: Cannot directly reference Bucket because we get CloudFormation circular dependencies error
              Resource: !Sub
                - arn:aws:s3:::logs.${Subdomain}.${RootDomainPrefix}${RootDomain}/*
                - Subdomain         : !Ref Subdomain
                  RootDomainPrefix  : !FindInMap [Environment, !Ref Environment, RootDomainPrefix]
                  RootDomain        : !Ref RootDomain
      Events:
        S3LogsListener:
          Type: S3
          Properties:
            Bucket: !Ref BucketLogs
            Events: s3:ObjectCreated:*
            Filter:
              S3Key:
                Rules:
                  - Name: suffix
                    Value: .gz
                  - Name: prefix
                    Value: cloudfront/

[keetonian]

Ok, here's what it looks like:

When SAM adds an event to an S3 bucket, it tries to load that bucket into an internal S3 bucket class which has all the accepted parameters hard coded. I submitted a fix in #700, but this brings up a good thing to think about- I would rather not need to manually update these resources every time some small change is made to them.

[yvele]

So it's not related to CloudFront but to S3 events, thanks for your explanation and your PR

[vasylk]

+1 can confirm the same behaviour in eu-central-1. PublicAccessBlockConfiguration is not defined

[keetonian]

Correct. We will update this issue when this is available. You can expect it with the upcoming v1.10.0 release

[rtjarvis]

I get this issue in eu-west-2 too.

[keetonian]

Released!

[vasylk]

Thanks, guys!