[BUG] Validate and Test actions on same CloudFormation produce different outputs

[0x10F8]

Describe the bug
It appears that validate and test evaluate differently when using references within Cloud Formation.

To Reproduce
Run the following guard rule against the provided test and CloudFormation yaml. The test fails but the CloudFormation validate passes although they both contain effectively the same resources.

The example in this case checks for HTTP block on S3 buckets

Guard:

# Select all bucket resources
let s3_buckets = Resources[ bucket_resource_name | Type == 'AWS::S3::Bucket' ]
let s3_bucket_policies = Resources.*[ Type == 'AWS::S3::BucketPolicy' ]

#
# Rule Intent
# ----
#
# The S3 bucket should not allow HTTP traffic. Ensure an appropriate
# bucket policy is attached which disables HTTP.
#
# Outcome:
# SKIP: when there are no S3 buckets
# FAIL: when the bucket allows HTTP get and put
# PASS: when the bucket does not allow HTTP get and put
#
rule s3_bucket_nohttp when %s3_buckets !empty {
    # Check that bucket policy is attached to bucket
    let s3_bucket_refs = some %s3_bucket_policies.Properties.Bucket.Ref
    %s3_bucket_refs not empty
    %bucket_resource_name == %s3_bucket_refs

    %s3_bucket_policies.Properties.PolicyDocument {
        some Statement[*] {
            Condition {
                Bool {
                    'aws:SecureTransport' == false
                }
            }
            some Action[*] == 's3:GetObject'
            some Action[*] == 's3:PutObject'
            Effect == 'Deny'
            Principal == '*'
        }
    }
}

Test Case:

- name: "Test blocked HTTP succeeds"
  input:
    Resources:
      S3Bucket:
        Type: "AWS::S3::Bucket"
        Properties:
          BucketName: test-sandbox-eu-west-1-example-s3-bucket
      S3BucketPolicy:
        DependsOn:
          - S3Bucket
        Type: 'AWS::S3::BucketPolicy'
        Properties:
          Bucket: !Ref S3Bucket
          PolicyDocument:
            Statement:
              - Action:
                  - 's3:GetObject'
                  - 's3:PutObject'
                Condition:
                  Bool:
                    'aws:SecureTransport': false
                Effect: Deny
                Principal: '*'
                Resource: !Sub 'arn:${AWS::Partition}:s3:::${S3Bucket}/*'
                Sid: DenyUnencryptedTransport
  expectations:
    rules:
      s3_bucket_nohttp: PASS

CloudFormation

AWSTemplateFormatVersion: "2010-09-09"
Resources:
  S3Bucket:
    Type: "AWS::S3::Bucket"
    Properties:
      BucketName: test-sandbox-eu-west-1-example-s3-bucket
  S3BucketPolicy:
    DependsOn:
      - S3Bucket
    Type: 'AWS::S3::BucketPolicy'
    Properties:
      Bucket: !Ref S3Bucket
      PolicyDocument:
        Statement:
          - Action:
              - 's3:GetObject'
              - 's3:PutObject'
            Condition:
              Bool:
                'aws:SecureTransport': false
            Effect: Deny
            Principal: '*'
            Resource: !Sub 'arn:${AWS::Partition}:s3:::${S3Bucket}/*'
            Sid: DenyUnencryptedTransport

Commands run:
cat cloudformation-example.yml | cfn-guard validate -r s3-tls.guard
cfn-guard test --rules-file s3-tls.guard --test-data s3-tls-tests.yml -v

The validate passes but the test fails.

Expected behavior
I believe the test should pass in this example?

Operating System:
Amazon Linux 2

OS Version
2

Additional context
None

[joshfried-aws]

Hi @0x10F8 thanks for pointing this out. This was recently brought to our attention. We currently have a PR up that has a proposed fix for this regression. We are actively working to resolve this issue.

#300

Thanks,
Josh

[joshfried-aws]

Closing this issue since 2.1.2 has now been released and this issue was addressed. Feel free to reopen this issue if need be.

Thanks,
Josh

[kimengu-david]

Hi @joshfried-aws, this bug is still present in the latest cloudformation guard version as of today ie 2.1.3

I tried this with the files provided by @0x10F8 on Amazon Linux 2, the test command is still failing.

Would you please reopen and addres the issue?

[joshfried-aws]

Hi @kimengu-david and @0x10F8 this has now been addressed in #331 . It will be a part of our next release. Feel free to reopen this issue if need be.