CloudFormation Registry resource schema support (--registry-schemas)

[PatMyron]

#1277

CloudFormation Registry announcement
Resource Schema repo

---

• jsonschema.validate -> RuleMatch format (only returns one error per resource + exact context isn't great, especially for combined JSON schemas)
  • investigate iterating all ValidationErrors per resource
• generalize beyond 1 resource type
• automate updating PUBLIC resource schemas (Get value constraints from AWS registry types #1867)
  • with caching
• handle CloudFormation template syntax
  • https://github.com/aws-cloudformation/cfn-python-lint/blob/338d5fb29a2f47069f0b1091fc420f6dd9e6013d/src/cfnlint/helpers.py#L77
  • https://github.com/aws-cloudformation/cfn-python-lint/blob/338d5fb29a2f47069f0b1091fc420f6dd9e6013d/src/cfnlint/helpers.py#L126
  • https://github.com/aws-cloudformation/cfn-python-lint/blob/338d5fb29a2f47069f0b1091fc420f6dd9e6013d/src/cfnlint/helpers.py#L66
  • https://github.com/aws-cloudformation/cfn-python-lint/blob/1a6657bafb5330ff8e68ae99538ff99f42bcd0bc/src/cfnlint/helpers.py#L65
  • JSON schema type checking stricter than CloudFormation's
  • Resource GetAtt referenced elsewhere: CloudFormation Registry resource support - GetAtt rules broken #2622
• allow non-PUBLIC resource schemas to be passed in
  • Update E3001 to include these resource types:https://github.com/aws-cloudformation/cfn-python-lint/blob/93659fba5cf86ee34177c55b28df0aa9140b7dc9/src/cfnlint/rules/resources/Configuration.py#L105
  • write list-types / describe-type script to make it easier:
    • need to fix CloudFormation paginators

import boto3
for page in boto3.client('cloudformation').get_paginator('list_types').paginate():
    for resource_type in page['TypeSummaries']:
        with open(resource_type['TypeName'].replace("::", "-").lower() + '.json', 'w') as f:
            print(boto3.client('cloudformation').describe_type(Arn=resource_type['TypeArn'])['Schema'], file=f)

[johnttompkins]

so this would assume all the potential schemas are downloaded?

[PatMyron]

Yeah, wrote list-types / describe-type script to make it easy, but don't think it makes sense to call every runtime due to

• credentials and networking requirements
• ability to provide additional resource schemas

[kddejong]

I'm still not sure the standard jsonschema packages are going to fit the CloudFormation functionality. We can expand the schema from the registry to support the intrinsic functions when we read them but the possibilities are massive.

[kddejong]

One of the big problems we are going to have with using this package is handling Fn::If or other intrinsic functions. Either way we are going to have to handle it outside of the schema validation which we have logic to handle or we are going to have to rewrite these validation functions that this package provides to handle these things differently.

[kddejong]

Looks like this says ignore intrinsic function errors but doesn't handle more complex Fn::If scenarios.

ResourceName:
   Type: AWS::CloudFormation::StackSet
   Properties:
     Fn::If:
     - ConditionName
     - PropertiesScenarioTrue
     - PropertiesScenarioFalse

We basically don't get a failure even though there could be one in the provided scenario.

[kddejong]

outside packages shouldn't be using run_cli by any concerns with the ordering here? I've been adding parameters and setting a default value to keep some backwards compatibility.

[PatMyron]

only found this from searching: https://github.com/Cloudzero/samwise/blob/793463e0774f9d1c702f4252377f19fa3408a558/samwise/__main__.py#L129 which looks already broken from #1411
cc @silvexis

[PatMyron]

have generally been debating compatibility for reliance on undocumented internal functions/variables

[kddejong]

Few changes but looks good to me

[PatMyron]

Example messages:

[cfn-lint] E3000 'RequiredEnumProperty' is a required property
[cfn-lint] E3000 Additional properties are not allowed ('InvalidProperty' was unexpected)
[cfn-lint] E3000 'string' is not of type 'integer'
[cfn-lint] E3000 200 is greater than the maximum of 100
[cfn-lint] E3000 -1 is less than the minimum of 0
[cfn-lint] E3000 'd' is not one of ['a', 'b', 'c']
[cfn-lint] E3000 'name' does not match 'arn:.*'

schema/template

{
  "typeName" : "Third::Party::Type",
  "description" : "",
  "additionalProperties" : false,
  "properties" : {
    "RequiredEnumProperty" : {
      "type" : "string",
      "enum": [
        "a",
        "b",
        "c"
      ]
    },
    "PatternProperty" : {
      "type" : "string",
      "pattern": "arn:.*"
    },
    "RangeProperty" : {
      "type": "integer",
      "minimum": 0,
      "maximum": 100
    }
  },
  "required" : ["RequiredEnumProperty"],
  "primaryIdentifier" : [ "/properties/RequiredEnumProperty" ]
}

Resources:
  ResourceMissingRequiredProperty:
    Type: Third::Party::Type
  ResourceMissingRequiredProperty2:
    Type: Third::Party::Type
    Properties:
      RangeProperty: 50
  ResourceWithInvalidProperty:
    Type: Third::Party::Type
    Properties:
      RequiredEnumProperty: required
      RangeProperty: 50
      InvalidProperty: invalid
  ResourceWithWrongPrimitiveType:
    Type: Third::Party::Type
    Properties:
      RequiredEnumProperty: a
      RangeProperty: string
  ResourceWithValueOutsideMinMaxRange:
    Type: Third::Party::Type
    Properties:
      RequiredEnumProperty: a
      RangeProperty: 200
  ResourceWithValueOutsideMinMaxRange2:
    Type: Third::Party::Type
    Properties:
      RequiredEnumProperty: a
      RangeProperty: -1
  ResourceWithInvalidValue:
    Type: Third::Party::Type
    Properties:
      RequiredEnumProperty: d
      RangeProperty: 50
  ResourceWithInvalidPattern:
    Type: Third::Party::Type
    Properties:
      RequiredEnumProperty: a
      PatternProperty: name
  ValidResource:
    Type: Third::Party::Type
    Properties:
      RequiredEnumProperty: a
      RangeProperty: 50
  ResourceWithUnresolvableSyntax:
    Type: Third::Party::Type
    Properties:
      RequiredEnumProperty: !Ref ValidResource
      RangeProperty: '{{resolve:secretsmanager:secret:secret:secret}}'

[jasonmimick]

Need to add this new parameter to the config file json schema:
cfnlint/data/CfnLintCli/config/schema.json
needs

...
 "ignore_templates": {
  79       "description": "Templates to ignore",
  80       "items": {
  81         "type": "string"
  82       },
  83       "type": "array"
  84     },
  85     "registry_schemas": {
  86       "description": "one or more directories of CloudFormation Registry Resource Schemas",
  87       "items": {
  88         "type": "string"
  89       },
  90       "type": "array"
  91     }