Cloudfront Looping into Deploying State when using cache behaviors

[kferrone]

What happened?

I have a cloudfront composition which will put the distribution in a constant loop into deploying. Everything was working fine until I added a key for cacheBehaviors in the CompositeDefinition which in turn would add to the same key in the Distribution which is created by the Composition. This was easier said than done.

Every single key in every object in the array of the incredibly detailed cacheBehaviors needed to be filled. No defaults at all.

It all seems to kinda work but it also seems every time Crossplane reconciles the resource, the Distribution will go back to deploying.

Here is what I had to do to get a single cache behavior as it defaults in the web console.

    cacheBehaviors:
      quantity: 1
      items:

      # so it seems all of these are required . . . 
      - pathPattern: /images/*
        targetOriginID: s3Origin
        viewerProtocolPolicy: allow-all
        smoothStreaming: false
        compress: false
        minTTL: 100
        maxTTL: 900
        defaultTTL: 100
        fieldLevelEncryptionID: ""
        lambdaFunctionAssociations:
          quantity: 0
          items: []
        trustedKeyGroups:
          enabled: false
          quantity: 0
        trustedSigners:
          enabled: false
          quantity: 0
        forwardedValues: 
          queryString: false
          cookies: 
            # all, whitelist, none
            forward: all
            whitelistedNames:
              quantity: 0
              items: []
          headers: 
            quantity: 0
            items: []
          queryStringCacheKeys:
            quantity: 0
            items: []
        allowedMethods:
          cachedMethods:
            items:
              - HEAD
              - GET
            quantity: 2
          items:
            - HEAD
            - GET
          quantity: 2

How can we reproduce it?

Here is what I wanted to do:

    cacheBehaviors:
      items:
      - pathPattern: /images/*
        targetOriginID: s3Origin

Use this snippet with a Cloudfront Distribution resource and see what happens.

I have shared the repo with @haarchri and other Crossplane members.
This one: https://github.com/Cubizoid/crosshook

To get the composition and xrd up from that project:

kubectl apply -k compositions/website

Use this as the example:

apiVersion: cubizoid.com/v1
kind: Website
metadata:
  name: cool-stack
spec:
  parameters:
    domain: app.example.com
    zone: example.com
    enabled: true
    description: |
      This directs static and dynamic 
      content to s3 bucket and ALB respectively.
    defaultRootObject: index.html
    bucket:
      path: /web
    loadBalancer:
      endpoint: foo-bar.us-west-2.elb.amazonaws.com
    cacheBehaviors:
      quantity: 1
      items:

      # i put defaults on all the keys in the definitions openapi schema
      - pathPattern: /images/*
        targetOriginID: s3Origin

[kferrone]

I commented out everything in the definition and composition relating to cacheBehaviors and the looping stopped. However, I expected the single cache behavior to be removed since I did remove it from the composition all together. Now it can't be deleted through crossplane. The cacheBehaviors on Distributions are pretty hard to work with at the moment.

[kferrone]

I feel this issue has something to do with trying to propagate an array of complex objects from my XRD resource into the composition. I honestly have had many issues with arrays of objects with their own nested arrays. If an an array of objects only has simple keys, ie no objects or arrays, then defining the array works pretty well.

In this case the cacheBehaviors array is very complex with many levels deep of objects with arrays of nested objects.

The effect in the web console I see is the Distribution completing, but in Kubernetes the Distribution just never says ready is true. I can kind of tell when the reconciliation loop comes around that Crossplane somehow sees the state it has as different from the state in AWS. Then it makes some kind of change every loop and therefore it is looping into deploying state.

[kferrone]

If I cut the cacheBehavior array from the XRD and paste it directly into the composition, then there are no issues. This clearly shows the issue has to do with passing the entire cacheBehaviors array from the XRD resource to the composition.

Also to note, the generation number on the Distribution is ticking up and up. Each time it increments, the Distribution goes back to deploying state.

[ezgidemirel]

Hello @kferrone, I opened #1186 which solves the issue you're experiencing. Please note that, viewerProtocolPolicy, minTTL and forwardedValues fields are required in CacheBehavior configuration.

Also, with this change, we're removing quantity fields from the CRD for having a better user experience. Which means you need to remove them from your manifests.

[haarchri]

tested the fix #1186 it is working

[kferrone]

Sweet! When is the next planned release?

[haarchri]

0.25 on 09.03