pulumi keeps reporting changes with AWS:CLOUDFRONT distribution when none happened (Typescript)

[johnny-v-cdg]

What happened?

Created an aws::cloudfront distribution, it provisioned correctly.

Later, when coding other resources and executing pulumi up --diff to provision them, pulumi detected changes to cloudfront where none happened, and the changes detected are updating values that are equal between themselves.

I created a second cloudfront distro. executed pulumi up --diff and got the new distro changes shown AND the same non existing changes to the first cloudfront distro again.

Immediately after all changes applied when creating the second cloudfront distro, and without making any change to any file, I did again pulumi up --diff.

Both distros now show the same non existing changes, at same environment variables, with the same values respective of each distro.

when I hit yes to apply changes it uploads those "changes" and the cloudfront distros redeploy.

---

pasting here the origins snippet from the typescript code

import * as aws from "@pulumi/aws";
// [...]
const S3DEVDist = new aws.cloudfront.Distribution("REDACTED-dev-distribution", {
// [...]
    origins: [{
        domainName: staticWebsiteDev.bucketRegionalDomainName,
        originId: S3DEVOrigin,
        connectionAttempts: 3,
        connectionTimeout: 10,
        s3OriginConfig: {
            originAccessIdentity: "",
        },
    }],
// [...]

---

pasting here the changes detected by pulumi up --diff:
REDACTED corresponds to our client name and is the same string

Previewing update (dev):
pulumi:pulumi:Stack: (same)
[urn=urn:pulumi:dev::devops_infra::pulumi:pulumi:Stack::devops_infra-dev]
~ aws:cloudfront/distribution:Distribution: (update)
[id=DISTRO-ID-dev]
[urn=urn:pulumi:dev::devops_infra::aws:cloudfront/distribution:Distribution::REDACTED-dev-distribution]
[provider=urn:pulumi:dev::devops_infra::pulumi:providers:aws::default_5_10_0::6bd6db8f-fd79-4f58-9916-3e0e29db167a]
~ origins: [
~ [0]: {
~ connectionAttempts: 3 => 3
~ connectionTimeout : 10 => 10
~ domainName : "REDACTED-dev.s3.amazonaws.com" => "REDACTED-dev.s3.amazonaws.com"
~ originId : "REDACTEDS3Dev" => "REDACTEDS3Dev"
}
]
~ aws:cloudfront/distribution:Distribution: (update)
[id=DISTRO-ID-Staging]
[urn=urn:pulumi:dev::devops_infra::aws:cloudfront/distribution:Distribution::REDACTED-staging-distribution]
[provider=urn:pulumi:dev::devops_infra::pulumi:providers:aws::default_5_10_0::6bd6db8f-fd79-4f58-9916-3e0e29db167a]
~ origins: [
~ [0]: {
~ connectionAttempts: 3 => 3
~ connectionTimeout : 10 => 10
~ domainName : "REDACTED-staging.s3.amazonaws.com" => "REDACTED-staging.s3.amazonaws.com"
~ originId : "REDACTEDS3Staging" => "REDACTEDS3Staging"
}
]
Resources:
~ 2 to update

as you can see it detects is has to update:

• connectionAttemps from 3 to 3
• connectionTimeout from 10 to 10
• domainName from same-string to same-string
• originId from same-string to same-string

Steps to reproduce

1. create a aws:cloudfront distro
2. deploy it
3. immediately after execute pulimi up --diff
4. observe that pulumi tries to update a set of values that have not changed with those same values
   4.1 connectionAttempts: 3 => 3
   4.2 connectionTimeout: 10 => 10
   4.3 domainName: domain-string-123 => domain-string-123
   4.4 originId: origin-id-abcd => origin-id-abcd

Expected Behavior

nothing changed => nothing should be updated

Actual Behavior

nothing changed => pulumi shows non existing updates and applies them causing resources to redeploy

Output of pulumi about

CLI          
Version      3.37.2
Go Version   go1.17.12
Go Compiler  gc

Plugins
NAME        VERSION
aws           5.10.0
docker       3.4.0
eks             0.41.2
kubernetes  3.20.3
nodejs      unknown

Host     
OS       ubuntu
Version  22.04
Arch     x86_64

This project is written in nodejs: executable='/home/USER/.nvm/versions/node/v18.6.0/bin/node' version='v18.6.0'

Current Stack: dev

Dependencies:
NAME                VERSION
@types/node         18.6.5
@pulumi/aws         5.10.0
@pulumi/awsx        0.40.0
@pulumi/eks         0.41.2
@pulumi/kubernetes  3.20.3
@pulumi/pulumi      3.37.2

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[ringods]

When using v6.29.0 of this package, the results are slightly better:

1. Using pulumi up --diff shows no changes for me when I didn't make any changes
2. when making a change, the diff shows too many properties marked for update. For example, I only changed connectionAttempts property`, but the diff is like this:

  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:ringo::4969::pulumi:pulumi:Stack::4969-ringo]
    ~ aws:cloudfront/distribution:Distribution: (update)
        [id=<redacted>]
        [urn=urn:pulumi:ringo::4969::aws:cloudfront/distribution:Distribution::cdn]
        [provider=urn:pulumi:ringo::4969::pulumi:providers:aws::default_6_29_0::67ee5990-60c8-49fe-a222-8be2b3080e68]
      ~ origins: [
          ~ [0]: {
                  ~ connectionAttempts: 3 => 2
                  ~ connectionTimeout : 10 => 10
                  ~ domainName        : "bucket-<redacted>.s3.eu-west-1.amazonaws.com" => "bucket-<redacted>.s3.eu-west-1.amazonaws.com"
                  ~ originId          : "arn:aws:s3:::bucket-<redacted>" => "arn:aws:s3:::bucket-<redacted>"
                  ~ s3OriginConfig    : {
                      + __defaults          : []
                      ~ originAccessIdentity: "origin-access-identity/cloudfront/<redacted>" => "origin-access-identity/cloudfront/<redacted>"
                      ~ originAccessIdentity: "origin-access-identity/cloudfront/<redacted>" => "origin-access-identity/cloudfront/<redacted>"
                    }
                }
        ]
Resources:
    ~ 1 to update
    6 unchanged

Although I redacted a number of property values, the old and new values didn't differ.

[corymhall]

I've confirmed that this same issue occurs with Terraform and it looks like there is an upstream issue hashicorp/terraform-provider-aws#10526

[t0yv0]

In my current understanding the original issue as stated is fixed but the comment by @ringods still reproduces on latest, and is caused by pulumi/pulumi-terraform-bridge#186 which we really should fix, I'm marking this blocked for now.

[t0yv0]

One possible workaround we could take here specifically for this resource is that we could exclude the following parameters from the element key computation, diverging from how upstream provider handles it:

connectionAttempts: 3,
connectionTimeout: 10,

If we do this, changing connectionAttempts will not change the identity of the origin struct anymore and the diff will be clearer. The underlying assumption is that these were added upstream in a hurry without thinking through the implications, and in fact, added in error to the Set function identity computation.

[corymhall]

I was able to get a better repro showing a diff on every run

  const bucket = new aws.s3.BucketV2('chall-web-bucket', { });
  new aws.cloudfront.Distribution('chall-distro', {
    enabled: false,
    viewerCertificate: {
      cloudfrontDefaultCertificate: true,
    },
    restrictions: {
      geoRestriction: {
        restrictionType: 'none',
      },
    },
    defaultCacheBehavior: {
      defaultTtl: 0,
      maxTtl: 0,
      cachedMethods: ['HEAD', 'GET'],
      allowedMethods: ['HEAD', 'GET'],
      targetOriginId: 'chall-origin',
      viewerProtocolPolicy: 'https-only',
      forwardedValues: {
        queryString: false,
        cookies: {
          forward: 'none'
        },
      }
    },
    origins: [
      {
        customHeaders: [{
          name: 'key',
          value: 'value',
        }],
        originId: 'chall-origin',
        domainName: bucket.bucketDomainName,
        connectionTimeout: 10,
        connectionAttempts: 3,
        s3OriginConfig: {
          originAccessIdentity: "",
        },
      },
      {
        customHeaders: [{
          name: 'key',
          value: 'value',
        }],
        originId: 'chall-origin2',
        domainName: bucket.bucketDomainName,
        connectionTimeout: 10,
        connectionAttempts: 3,
        customOriginConfig: {
          httpPort: 80,
          httpsPort: 443,
          originReadTimeout: 30,
          originKeepaliveTimeout: 5,
          originSslProtocols: ['TLSv1.2'],
          originProtocolPolicy: 'https-only',
        },
      },
      {
        customHeaders: [{
          name: 'key',
          value: 'value',
        }],
        originId: 'chall-origin3',
        domainName: bucket.bucketDomainName,
        connectionTimeout: 10,
        connectionAttempts: 3,
        customOriginConfig: {
          httpPort: 80,
          httpsPort: 443,
          originReadTimeout: 30,
          originKeepaliveTimeout: 5,
          originSslProtocols: ['TLSv1.2'],
          originProtocolPolicy: 'https-only',
        },
      },
    ],
  })

Every diff shows something like

~ pulumi:pulumi:Stack: (refresh)
    [urn=urn:pulumi:dev::pulumi-typescript-app::pulumi:pulumi:Stack::pulumi-typescript-app-dev]
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::pulumi-typescript-app::pulumi:pulumi:Stack::pulumi-typescript-app-dev]
    ~ aws:cloudfront/distribution:Distribution: (update)
        [id=EQZ4LAW0MD4V9]
        [urn=urn:pulumi:dev::pulumi-typescript-app::aws:cloudfront/distribution:Distribution::chall-distro]
        [provider=urn:pulumi:dev::pulumi-typescript-app::pulumi:providers:aws::default_6_32_0::ba7105b7-0c96-48e7-b1dc-3a704bda0db0]
      ~ origins: [
          ~ [0]: {
                  + connectionAttempts: 3
                  + connectionTimeout : 10
                  ~ customHeaders     : [
                      ~ [0]: {
                              + __defaults: []
                                name      : "key"
                              + value     : "value"
                              + value     : "value"
                            }
                    ]
                  + domainName        : "web-bucket-c4bc447.s3.amazonaws.com"
                  + originId          : "chall-origin"
                  + s3OriginConfig    : {
                      + originAccessIdentity: ""
                    }
                }
          ~ [1]: {
                  - connectionAttempts: 3
                  - connectionTimeout : 10
                  ~ customHeaders     : [
                      ~ [0]: {
                              + __defaults: []
                                name      : "key"
                              - value     : "value"
                              - value     : "value"
                            }
                    ]
                  - domainName        : "web-bucket-c4bc447.s3.amazonaws.com"
                  - originId          : "chall-origin"
                }
        ]
Resources:
    ~ 1 to update
    2 unchanged

[t0yv0]

This is very helpful to have a repro, thanks so much! At a glance there's a few things still involved here:

pulumi/pulumi-terraform-bridge#186 is in play obfuscating the actual diff

pulumi/pulumi-terraform-bridge#1927 (and enrolling the resource in the PlanResourceChange flag) may be worth trying here as it fixed some very similar-looking issues with WafV2 resources

[corymhall]

I was able to reproduce this in Terraform as well so there is probably an upstream issue.

[corymhall]

Actually the issue goes away if I provide a s3OriginConfig.originAccessIdentity. So I'm back to being unable to reproduce a constant diff.

  const bucket = new aws.s3.BucketV2('chall-web-bucket', { });
  const id = new aws.cloudfront.OriginAccessIdentity('identity', {
    comment: 'chall-id',
  });
  new aws.cloudfront.Distribution('chall-distro', {
    enabled: false,
    viewerCertificate: {
      cloudfrontDefaultCertificate: true,
    },
    restrictions: {
      geoRestriction: {
        restrictionType: 'none',
      },
    },
    defaultCacheBehavior: {
      defaultTtl: 0,
      maxTtl: 0,
      cachedMethods: ['HEAD', 'GET'],
      allowedMethods: ['HEAD', 'GET'],
      targetOriginId: 'chall-origin',
      viewerProtocolPolicy: 'https-only',
      forwardedValues: {
        queryString: false,
        cookies: {
          forward: 'none'
        },
      }
    },
    origins: [
      {
        customHeaders: [{
          name: 'key',
          value: 'value',
        }],
        originId: 'chall-origin',
        domainName: bucket.bucketDomainName,
        connectionTimeout: 10,
        connectionAttempts: 3,
        s3OriginConfig: {
          originAccessIdentity: id.cloudfrontAccessIdentityPath,
        },
      },
      {
        customHeaders: [{
          name: 'key',
          value: 'value',
        }],
        originId: 'chall-origin2',
        domainName: bucket.bucketDomainName,
        connectionTimeout: 10,
        connectionAttempts: 3,
        customOriginConfig: {
          httpPort: 80,
          httpsPort: 443,
          originReadTimeout: 30,
          originKeepaliveTimeout: 5,
          originSslProtocols: ['TLSv1.2'],
          originProtocolPolicy: 'https-only',
        },
      },
      {
        customHeaders: [{
          name: 'key',
          value: 'value',
        }],
        originId: 'chall-origin3',
        domainName: bucket.bucketDomainName,
        connectionTimeout: 10,
        connectionAttempts: 3,
        customOriginConfig: {
          httpPort: 80,
          httpsPort: 443,
          originReadTimeout: 30,
          originKeepaliveTimeout: 5,
          originSslProtocols: ['TLSv1.2'],
          originProtocolPolicy: 'https-only',
        },
      },
    ],
  })

[corymhall]

Ok after looking into this more I think this should be changed to an enhancement. There were two possible issues being tracked here:

1. Diff shown even when no changes were made to origins
2. When any properties are changed on an individual origin, the diff shows every property, not just the one changing.

The first issue I believe is fixed since it can no longer be reproduced.

For #2 this is the way that Terraform works and it doesn't appear that there is a way to change it (See this comment). And actually our handling of it is slightly better than Terraform. In Terraform if we change a single property of a single origin the diff will show all the origins being removed and all being added again. Pulumi handles this slightly better and will only show a diff for the origin that is changing. We might be able to further improve it via 186, but that is a larger enhancement effort.

[mikhailshilkov]

@corymhall My suggestion is to keep this issue as a bug, but to actually close it, while moving the enhancement in a separate issue. It's a good practice against scope creep, and also on the way we track customer requests. What do you think?

[corymhall]

@mikhailshilkov make sense to me!

[mikhailshilkov]

@corymhall Great! To double-check: will you open the new issue? Please link these two together once you do. TY!

[mjeffryes]

I believe Present clearer diff for single element changes to sets· pulumi-terraform-bridge/186 is the relevant enhancement ticket