Manually changing CA certificate in AWS TLS inspection prevents further Pulumi updates

[aaronnguyenwh]

What happened?

When manually modifying the "CA certificate for outbound SSL/TLS inspection" in AWS TLS inspection configurations, Pulumi won't be able to make further changes to the CertificateAuthorityArn.

Example

const tlsInspectionConfiguration = new aws.networkfirewall.TlsInspectionConfiguration('egress', {
  name: 'egress',
  description: 'Configuration for outbound TLS inspection in Network Firewall',
  tlsInspectionConfiguration: {
    serverCertificateConfiguration: {
      certificateAuthorityArn: certArn,
      scopes: [
        {
          protocols: [6],
          destinationPorts: [
            {
              fromPort: 443,
              toPort: 443,
            },
          ],
          destinations: [
            {
              addressDefinition: '0.0.0.0/0',
            },
          ],
          sourcePorts: [
            {
              fromPort: 0,
              toPort: 65535,
            },
          ],
          sources: [
            {
              addressDefinition: '0.0.0.0/0',
            },
          ],
        },
      ],
    },
  },
});

Steps:

• Set certificateAuthorityArn to certArn1 via Pulumi
• Manually change the "CA certificate for outbound SSL/TLS inspection" in AWS TLS inspection configurations on AWS console to a different certArn2
• Re-run Pulumi to update the certificateAuthorityArn to certArn1

Expected Behavior: The CA Arn is changed to certArn1 in both Pulumi and AWS Console

Actual Behavior: Pulumi returns error

     Type                                               Name                                                Status                  Info
     pulumi:pulumi:Stack                             ....us-gov-west-1                                    **failed**                1 error
 ~   └─ aws:networkfirewall:TlsInspectionConfiguration  egress                                          **updating failed**     [diff: ~tlsInspectionConfiguration];
                    "__meta": "{\"...\":{\"create\":300000000000,\"delete\
Diagnostics:
  pulumi:pulumi:Stack (....us-gov-west-1):
    error: update failed
                    "destinationCidrBlock": "...",
  aws:networkfirewall:TlsInspectionConfiguration (tls-egress):
    error: updating NetworkFirewall TLS Inspection Configuration (arn:aws-us-gov:network-firewall:us-gov-west-1:...:tls-configuration/egress): operation error Network Firewall: UpdateTLSInspectionConfiguration, https response error StatusCode: 400, RequestID: ..., InvalidTokenException: Update token isn't valid.
                    "instanceId": "",

Workaround: The only known solution is to remove the resource entirely and provision it again

Output of pulumi about

CLI          
Version      3.124.0
Go Version   go1.22.5
Go Compiler  gc

Plugins
KIND      NAME    VERSION
language  nodejs  unknown

Host     
OS       darwin
Version  14.5
Arch     arm64```

### Additional context

_No response_

### Contributing

Vote on this issue by adding a 👍 reaction. 
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already). 

[justinvp]

This looks like it may be specific to AWS. Moving to the aws repo for further triage.

[corymhall]

@aaronnguyenwh thanks for reporting this! When you run pulumi up to update the configuration are you running it with --refresh, or did you first run pulumi refresh? My hunch is that you first need to sync the cloud state through a refresh to get a valid update token.

[aaronnguyenwh]

@corymhall Yes, I did. But pulumi refresh output for certificateAuthorityArn doesn't match the actual one on TLS inspection configuration.

[corymhall]

@aaronnguyenwh do you have an example of creating the certificate? I'm trying, but can't figure out how to create one that networkfirewall accepts.

[aaronnguyenwh]

@corymhall our usage is very similar to this snippet:

import * as aws from "@pulumi/aws";
import * as tls from "@pulumi/tls";

const rootCA = new aws.acmpca.CertificateAuthority("root-ca", {
    type: "ROOT",
    certificateAuthorityConfiguration: {
        keyAlgorithm: "RSA_4096",
        signingAlgorithm: "SHA512WITHRSA",
        subject: {
            commonName: "Root CA",
        },
    },
    permanentDeletionTimeInDays: 7,
});

const key = new tls.PrivateKey("key", {algorithm: "RSA"});
const csr = new tls.CertRequest('csr', {
    privateKeyPem: key.privateKeyPem,
    subject: {
        commonName: 'Network Firewall',
    },
});

const tlsCert = new aws.acm.Certificate("tls-cert", {
    certificateAuthorityArn: rootCA.arn,
    privateKey: key.privateKeyPem,
    certificateSigningRequest: csr.certRequestPem,
});

// then you can use `tlsCert` for TLS inspection configuration

More insights on some of the restrictions TLS inspection has CA certificate - Outbound SSL/TLS inspection

To configure outbound TLS inspection, you must first import a certificate authority (CA) certificate into AWS Certificate Manager (ACM). After you import the CA certificate in ACM, you can associate the CA certificate with your TLS inspection configuration. Network Firewall uses the CA certificate to generate a server certificate, which the service uses to establish trust between the client and the server.

[corymhall]

Unfortunately I'm unable to get anything working. I get an error from networkfirewall. It would help to have a self contained example that we can deploy to reproduce, otherwise I can try to get something working as time permits.

CertificateAuthorityArn is invalid because it references a certificate authority that doesn't comply with RFC 5280 basic constraints.

[aaronnguyenwh]

@corymhall can you try doing this manually with openssl?

1. Create a file named subordinateCA.cnf with the following contents:

[ v3_ca ]
basicConstraints = critical,CA:TRUE,pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer

3. Run the following commands to create the certificate authorities using OpenSSL:

 openssl genpkey -algorithm RSA -out rootCA.key -pass pass:password -pkeyopt rsa_keygen_bits:4096
 openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 -out rootCA.crt -passin pass:password -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=RootCA"

 openssl genpkey -algorithm RSA -out subordinateCA-1.key -pkeyopt rsa_keygen_bits:4096
 openssl req -new -key subordinateCA-1.key -out subordinateCA-1.csr -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=SubordinateCA"
 openssl x509 -req -in subordinateCA-1.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out subordinateCA-1.crt -days 3650 -sha256 -extfile subordinateCA.cnf -extensions v3_ca -passin pass:password
 cat rootCA.crt subordinateCA-1.crt > subordinateCAChain-1.pem

 openssl genpkey -algorithm RSA -out subordinateCA-2.key -pkeyopt rsa_keygen_bits:4096
 openssl req -new -key subordinateCA-2.key -out subordinateCA-2.csr -subj "/C=US/ST=State/L=City/O=Organization/OU=OrgUnit/CN=SubordinateCA"
 openssl x509 -req -in subordinateCA-2.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out subordinateCA-2.crt -days 3650 -sha256 -extfile subordinateCA.cnf -extensions v3_ca -passin pass:password
 cat rootCA.crt subordinateCA-2.crt > subordinateCAChain-2.pem

5. Upload the subordinateCA-1.crt and subordinateCA-2.crt certificate to ACM along with their respective chain and key files.
6. Create a tls inspection configuration through Pulumi that uses the subordinateCA-1 ARN as the CA certificate.
7. In the AWS console, go to manually change the CA certificate and notice that the dropdown is not set when you go to change it, similar to screenshot below
   
8. Manually set the certificate to the subordinateCA-1 certificate in the dropdown
9. Run a pulumi refresh and then change the certificateAuthorityArn to subordinateCA-2 ARN in the Pulumi code
10. Run pulumi up and see the error

[mhamill2]

It seems like the update token isn't being updated. When you do a refresh after manually changing the tls certificate in the console, the update token does not change when looking at the stack export for that tls inspection resource.