Add Custom CA Bundle to terraform cli

[lmmart]

Hi there,

Current Terraform Version

Terraform v0.15.1

Use-cases

I am using a SSL Intercepting Proxy.

I have an issue only with terraform init command :

 
 Could not retrieve the list of available versions for provider hashicorp/azurerm: could not connect to registry.terraform.io: Failed
 to request discovery document: Get "https://registry.terraform.io/.well-known/terraform.json": x509: certificate signed by unknown
 authority

Others commands like terraform apply, terraform destroy which are using aws sdk/cli works fine with the use of env var AWS_CA_BUNDLE

Attempted Solutions

• Install custom CA Root in system bundle. But that's not what i want.
• Use NO_PROXY=registry.terraform.io

Proposal

Use an env var to add custom CA Bundle to terraform cli, like AWS_CA_BUNDLE for aws cli.

References

#21632

EDIT : Using env var SSL_CERT_FILE works

[apparentlymart]

Hi @lmmart! Thanks for sharing this use-case.

As I think you already know (since you mentioned it as one of your attempted solutions) our current intended solution here is to add the certificate to your system-wide trusted certificate store, because we typically don't try to replicate functionality already provided at the system level, especially in cases where various different parts of Terraform all interact with those functions independently and so it would likely end up requiring lots of separate mechanisms to replicate the system-wide feature.

(We can already see an example of that in AWS_CA_BUNDLE which you mentioned; I assume that's coming as part of the AWS SDK, and so is functionality provided outside rather than within Terraform, although I didn't confirm that yet.)

We'll certainly consider this new use-case, but I just wanted to note the solution that's available right now in case anyone else finds this issue and considers that a suitable solution.

In order to better evaluate this request, it would help if you could say something about why adding this certificate to your system-wide certificate store is not an appropriate solution for you. Thanks!

[lmmart]

Because i'm not root on the devices using the SSL Intercepting proxy, so i can't add ca root to system-wide certificate store.

[geoff-reason]

I also ran into this, using CharlesProxy to debug a very difficult terraform https call which wasn't responding correctly, but maximum TF_LOG setting revealed nothing of interest.

I agree that an environment variable to override the system-wide certificate store has large precedent in other tools and libraries.

Examples:

• AWS (AWS_CA_BUNDLE) as you mentioned already @apparentlymart
• GCP (CLOUDSDK_AUTH_CORE_CUSTOM_CA_CERTS_FILE)
• requests (REQUESTS_CA_BUNDLE)
• curl (CURL_CA_BUNDLE)

I think its exactly the usecase @lmmart describes above where you dont have access to modify system-wide cert store but as an unprivileged user running a tool, you want to be able to control things.

In the end I was able to achieve it through using terraform within docker and having "system" level privs in the docker container - incidentally I had to set

• http_proxy=x.x.x.x:3128
• https_proxy=x.x.x.x:3128
• noproxy=localhost
• AWS_CA_BUNDLE=/path/to/certs.pem
• CLOUDSDK_PROXY_ADDRESS="$( echo "${https_proxy}" | cut -d: -f1)"
• CLOUDSDK_PROXY_PORT="$( echo "${https_proxy}" | cut -d: -f2)"
• CLOUDSDK_PROXY_TYPE=http
• CLOUDSDK_AUTH_DISABLE_SSL_VALIDATION=true # :-( couldnt get it working w/out it
• CLOUDSDK_AUTH_CORE_CUSTOM_CA_CERTS_FILE=/path/to/certs.pem

and

• cp "${HOME}/.tools/ssl-proxying-certificate.pem" /usr/local/share/ca-certificates/
• update-ca-certificates

And then I could see the full content of requests in my host CharlesProxy, from a containerised terraform using both AWS and GCP providers.

but if that isn't an option, an environment variable seems sensible (but the underlying go request library would have to support it I'm guessing).

[apparentlymart]

After spending some time studying Go's crypto/x509 package, I've found a few things that seem relevant to (and might potentially be a pre-existing solution to) this issue:

• On macOS, the Go standard library uses a system API function SecTrustSettingsCopyCertificates to look up certificates in three stores: user-specific, admin-defined, and system-supplied. I'm not a macOS user so I cannot be sure, but from what I was able to determine in some light research it seems that the user-specific store applies to the user of the current session and so can be edited without system-level administrative access, both to add new certificates and to mark certificates from the other stores as not trusted.
• I wasn't able to quickly understand the behavior on Windows. It uses some Windows API functions I'm not familiar with which seem to totally encapsulate where the certificates are stored, and so I can't tell if there's a way for an unprivileged user to influence what certificates are allowed.
• On other Unix systems, the Go standard library consults a set of directories which varies depending on which specific Unix flavor its targeting (Linux vs. BSD vs. Solaris, etc). However, on these platforms it seems like the Go library also attempts to honor the OpenSSL-originated environment variable SSL_CERT_DIR, which is a PATH-like colon-separated list of directories to search. If set to a non-empty value then it seems that this environment variable overrides the default search paths.

I've not yet verified whether any of the above are technically correct; this was just the result of spending a little while reviewing the code.

However, this SSL_CERT_DIR environment variable on (non-Darwin) Unix systems does look promising as a way to override the certificate directory on a per-process basis in a way that is something resembling a platform convention rather than a Terraform-specific exception, and thus it should hopefully therefore apply to all of the different subsystems that Terraform is made from, including any plugins which are using the Go standard library in the same way that Terraform CLI does. Hopefully that's useful to those who have this requirement on a system where it's supported. We'll investigate this in some more detail later, though if you see this and try it in the meantime I'd much appreciate if you can share what you tried and what the results were. Thanks!

[apparentlymart]

After my above comment I moved on to some other work that happened to involve referring to older Terraform release changelogs, and so by coincidence I happened to see the note that support for SSL_CERT_DIR arrived in Terraform v0.14.0 as a side-effect of our upgrade to Go 1.15. Therefore I assume that SSL_CERT_DIR won't work on earlier Terraform versions, but I'd still be interested to hear about experiences using it on Terraform v0.14 and later.

[lmmart]

Using env var SSL_CERT_FILE works

[ifelsefi]

Neither SSL_CERT_FILE or SSL_CERT_DIR work for me on OSX 11.2.3 when running Terraform 1.0.5.

I added the certificate in question to the trusted system keystore as well as well to my login items.

Does anyone have experience with this problem using the aws backend? I get his problem when trying to init using s3 and AWS_CA_BUNDLE does not work either.

[ifelsefi]

Hi

What library does Terraform use on OSX?

Can Terraform show more verbose logging?

Both curl and openssl do not show chain issues with Amazon as I already trust DigiCert CA.

[novettaberin]

I know I'm late to the party, but I'm on Windows, and I have added the custom CA Cert to my system's trusted certificate authorities. When hitting the TFE url with my browser I don't have any certificate violations as expected. However, running Terraform 1.3.6 does not recognize the certificate the same way my browser does. I consistently get the dreaded:

x509: certificate signed by unknown authority

[jsosic]

Neither SSL_CERT_FILE nor SSL_CERT_DIR work in alpine docker image. Tested with 1.4.5 and 1.3.9