Error: Private key not recognized, key is not in PEM format

[ToxicCypher]

Terraform CLI and Provider Versions

terraform {
  required_version = ">= 1.3.0"
  required_providers {
    snowflake = {
      source  = "Snowflake-Labs/snowflake"
      version = "0.84.1"
    }
  }
  backend "s3" {}
}

Terraform Configuration

provider "snowflake" {
  account = jsondecode(module.aws-secrets-manager-get-secret.snowflake-credentials-secret-string)["SnowflakeAccount"]
  user    = jsondecode(module.aws-secrets-manager-get-secret.snowflake-credentials-secret-string)["SnowflakeUser"]
  authenticator = "JWT"
  private_key   = local.my-rsa-key # Stored locally for testing. The key itself is represented in the Expected Behavior section.
  role          = var.default-snowflake-role
  warehouse     = var.default-snowflake-warehouse
  params = {
    query_tag = "Snowflake CI/CD"
  }
}

Expected Behavior

The behavior I am expecting is that the private key is accepted by the Snowflake provider. The structure of my private key is as follows (This is a dummy key, but follows the exact same structure as the actual private key that I am providing to the provider) :

The key is of length: 4096

-----BEGIN PRIVATE KEY-----
MIIJJwIBAAKCAgBbKusUIihl3myn27+OKsV1xpuRv9ik9wTNhbJOPl8+I5b1zjsQ
F4CZ3kdJCyi/gHRPGEVn7NV5d066v8dMvkXzy1mRIH9Lysy2RVjEBjO0J+Gk9kDL
DciTdkkYMdI+9p016vE7UdxbyIJRT5uJ9KsdZMkkt5CS9sUiDophDw4b7WT7ijOR
...
yfHY0fgAO9+6d5l5d5u4iNowbGGPEOkqSeM9qVj4VamIpauHVhO0R8v4sySaukF4
KWKKojFMSNUCYGk8f0v6+RYeUAIteNCJq5xgmEqZ+sYyyLCit38dSve9TZ8erBGg
GwZ92w+osBZvR0qia2xYuJUpoS8NaJSlS2GCYigNKoldaN3QDEfHvm7Kxw==
-----END PRIVATE KEY-----

Actual Behavior

Below is the output I receive when performing terraform apply.

│ Error: could not retrieve private key: could not parse private key, key is not in PEM format
│ 
│   with provider["[registry.terraform.io/snowflake-labs/snowflake](http://registry.terraform.io/snowflake-labs/snowflake)"],
│   on [main.tf](http://main.tf/) line 34, in provider "snowflake":
│   34: provider "snowflake" {

I have read through some open issues regarding private keys with the Snowflake provider and have adhered to the suggestions found, such as providing the Snowflake provider with the authenticator = "JWT" parameter.

Steps to Reproduce

1. Create an RSA key pair in Snowflake.
2. Store the private key in an AWS Secrets Manager secret.
3. Run terraform plan.

How much impact is this issue causing?

High

Logs

No response

Additional Information

Using a password works just fine, but is no viable for our production environment.

[ToxicCypher]

I am wondering if there is some form of pre-processing that must be done to the private key before passing it to the Snowflake provider configuration. We typically perform some processing when using the python connector with a private key but to my knowledge this is using the go gosnowflake library.

[ToxicCypher]

Removing new lines from the private key body stops complaining about the RSA key not being in a PEM format, but then compains about the JWT being invalid.

Sample Key:

-----BEGIN PRIVATE KEY-----
MIIJJwIBAAKCAgBbKusUIihl3myn27+OKsV1xpuRv9ik9wTNhbJOPl8+I5b1zjsQ...GwZ92w+osBZvR0qia2xYuJUpoS8NaJSlS2GCYigNKoldaN3QDEfHvm7Kxw==
-----END PRIVATE KEY-----

│ Error: open snowflake connection: 390144 (08004): JWT token is invalid. [ff4e4855-9e30-4213-b0b6-edcc5d94f200]
│ 
│   with provider["[registry.terraform.io/snowflake-labs/snowflake](http://registry.terraform.io/snowflake-labs/snowflake)"],
│   on [main.tf](http://main.tf/) line 36, in provider "snowflake":
│   36: provider "snowflake" {
│ 
╵

[sfc-gh-asawicki]

Hey @ToxicCypher. Thanks for reporting the issue.

At first glance, it does not look like a problem with the provider. The error code 08004 indicates that the connection was rejected. The error 390144 is mentioned here: https://docs.snowflake.com/en/user-guide/key-pair-auth-troubleshooting#list-of-errors with possible troubleshooting:

390144	JWT_TOKEN_INVALID	There is a general issue with the JWT token. For possible solutions, see Common Errors and Solutions.

Please make sure that your key is generated correctly according to docs.
You should copy the exact content from the generated key file.

Also, you can try using private_key with terraform file function like:

private_key = file("<filepath>")

where <filepath> points to your private key file.

[ToxicCypher]

Thank you for the response @sfc-gh-asawicki, I literally just figured out what was wrong and it had everything to do with how the private key was being stored in AWS Secrets Manager and all is working as expected now!

Moving this issue to closed.

[NaveenAutomate]

HI @ToxicCypher , I am having same issue, when I give private key directly into terraform provider it works but when I retrieve from aws secrets manager it doesnt work, looks like its not copying the newlines to secrets maanger.

How did you manage to resolve this ?

[ToxicCypher]

Hey, I will get back to you in the morning, it has to do with the way the private key is formatted so I had to use a couple of regex functions to deconstruct the private key after I pulled it from AWS and then put it back together in the correct format. I'll send you the actual terraform when I get to work. Also, if you don't want to wait, the way I figured this out was I looked at the go backend for the go-snowflake package which the Snowflake terraform provider uses as its back end. There is some code in there (don't remember where) that shows an exact regular expression that the private key is retrieved with which will tell you exactly how the key needs to be formatted.

…

On Sun, Jun 23, 2024, 7:00 PM Naveen kumar ***@***.***> wrote: HI @ToxicCypher <https://github.com/ToxicCypher> , I am having same issue, when I give private key directly into terraform provider it works but when I retrieve from aws secrets manager it doesnt work, looks like its not copying the newlines to secrets maanger. How did you manage to resolve this ? — Reply to this email directly, view it on GitHub <#2432 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A64IBEYNNLAA7LU4MWIP3TDZI5HPZAVCNFSM6AAAAABCMQEOQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVGM2DSOBRGE> . You are receiving this because you were mentioned.Message ID: ***@***.*** .com>

[ToxicCypher]

This shows how the Snowflake terraform provider is expecting the Private Key: https://github.com/snowflakedb/gosnowflake/blob/master/secret_detector.go privateKeyPattern = `(?im)-----BEGIN PRIVATE KEY-----\\n([a-z0-9/+=\\n]{32,})\\n-----END PRIVATE KEY-----` If I recall correctly, the AWS Terraform provider returns the private key/rsa key with a bunch of new lines and return carriages. On Sun, Jun 23, 2024 at 11:16 PM Cameron Gibson ***@***.***> wrote:

…

Hey, I will get back to you in the morning, it has to do with the way the private key is formatted so I had to use a couple of regex functions to deconstruct the private key after I pulled it from AWS and then put it back together in the correct format. I'll send you the actual terraform when I get to work. Also, if you don't want to wait, the way I figured this out was I looked at the go backend for the go-snowflake package which the Snowflake terraform provider uses as its back end. There is some code in there (don't remember where) that shows an exact regular expression that the private key is retrieved with which will tell you exactly how the key needs to be formatted. On Sun, Jun 23, 2024, 7:00 PM Naveen kumar ***@***.***> wrote: > HI @ToxicCypher <https://github.com/ToxicCypher> , I am having same > issue, when I give private key directly into terraform provider it works > but when I retrieve from aws secrets manager it doesnt work, looks like its > not copying the newlines to secrets maanger. > > How did you manage to resolve this ? > > — > Reply to this email directly, view it on GitHub > <#2432 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/A64IBEYNNLAA7LU4MWIP3TDZI5HPZAVCNFSM6AAAAABCMQEOQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVGM2DSOBRGE> > . > You are receiving this because you were mentioned.Message ID: > <Snowflake-Labs/terraform-provider-snowflake/issues/2432/2185349811@ > github.com> >

[NaveenAutomate]

@ToxicCypher It would be great if you can share the bits of code.
I tried different with different newlines but nothing worked! :(
TIA

[ToxicCypher]

This is how I reconstruct the private key locals { processed-secret-string = jsondecode(data.aws_secretsmanager_secret_version.snowflake-secret-version.secret_string) pre-processed-private-key-body = regex("-----BEGIN PRIVATE KEY----- (.*) -----END PRIVATE KEY-----", local.processed-secret-string["PrivateKey"])[0] processed-private-key-body = replace(local.pre-processed-private-key-body, " ", "\n") processed-private-key = join("\n", ["-----BEGIN PRIVATE KEY-----", local.processed-private-key-body, "-----END PRIVATE KEY-----"]) } Sorry for the bad formatting, hope that helps!

…

On Mon, Jun 24, 2024, 6:34 AM Naveen kumar ***@***.***> wrote: @ToxicCypher <https://github.com/ToxicCypher> It would be great if you can share the bits of code. I tried different with different newlines but nothing worked! :( TIA — Reply to this email directly, view it on GitHub <#2432 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A64IBE35J2YGCXFCICCPYKTZI7Y3TAVCNFSM6AAAAABCMQEOQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBWGIZDSOJRHE> . You are receiving this because you were mentioned.Message ID: ***@***.*** .com>

[NaveenAutomate]

Thanks @ToxicCypher it worked!

[ToxicCypher]

Glad to hear it!

…

On Mon, Jun 24, 2024 at 10:07 AM Naveen kumar ***@***.***> wrote: Thanks @ToxicCypher <https://github.com/ToxicCypher> it worked! — Reply to this email directly, view it on GitHub <#2432 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A64IBE5FHJM36MX3UQ34TFLZJARYNAVCNFSM6AAAAABCMQEOQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBWGY3DQOJTGU> . You are receiving this because you were mentioned.Message ID: ***@***.*** .com>