Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v3.0.1: azurerm_key_vault: Cannot read the ressource with KeyVault: keyvault.BaseClient#GetCertificateContacts #16052

Closed
uncycler opened this issue Mar 25, 2022 · 32 comments

Comments

@uncycler
Copy link

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v1.1.7
azurerm v3.0.1

Affected Resource(s)

  • azurerm_key_vault

Expected Behaviour

No error

Actual Behaviour

Cannot read the ressource with `KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

Steps to Reproduce

  1. Create a keyvault with v2:
resource "azurerm_key_vault" "mykeyvault" {
  name                = "mykeyvault"
  sku_name            = "standard"
  location            = "myloc"
  resource_group_name = "myrg"
  tenant_id           = "mytenant"
  purge_protection_enabled = true
}
  1. with v3, do a terraform plan

Important Factoids

Since v3, azurerm is trying to fetch "contact" from KeyVault and for some reason, the access is timing out. I've tried adding ManagedContacts permission but no luck.

Using same credentials, I can access the API without isuse.

az rest --method GET --uri "https://mykeyvault.vault.azure.net/certificates/contacts?api-version=7.1" --resource "https://vault.azure.net"
Not Found({"error":{"code":"ContactsNotFound","message":"Contacts not found"}})
  • #0000
@noiano
Copy link

noiano commented Mar 25, 2022

I have a similar issue. Simply referencing a certificate in a key vault causes the plan to fail

`data "azurerm_key_vault" "keyVault" {
  name                = "XXX"
  resource_group_name = "YYY"
}

# Get ID for latest certificate version in KV
data "azurerm_key_vault_secret" "certificate" {
  name         = "ZZZ"
  key_vault_id = data.azurerm_key_vault.keyVault.id
}

The plan command fails (after a long while) with the following error:

╷
│ Error: making Read request on Azure KeyVault Secret <name redacted>: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: authorizer was not an auth.CachedAuthorizer for https://vault.azure.net
│ 
│   with data.azurerm_key_vault_secret.certificate,
│   on subscription.tf line 13, in data "azurerm_key_vault_secret" "certificate":
│   13: data "azurerm_key_vault_secret" "certificate" {
│ 
╵
Releasing state lock. This may take a few moments...

@kovaliovsg
Copy link

I have these both issues as well. Had to roll back to provider 2.99.0

@cmendible
Copy link
Contributor

Just hit this one...

Terraform 1.1.7
azurerm 3.0.1

@nbjohnson
Copy link

nbjohnson commented Mar 25, 2022

I just ran into a similar issue with azurerm_key_vault_secret data source

Error: making Read request on Azure KeyVault Secret test-secret: autorest/Client#Do: 
Preparing request failed: StatusCode=0 -- Original Error: failed to acquire MSAL token

Error: keyvault.BaseClient#GetKey: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

I see this for both v3.0.0 and v3.0.1

@manicminer manicminer self-assigned this Mar 25, 2022
@manicminer
Copy link
Member

Hi, thanks for reporting this. I'm looking at each report in this issue in turn and trying to reproduce across the authentication methods we support, however it would help greatly if anyone getting a timeout or "context deadline exceeded" error, were able to post a debug log (via gist) whilst this is happening.

You can do this by setting the TF_LOG environment variable to debug, for example with bash-compatibles:

TF_LOG=debug terraform apply

@SimonGasparKentico
Copy link

SimonGasparKentico commented Mar 25, 2022

I got another error
Error: retrieving "contact" for KeyVault: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: authorizer was not an auth.CachedAuthorizer for https://vault.azure.net
I'm not sure if this is related to this problem or if the problem is on my side :)

We are using:
Az CLI 2.33.0
Terraform v1.1.7
Azurerm v3.0.1

@djryanj
Copy link
Contributor

djryanj commented Mar 25, 2022

@manicminer I'm preparing a gist but does the azurerm provider team have a different public key than the terraform one? I'm going to redact the file but would still prefer to encrypt it.

@manicminer
Copy link
Member

@SimonGasparKentico, @noiano I believe I may have found an issue with Azure CLI authentication and Key Vault.

@manicminer
Copy link
Member

@djryanj If you wish to encrypt your log, please use the public key for the "terraform" Keybase user. Thanks!

@djryanj
Copy link
Contributor

djryanj commented Mar 25, 2022

@manicminer
Copy link
Member

manicminer commented Mar 25, 2022

@djryanj Many thanks, I'm having some trouble accessing that privkey at the moment - in the interest of expedience would you be able to re-encrypt with my personal key?

Nevermind, I was able to decrypt the log! It looks like you're getting the authorizer was not an auth.CachedAuthorizer for https://vault.azure.net error, can you confirm if you're using Azure CLI or MSI authentication? We should have a fix for this shortly.

@manicminer
Copy link
Member

manicminer commented Mar 25, 2022

@uncycler, @kovaliovsg, @cmendible, @nbjohnson Would you be able to share a debug log for when you're seeing the "context deadline exceeded" error for a Key Vault related resource? Thanks!

@djryanj
Copy link
Contributor

djryanj commented Mar 25, 2022

@manicminer glad you found the key, I promptly broke my terraform config after hitting this bug (developing something else, this was non-blocking for the moment) and also (like an idiot) deleted the debug log so couldn't reproduce quickly enough.

I am using Azure CLI authentication.

@cmendible
Copy link
Contributor

Sorry no logs since I reverted back.

I'm also using Azure CLI authentication

@eoly
Copy link

eoly commented Mar 25, 2022

@manicminer

Here are my TF_DEBUG logs. I have the context deadline exceeded issue too.

Using Azure CLI auth.

https://gist.github.com/eoly/702d9e6da370cd946372bce3677db830

@derekwilliamsliquidx
Copy link

I am hitting this error as well

Terraform version 1.1.6
Azure CLI Version 2.26.1
AzureRM Provider Version = 3.0.1

Reverting back to 2.99 corrected the issue.

@manicminer
Copy link
Member

manicminer commented Mar 25, 2022

@eoly Thanks for the log, that's super helpful 👍

Anyone else, please kindly refrain from posting +1 comments and instead upvote the issue. Debug logs are always happily received!

katbyte pushed a commit that referenced this issue Mar 25, 2022
Fixes the authorizer was not an auth.CachedAuthorizer for https://vault.azure.net error reported in #16052
@noiano
Copy link

noiano commented Mar 27, 2022

I'm actually using a service principal to authenticate all actions against my subscription ...

I'd also like to add that, using azurerm 3.0.2, the terraform plan command which was previously stuck and ended up in error now goes through.

It's only much slower.

@nbjohnson
Copy link

Even after upgrading to v3.0.2, I am still getting a similar, but not exactly the same error. I am attempting to do a data source lookup for a key vault secret. This works on versions < 3.0.0, but is now just broken. If there is a breaking change that I am missing that could be affecting this let me know, or is this a bug related to the other errors users are reporting?

:51:05.293Z [ERROR] vertex "module.storage_account[\"<storage account name>\"].azurerm_key_vault_key.this" error: keyvault.BaseClient#GetKey: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
:51:05.294Z [ERROR] vertex "module.storage_account[\"<storage account name>\"].azurerm_key_vault_key.this" error: keyvault.BaseClient#GetKey: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
:51:05.294Z [ERROR] vertex "module.storage_account.azurerm_key_vault_key.this (expand)" error: keyvault.BaseClient#GetKey: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
:53:35.590Z [ERROR] vertex "data.azurerm_key_vault_secret.lookups[\"<secret local var name>\"]" error: making Read request on Azure KeyVault Secret <secret name>: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: failed to acquire MSAL token for <vault url>
:53:35.590Z [ERROR] vertex "data.azurerm_key_vault_secret.lookups" error: making Read request on Azure KeyVault Secret <secret name>: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: failed to acquire MSAL token for <vault url>
:53:35.591Z [ERROR] vertex "data.azurerm_key_vault_secret.lookups (expand)" error: making Read request on Azure KeyVault Secret <secret name>: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: failed to acquire MSAL token for <vault url>

For initial Azure auth I am using CLI

@djryanj
Copy link
Contributor

djryanj commented Mar 28, 2022

@nbjohnson - try doing an az logout and az login. I didn't see a context deadline exceeded like in your error but did get something related to failing to get an MSAL token this morning, and this fixed it. Note my error may not have been the same as yours, but it's worth a try.

@nbjohnson
Copy link

@djryanj Thanks for the suggestion, but unfortunately that didn't fix my issue. Still keep getting failed to acquire MSAL token for at least key_vault_key creation and key_vault_secret data resource

@manicminer
Copy link
Member

@nbjohnson Thanks for the feedback. I've been unable to reproduce that error, which is related to a failure invoking az-cli to acquire an access token. I would suggest upgrading to the latest version of Azure CLI, and deleting your ~/.azure directory before signing in again. I will look at ways we can improve the error reporting here.

@mark-strasser-nnl
Copy link

With USGovernment environment, the example key vault fails to get created or read from state, even with the latest versions of Terraform or azurerm.

│ Error: retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
│
│   with azurerm_key_vault.example,
│   on main.tf line 15, in resource "azurerm_key_vault" "example":
│   15: resource "azurerm_key_vault" "example" {

See relevant gist.

Terraform versions:

$ terraform version
Terraform v1.1.8
on windows_amd64
+ provider registry.terraform.io/hashicorp/azurerm v3.1.0

@bgrainger
Copy link

I believe this issue may be related to use_msal = true (which is now on by default in azurerm >= 3.0).

Works: v2.99 with use_msal= false

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "2.99"
    }
  }
}

provider "azurerm" {
  features {}

  use_msal = false
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "this" {
  name     = "V299ADAL"
  location = "westus2"
}

resource "azurerm_key_vault" "this" {
  name                = "v299adal-kv"
  location            = resource.azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  tenant_id           = data.azurerm_client_config.current.tenant_id
  sku_name            = "standard"
}

Output:

azurerm_key_vault.this: Still creating... [2m30s elapsed]
azurerm_key_vault.this: Creation complete after 2m34s [id=/subscriptions/***/resourceGroups/V299ADAL/providers/Microsoft.KeyVault/vaults/v299adal-kv]

Fails: v2.99 with use_msal = true

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "2.99"
    }
  }
}

provider "azurerm" {
  features {}

  use_msal = true
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "this" {
  name     = "V299MSAL"
  location = "westus2"
}

resource "azurerm_key_vault" "this" {
  name                = "v299msal-kv"
  location            = resource.azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  tenant_id           = data.azurerm_client_config.current.tenant_id
  sku_name            = "standard"
}

Output:

azurerm_key_vault.this: Still creating... [7m30s elapsed]
╷
│ Error: retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
│
│   with azurerm_key_vault.this,
│   on main.tf line 23, in resource "azurerm_key_vault" "this":
│   23: resource "azurerm_key_vault" "this" {
│
╵

FWIW, I'm not getting a failure to create this key_vault with azurerm 3.1.0. However, I wanted to paste my findings above since it seemed to be the exact same error message from the OP, and in case it provides a clue to where the problem may be.

@manicminer
Copy link
Member

There is a related issue #16291 which affects USGovernment users when managing Key Vaults (or any resource that uses the Key Vault API). This will be fixed in next week's release.

Is anyone here not using the USGovernment cloud and experiencing a context deadline exceeded error when trying to manage or reference a Key Vault?

@xinity
Copy link

xinity commented Apr 18, 2022

@manicminer i do and i'm pretty sure i'm not the only one.
I'm using "plain" Azure Subscription not a USGovernment

@mkemmerz
Copy link

Any update on this? Same error with AzureRm 3.1 on our side.

@tombuildsstuff
Copy link
Contributor

Per @manicminer's comment above this has been fixed in more recent versions of the AzureRM Provider, we're currently on version v3.5.0 - if you're still encountering this issue when using v3.5.0 please let us know, but I'm going to close this out as fixed at the moment since we believe this has since been fixed.

Thanks!

@mkemmerz
Copy link

mkemmerz commented May 12, 2022

@tombuildsstuff thanks for the reply. We are having this issue on the global Azure cloud so maybe this issue doesn't really fit for me because it is only for the USGov Cloud. Using AzureRM 3.5. doesn't fix my issue. As we are using a Private Endpoint for the connection it seems more related to #9738

@kevinwedwards
Copy link

kevinwedwards commented Jun 8, 2022

We too are having this issue on AzureRM 3.7. global azure cloud (non USGov).

@elongstreet88
Copy link

elongstreet88 commented Jun 27, 2022

Same, seems like our general performance of terraform/azurerm provider have drastically reduced lately and we are seeing the context deadline issues as well (none gov as well).

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests