improve error when missing `required_providers` causes lookup in `hashicorp` namespace #31645

roy-work · 2022-08-15T16:49:01Z

Terraform Version

Terraform v1.0.9

Terraform Configuration Files

Add a new module using Cloudflare records, but "forget" (i.e., not even realize you need) to add:

terraform {
  required_providers {
    cloudflare = {
      source = "cloudflare/cloudflare"
      version = ">= 3.1.0"
    }
  }
}

Non-Debug Output

Initializing provider plugins...
- Reusing previous version of cloudflare/cloudflare from the dependency lock file
- Finding latest version of hashicorp/cloudflare...
- Installing cloudflare/cloudflare v3.1.0...
- Installed cloudflare/cloudflare v3.1.0 (signed by a HashiCorp partner, key ID DE413CEC881C3283)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider
│ hashicorp/cloudflare: provider registry registry.terraform.io does not have
│ a provider named registry.terraform.io/hashicorp/cloudflare
│ 
│ Did you intend to use cloudflare/cloudflare? If so, you must specify that
│ source address in each module which requires that provider. To see which
│ modules are currently depending on hashicorp/cloudflare, run the following
│ command:
│     terraform providers
╵

Debug output

The debug output, which isn't much more informative, either.

Initializing provider plugins...
- Reusing previous version of cloudflare/cloudflare from the dependency lock file
2022-08-15T16:44:30.866Z [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2022-08-15T16:44:30.897Z [DEBUG] Using modified User-Agent: Terraform/1.0.9 TFE/v202205-1
2022-08-15T16:44:30.897Z [DEBUG] GET https://registry.terraform.io/v1/providers/cloudflare/cloudflare/versions
- Finding latest version of hashicorp/cloudflare...
2022-08-15T16:44:30.912Z [DEBUG] Using modified User-Agent: Terraform/1.0.9 TFE/v202205-1
2022-08-15T16:44:30.912Z [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/cloudflare/versions
2022-08-15T16:44:30.929Z [DEBUG] Using modified User-Agent: Terraform/1.0.9 TFE/v202205-1
2022-08-15T16:44:30.929Z [DEBUG] GET https://registry.terraform.io/v1/providers/cloudflare/cloudflare/3.1.0/download/linux/amd64
2022-08-15T16:44:30.945Z [DEBUG] GET https://github.com/cloudflare/terraform-provider-cloudflare/releases/download/v3.1.0/terraform-provider-cloudflare_3.1.0_SHA256SUMS
2022-08-15T16:44:31.077Z [DEBUG] GET https://github.com/cloudflare/terraform-provider-cloudflare/releases/download/v3.1.0/terraform-provider-cloudflare_3.1.0_SHA256SUMS.sig
- Installing cloudflare/cloudflare v3.1.0...
2022-08-15T16:44:31.164Z [DEBUG] Using modified User-Agent: Terraform/1.0.9 TFE/v202205-1
2022-08-15T16:44:31.448Z [DEBUG] Provider signed by DE413CEC881C3283 Cloudflare Terraform Provider Team <api@cloudflare.com>
- Installed cloudflare/cloudflare v3.1.0 (signed by a HashiCorp partner, key ID DE413CEC881C3283)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider
│ hashicorp/cloudflare: provider registry registry.terraform.io does not have
│ a provider named registry.terraform.io/hashicorp/cloudflare
│
│ Did you intend to use cloudflare/cloudflare? If so, you must specify that
│ source address in each module which requires that provider. To see which
│ modules are currently depending on hashicorp/cloudflare, run the following
│ command:
│     terraform providers
╵

Expected Behavior

The diagnostics emitted by a non-debug run of Terraform are informative, and useful, enough so to locate & fix the issue.

Actual Behavior

I got the above diagnostic.

The diagnostic is correct that, yes, I do intend cloudflare/cloudflare. So my question here is "what is installing hashicorp/cloudflare?" and the diagnostic just hangs me out to dry in that regard. No clues. In the change that triggered this, I'm not attempting to change what providers are installed: I'm merely adding cloudflare_record resources, so I'm doubly confused.

rg thinks I'm not using it:

» rg 'hasi.*cloud'
(last command returned 1.)
»

I check the lock file:

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/cloudflare/cloudflare" {
  version     = "3.1.0"
  constraints = ">= 3.1.0"
  hashes = [
    "h1:gkeUHSdRa8en9whBRp4MAwzBC3gVIj6rCD2aMddEmZo=",
    "zh:03b1e385aa7f5aa85cd002d688ec7ebf0a58862de5c7b5a9969e2669b1ba5071",
    "zh:043becbf9b37bf1616cd20c541b9a6a3e869a2c160c4a33b1ed0118a0847a1c5",
    "zh:0497eaff64e8598d1545e0efed5b1e8eb1a6418326d949fb60428ffd1b0bd05f",
    "zh:41f3fe35c8562ca2120d51c0a10932c1c85fae8084c447fc0fd7318674f0d36c",
    "zh:4c0794756d0bb194959f33ffb378fc795db1ccd2d4526fad0f00aae3f4cef4ec",
    "zh:5c94814960afd9f7c93f5396655255b7503bd5497c6fe6be759832b73225c7b3",
    "zh:71d6d24afb7015914e0bbfb94ef256f8bb9d60a0f9152577692fcfb8627c1ed0",
    "zh:780ec8088f3c385f3d673f28f8f0782cff2eec7a4e2cf21c676e5dcb6016b899",
    "zh:7bac9a9ec4f8db40f8d398fb6c9a9a2c2ccc83c46649d1b82133bf632d9de9ae",
    "zh:a2d7e593d6b03fda80f77ac89fe10c7403f8b52b352c43f044ca23910b43d49a",
    "zh:b6ac0c6720efa08bd4ece19096771f7676ae1194f1e7929ce33566146d3b6f6e",
    "zh:c6466acf245b6d816a1502eebf6f781ad779d13390aa8f14404c7fdb2dfe18d3",
    "zh:d6a1c7d4633975a8fbbf45775104824eeb04965eb8e7172fd0515ced6bc4e67b",
    "zh:f5d25cd665ecd24099bde35d2f5070593beae8a421cfa5044a38e1d8d5593644",
  ]
}

So now I'm really confused. Why is this provider being installed?

Steps to Reproduce

terraform plan

… with a module missing the provider section that requires a provider section.

Additional Context

We use TFE.

in the module. I have no idea why a providers section is sometimes required, and when it is, it continually bites me in the ass like this, and the diagnostics are no help in getting me to realizing that TF is biting me in the ass in this particular manner. To me, either one of the following should hold true:

We already have a provider for cloudflare_* in the main workspace, so we don't need to suddenly invent a provider name and try to download some random stuff that doesn't exist.
Modules don't inherit providers from the workspace, and so we should just error out, instead of suddenly trying to invent a provider name and then trying to download random stuff that doesn't exist.

But I have no explanations for TF's behavior. It isn't consistent, nor predictable. I've merely learned to recognize this particular pit, when I'm in it.

References

The text was updated successfully, but these errors were encountered:

jbardin · 2022-08-15T17:50:20Z

Hi @roy-work,

Sorry you're having trouble with this, does the output of the suggested command of terraform providers point to the module which is associating the cloudflare name with hashicorp/cloudflare? I don't think that a current Terraform release will give any more information in this particular case, but it may be worth comparing to see if it's helpful at all.

In general, all modules must have the correct required_providers in order to locate the necessary providers for that module. The exception is for providers in the hashicorp/ namespace, which for backwards compatibility, will have an implied source of registry.terraform.io/hashicorp/providername and no version constraint if it's not specified. This backward compatibility is what can make the situation confusing, since not specifying the provider source for a provider outside of the hashicorp namespace will cause Terraform to lookup the provider in the default location.

Since you mention that you recently added a resource to a module, that module would be the first place I check to ensure it has the cloudflare provider listed in required_providers.

While Terraform is working correctly here, perhaps there is a way we can try to correlate a possible source of the incorrect configuration for the Did you intend to use ... suggestion within the diagnostic message.

roy-work · 2022-08-17T14:54:38Z

Good grief.

So, if you take the recommended advice of running terraform providers, that will fail.

» terraform providers
╷
│ Error: Module not installed
│
│   on snip.tf line 144:
│  144: module "snip" {
│
│ This module is not yet installed. Run "terraform init" to install all modules required by this configuration.
╵

╷
│ Error: Module not installed
│
│   on snip.tf line 189:
│  189: module "snip" {
│
│ This module is not yet installed. Run "terraform init" to install all modules required by this configuration.
╵

╷
│ Error: Module not installed
│
│   on snip.tf line 197:
│  197: module "snip" {
│
│ This module is not yet installed. Run "terraform init" to install all modules required by this configuration.
╵
(last command returned 1.)

So you keep tugging that root:

» terraform init
Initializing modules...
- snip in ../modules/snip
- snip in ../modules/snip
- snip in ../modules/snip

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of cloudflare/cloudflare from the dependency lock file
- Finding latest version of hashicorp/cloudflare...
- Using previously-installed cloudflare/cloudflare v3.1.0
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider hashicorp/cloudflare: provider registry registry.terraform.io does not have a
│ provider named registry.terraform.io/hashicorp/cloudflare
│
│ Did you intend to use cloudflare/cloudflare? If so, you must specify that source address in each module which requires that provider. To see
│ which modules are currently depending on hashicorp/cloudflare, run the following command:
│     terraform providers
╵
(last command returned 1.)

Which is then becomes a loop.

Surprisingly, if you go actually, for whatever reason, decide to go around the loop again, despite those commands both failing, some progress apparently did happen during the previous failures!

Providers required by configuration:
.
├── provider[registry.terraform.io/cloudflare/cloudflare] >= 3.1.0
├── module.snip
│   └── provider[registry.terraform.io/cloudflare/cloudflare] >= 3.1.0
├── module.snip
│   └── provider[registry.terraform.io/cloudflare/cloudflare] >= 3.1.0
├── module.snip
│   └── provider[registry.terraform.io/cloudflare/cloudflare] >= 3.1.0
├── module.snip
│   └── provider[registry.terraform.io/cloudflare/cloudflare] >= 3.1.0
├── module.snip
│   └── provider[registry.terraform.io/hashicorp/cloudflare]
├── module.snip
│   └── provider[registry.terraform.io/hashicorp/cloudflare]
└── module.snip
    └── provider[registry.terraform.io/hashicorp/cloudflare]

Providers required by state:

    provider[registry.terraform.io/cloudflare/cloudflare]

… so I'll admit I missed that in the original error, but … just do the resolution for the user, and make it obvious?
And in the error, that the provider is being automatically added by TF due to backwards-compat could be noted, or the logic could be there, or something. Something to fix my broken mental model.

The exception is for providers in the hashicorp/ namespace, which for backwards compatibility

… I feel like I've struggled with this through any number of breaking changes in TF.

siavashingka · 2022-11-16T21:07:47Z

Hello, I am experiencing the same issue with splunk-terraform/signalfx provider and it is really annoying, have came up with any solution?
@jbardin is it a known bug?

siavashingka · 2022-11-17T08:36:28Z

So, we were facing this error message for the planning stage:
Error: failed to read schema for module.signalfx_alert_heartbeat_article_aggregator_service.signalfx_detector.heartbeat_check in registry.terraform.io/hashicorp/signalfx: failed to instantiate provider "registry.terraform.io/hashicorp/signalfx" to obtain schema: unavailable provider "registry.terraform.io/hashicorp/signalfx"
This is really weird in some cases for example it says it is trying to get the schema from registry.terraform.io/hashicorp/signalfx which is not correct it has to be something like registry.terraform.io/splunk-terraform/signalfx
The error says it cannot get the schema from the registry which is misleading as well. (will see why)
Then I came up with this open bug on Terraform: #31645 the case is exactly the same with another provider.
But after more investigation, I tried to skip the refresh step in terraform apply command with -refresh=false and I saw that the plan is working so I discovered that there is a problem with refreshing.
Basically, the refresh step is about comparing the state of the terraform with the actual state of the destination service (here Splunk).
Then I realized that I removed an alert from Splunk manually which is the root cause of this out-of-sync state.
I used terraform rm state command to remove the manually deleted resource from the state file in order to recreate it again and then Wallah! it worked!
I would say the terraform error is absolutely misleading.

toMyLord · 2022-12-07T08:40:24Z

Hello, I am facing the same issue when i use terraform init command. Is there any way to resolve or bypass this problem？ @jbardin

jbardin · 2022-12-07T14:18:46Z

@toMyLord, you likely have a module missing the necessary required_providers entry. Try the instructions above, and if you have questions you can ask in the community forum where there are more people ready to help.

Thanks!

toMyLord · 2022-12-08T02:35:46Z

@jbardin, thanks for your reply.
I tried to open the community forum to resolve my problem, but I failed.

Actually I had added the necessary required_providers entry before used terraform init command. Those are error messages.

Initializing the backend...

Initializing provider plugins...
- Finding xxx.com/yyy/zzz versions matching "0.0.1"...
- Finding latest version of hashicorp/zzz...
- Installing xxx.com/yyy/zzz v0.0.1...
- Installed xxx.com/yyy/zzz v0.0.1 (unauthenticated)
╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/zzz: provider registry registry.terraform.io does not have a provider named registry.terraform.io/hashicorp/zzz
│ 
│ All modules should specify their required_providers so that external consumers will get the correct providers when using a module. To see which modules are currently depending on hashicorp/zzz, run
│ the following command:
│     terraform providers

It seems like terraform try to find two providers from xxx.com/yyy and hashicorp/zzz, but I had set the required_providers like this:

terraform {
  required_providers {
    zzz = {
      version = "0.0.1"
      source = "xxx.com/yyy/zzz"
    }
  }
}

ps: I hadn't uploaded my providers(xxx.com/yyy/zzz) to any registry. instead, I had used mkdir command to build the path ~/.terraform.d/plugins/xxx.com/yyy/zzz/0.0.1/darwin_arm64 and copied terraform-provider-zzz_0.0.1.

Anyway, thank you again and I will be looking forward to you reply.

jonbaetz-qz · 2023-03-09T18:38:45Z

As I try terraform plans I get about 85% through and then

Error: failed to read schema for module.zero_trust.cloudflare_access_policy.assembly_quizlet_design_default_deny in
 registry.terraform.io/hashicorp/cloudflare: failed to instantiate provider 
"registry.terraform.io/hashicorp/cloudflare" to obtain schema: unavailable provider 
"registry.terraform.io/hashicorp/cloudflare"

The cruel and unusual part of this is I am not using or calling that provider at all.

cloudflare git:(tf) terraform providers

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/local] 2.3.0
├── provider[registry.terraform.io/cloudflare/cloudflare] 3.34.0
└── module.zero_trust
    ├── provider[registry.terraform.io/cloudflare/cloudflare] 3.34.0
    └── provider[registry.terraform.io/hashicorp/local] 2.3.0

Providers required by state:

    provider[registry.terraform.io/cloudflare/cloudflare]

    provider[registry.terraform.io/hashicorp/local]

So what is going on here? This just started happening this morning.

PedroEss · 2023-03-28T20:51:43Z

Currently I am facing identical issue but with Azure/azapi provider.

copy paste required_providers from my providers.tf file:

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.46.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "3.1.0"
    }
    azapi = {
      source  = "Azure/azapi"
      version = "=1.4.0"
    }
  }

below:

provider "azapi" {
  default_location = "westeurope"
}

And it returns me an error like below:

│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider hashicorp/azapi: provider registry registry.terraform.io does not have a provider named registry.terraform.io/hashicorp/azapi
│
│ Did you intend to use azure/azapi? If so, you must specify that source address in each module which requires that provider. To see which modules are currently depending on hashicorp/azapi, run the following command:
│     terraform providers

then when I do above error instruction I will get another error:

│ If you control this module, you can migrate to the new declaration syntax by removing all of the empty provider "azapi" blocks and then adding or updating an entry like the following to the required_providers block of
│ module.postgresql:
│     azapi = {
│       source = "hashicorp/azapi"
│     }
╵

╷
│ Error: Reserved argument name in provider block
│
│   on modules\Postrgesql\main.tf line 2, in provider "azapi":
│    2:   source = "Azure/azapi"
│
│ The provider argument name "source" is reserved for use by Terraform in a future version.

And when I change to hashicorp/azapi it is dead end.

Any recommendations to resolve this issue?

jonbaetz-qz · 2023-03-29T20:27:25Z

I actually fixed mine by running terraform plan --refresh-only -out=tfplan This gave me the clues I needed to repair my state. It was weird.

blue928 · 2023-03-29T21:39:51Z

The documentation and errors about this are very unclear. @jbardin When you referenced the documentation above about the required_providers block, are you saying that every child module also has to have an entire terraform {required_providers{<provider>=some/provider}} block?

The documentation says that providers are inherited; then says the entire terraform_block is required; then says that modules shouldn't define provider information. And, indeed, this is the case with the Build and Use Terraform modules tutorial - the aws s3 bucket module that's created has no provider reference nor terraform{} block in it. But even the error that's thrown says, you must specify that source address in each module which requires that provider

In any case, I cannot get any module to work without explicitly duplicating the terraform{} block inside each and every module I create and reference from the root module. Is this a bug or by design per the error text?

jbardin · 2023-03-30T12:38:26Z

@blue928, If a provider is not in the default hashicorp namespace, terraform needs to know exactly which provider is used within the module., so that is working as designed. To quote the linked documentation above:

Although provider configurations are shared between modules, each module must declare its own provider requirements

roy-work added bug new new issue not yet triaged labels Aug 15, 2022

jbardin added enhancement cli config and removed bug new new issue not yet triaged labels Aug 15, 2022

jbardin changed the title ~~Terrible error about non-existent provider~~ improve error when missing required_providers causes lookup in hashicorp namespace Aug 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve error when missing `required_providers` causes lookup in `hashicorp` namespace #31645

improve error when missing `required_providers` causes lookup in `hashicorp` namespace #31645

roy-work commented Aug 15, 2022 •

edited

jbardin commented Aug 15, 2022 •

edited

roy-work commented Aug 17, 2022

siavashingka commented Nov 16, 2022

siavashingka commented Nov 17, 2022

toMyLord commented Dec 7, 2022

jbardin commented Dec 7, 2022

toMyLord commented Dec 8, 2022

jonbaetz-qz commented Mar 9, 2023

PedroEss commented Mar 28, 2023

jonbaetz-qz commented Mar 29, 2023

blue928 commented Mar 29, 2023

jbardin commented Mar 30, 2023

improve error when missing required_providers causes lookup in hashicorp namespace #31645

improve error when missing required_providers causes lookup in hashicorp namespace #31645

Comments

roy-work commented Aug 15, 2022 • edited

Terraform Version

Terraform Configuration Files

Non-Debug Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Additional Context

References

jbardin commented Aug 15, 2022 • edited

roy-work commented Aug 17, 2022

siavashingka commented Nov 16, 2022

siavashingka commented Nov 17, 2022

toMyLord commented Dec 7, 2022

jbardin commented Dec 7, 2022

toMyLord commented Dec 8, 2022

jonbaetz-qz commented Mar 9, 2023

PedroEss commented Mar 28, 2023

jonbaetz-qz commented Mar 29, 2023

blue928 commented Mar 29, 2023

jbardin commented Mar 30, 2023

improve error when missing `required_providers` causes lookup in `hashicorp` namespace #31645

improve error when missing `required_providers` causes lookup in `hashicorp` namespace #31645

roy-work commented Aug 15, 2022 •

edited

jbardin commented Aug 15, 2022 •

edited