init fails for development overrides

[tombuildsstuff]

Terraform Version

$ terraform version
Terraform v0.14.4

Description

With the following ~/.terraformrc:

provider_installation {
   dev_overrides {
     "hashicorp/azurerm" = "/Users/tharvey/code/bin"
   }

   # For all other providers, install them directly from their origin provider
   # registries as normal. If you omit this, Terraform will _only_ use
   # the dev_overrides block, and so no other providers will be available.
   direct {}
 }

And a new directory containing the following main.tf:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test" {
  name = "tom-dev"
  location = "west europe"
}

Then terraform init should pick up the local override, however it instead pulls this from the registry:

$ ls -la
total 8
drwxr-xr-x  3 tharvey  staff   96 11 Jan 09:30 .
drwxr-xr-x  8 tharvey  staff  256 11 Jan 09:17 ..
-rw-r--r--  1 tharvey  staff  229 11 Jan 09:30 main.tf

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/azurerm...
- Installing hashicorp/azurerm v2.42.0...
- Installed hashicorp/azurerm v2.42.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.


Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
 - hashicorp/azurerm in /Users/tharvey/code/bin

The behavior may therefore not match any released version of the provider and
applying changes may cause the state to become incompatible with published
releases.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary

However this actually installs the latest version from the registry, currently v2.42.0:

$ tree -la
.
├── .terraform
│   └── providers
│       └── registry.terraform.io
│           └── hashicorp
│               └── azurerm
│                   └── 2.42.0
│                       └── darwin_amd64
│                           └── terraform-provider-azurerm_v2.42.0_x5
├── .terraform.lock.hcl
└── main.tf

7 directories, 3 files

$ cat .terraform.lock.hcl
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/azurerm" {
  version = "2.42.0"
  hashes = [
    "h1:oxyi21Gz35Cup/JPqPaGtxunuqMuJHHYMBzBEieMhbw=",
    "zh:5b6d6b9d53efb09d1bff83fff019107ffaf2a79a68c9f59b9541098503c9d7a8",
    "zh:87886113155f25a9875e5b0812568953bf6a523b2c604ae0860054b464d1e2fe",
    "zh:9022504ad36b9ec3132cf8a74ae150583a3798f4aea39efd323db52fe180cdbb",
    "zh:98b06bababc7dfc888e8723f563f3da8bc87ee9e0e2a6fe681a157b54e1eeb01",
    "zh:b58b6e959ce7f5d37545e9abe2375adab474845e37d6b192dd9f94ca213bc461",
    "zh:c36fd2e03bf83b6e36a59712d1e57234194455ff761ea01e62af959d08fb51b4",
    "zh:c3ac50af9682c9719f3953ade2bd545cdc02e3d23741b805dc4b891916df0d73",
    "zh:c84e0c84ae17a8d2322a6a4d3a9e48077247683591b113cf1dd14cef7e64e1d8",
    "zh:f328085cb6f6ec5b6b38476b219f9caca8c02c86b0013e5377cd100a8e9169e5",
    "zh:f33cd91ad1a173ded18fe2ccf5648d4f40bcfab893c1b5b80df7a3123f7b8297",
  ]
}

Removing the .terraform and .terraform.lock.hcl files, it's possible to run terraform plan|apply|destroy just fine, using the local/development provider however:

$ terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
 + create

Terraform will perform the following actions:

 # azurerm_resource_group.test will be created
 + resource "azurerm_resource_group" "test" {
     + id       = (known after apply)
     + location = "westeurope"
     + name     = "tom-dev"
   }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

$ terraform apply

Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
- hashicorp/azurerm in /Users/tharvey/code/bin

The behavior may therefore not match any released version of the provider and
applying changes may cause the state to become incompatible with published
releases.


An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
 + create

Terraform will perform the following actions:

 # azurerm_resource_group.test will be created
 + resource "azurerm_resource_group" "test" {
     + id       = (known after apply)
     + location = "westeurope"
     + name     = "tom-dev"
   }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
 Terraform will perform the actions described above.
 Only 'yes' will be accepted to approve.

 Enter a value: yes

azurerm_resource_group.test: Creating...
azurerm_resource_group.test: Creation complete after 2s [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

$ terraform destroy

Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
 - hashicorp/azurerm in /Users/tharvey/code/bin

The behavior may therefore not match any released version of the provider and
applying changes may cause the state to become incompatible with published
releases.


An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # azurerm_resource_group.test will be destroyed
  - resource "azurerm_resource_group" "test" {
      - id       = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev" -> null
      - location = "westeurope" -> null
      - name     = "tom-dev" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

azurerm_resource_group.test: Destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev]
azurerm_resource_group.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev, 10s elapsed]
azurerm_resource_group.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev, 20s elapsed]
azurerm_resource_group.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev, 30s elapsed]
azurerm_resource_group.test: Still destroying... [id=/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/tom-dev, 40s elapsed]
azurerm_resource_group.test: Destruction complete after 47s

Destroy complete! Resources: 1 destroyed.

---

It's worth noting that if using a provider not available in the registry, such as an alias in this case to intentionally ignore it, terraform init fails instead. For example, updating the ~/.terraformrc to:

provider_installation {
   dev_overrides {
     "tombuildsstuff/azurerm" = "/Users/tharvey/code/bin"
   }

   # For all other providers, install them directly from their origin provider
   # registries as normal. If you omit this, Terraform will _only_ use
   # the dev_overrides block, and so no other providers will be available.
   direct {}
 }

and using the following main.tf:

terraform {
  required_providers {
    azurerm = {
      source  = "tombuildsstuff/azurerm"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test" {
  name = "tom-dev"
  location = "west europe"
}

Returns a failure when using terraform init:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of tombuildsstuff/azurerm...

Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
 - tombuildsstuff/azurerm in /Users/tharvey/code/bin

The behavior may therefore not match any released version of the provider and
applying changes may cause the state to become incompatible with published
releases.


Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
tombuildsstuff/azurerm: provider registry registry.terraform.io does not have
a provider named registry.terraform.io/tombuildsstuff/azurerm

If you have just upgraded directly from Terraform v0.12 to Terraform v0.14
then please upgrade to Terraform v0.13 first and follow the upgrade guide for
that release, which might help you address this problem.

But again a terraform plan (etc) work fine:

$ terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
 + create

Terraform will perform the following actions:

 # azurerm_resource_group.test will be created
 + resource "azurerm_resource_group" "test" {
     + id       = (known after apply)
     + location = "westeurope"
     + name     = "tom-dev"
   }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

As such it appears this failure is specific to init.

References

• in-house provider not working #27358
• "missing provider" when attempting to override default #27409 - although this appears to be the inverse

[apparentlymart]

As noted in the documentation for dev_overrides, these settings override how Terraform runs providers, and not how Terraform installs providers. A development override bypasses all of Terraform's usual machinery for checking whether a suitable version is installed, whether the checksums match, etc, but as a result a development override can't participate in the installation process: it doesn't provide any of the necessary metadata to satisfy version constraints, populate the lock file, etc.

However, the fact that the dev_overrides feature completely bypasses the usual metadata checks is also what avoids this behavior being a significant problem in most cases: you can skip running terraform init at all, and just skip straight to running terraform plan or terraform apply. In that case Terraform would normally complain that the local cache of the provider doesn't exist or doesn't match the recorded metadata, but the dev_override disables all of that and just instructs Terraform to run the specified binary unconditionally, in a very similar way to how it deals with the provider acceptance test framework forcing it to use an already-running copy of the provider plugin (which also doesn't rely on or affect the behavior of terraform init, for the same reasons).

This feature was just a bonus stretch goal at the end of another project, so I think it's unlikely we'll be able to spend much more time on it in the near future, but nonetheless I'm going to relabel this as an enhancement with the intent that it represent a more complete design for development overrides, without this confusing oddity. The assumption that we'd do such a thing at some future point is what prompted the inclusion of the final paragraph of the docs:

This development overrides mechanism is intended as a pragmatic way to enable smoother provider development. The details of how it behaves, how to configure it, and how it interacts with the dependency lock file may all evolve in future Terraform releases, including possible breaking changes. We therefore recommend using development overrides only temporarily during provider development work.

[bflad]

Other provider developers including myself have run into similarly confusing situations with the development overrides and terraform init. Would you accept adding a warning to the top of the output noting that the configuration is not compatible/necessary for the command?

[bflad]

Actually it seems like the logic is already present and working for me with Terraform 0.14.4:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/random from the dependency lock file
- Installing hashicorp/random v3.0.1...
- Installed hashicorp/random v3.0.1 (signed by HashiCorp)


Warning: Provider development overrides are in effect

The following provider development overrides are set in the CLI configuration:
 - hashicorp/aws in /Users/bflad/go/bin

The behavior may therefore not match any released version of the provider and
applying changes may cause the state to become incompatible with published
releases.

Terraform has been successfully initialized!

Maybe moving that above everything, similar to #27507 can make the behavior more obvious. 👍

[jkburges]

in most cases: you can skip running terraform init at all, and just skip straight to running terraform plan or terraform apply

What if my tf code contains two providers: a local one being looked up via dev_overrides, and not published anywhere at all, along with a published/public provider (e.g. hashicorp/random)? terraform init seems necessary in this case, to fetch the published provider, but fails because of the local unpublished one.

[apparentlymart]

Indeed, once you are using dev_overrides for one provider you will typically need to use it for all of them, or at least tolerate terraform init failing (it should still install the providers that are accessible, before returning the error) and proceed to run terraform plan or terraform apply anyway.

---

I think the main requirement for a full solution to this problem is ensuring that running terraform init with dev_overrides set can't produce a .terraform.lock.hcl file that would then lead to a failure running without dev_overrides set.

In particular, that means it would not be correct to record a checksum of the override directory in the lock file, because that checksum would typically not be reproducible by anyone running terraform init normally, without any overrides.

Currently we meet this requirement by trying to install the upstream provider anyway and recording the lock file entries that result from it, even though that provider won't actually be used at runtime with the dev override in effect. It's possible that we could meet it a different way by omitting the overridden providers from the lock file altogether; I think if we did that we'd need to generate a loud warning that Terraform has produced an incomplete lock file, because that would subvert the typical terraform init guarantee that if you save the lock file then you will always get exactly the same providers on subsequent runs. Without a lock file entry for a particular provider, a subsequent run of terraform init would see it as an entirely new provider, and thus there would be no verification that the package matches a previously-recorded checksum.

[tekumara]

Indeed, once you are using dev_overrides for one provider you will typically need to use it for all of them, or at least tolerate terraform init failing (it should still install the providers that are accessible, before returning the error) and proceed to run terraform plan or terraform apply anyway.

In this scenario, terraform init will fail, eg:

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Finding latest version of prefecthq/prefect...
- Installing hashicorp/aws v4.50.0...
- Installed hashicorp/aws v4.50.0 (signed by HashiCorp)
╷
│ Warning: Provider development overrides are in effect
│ 
│ The following provider development overrides are set in the CLI configuration:
│  - prefecthq/prefect in /Users/tekumara/go/bin
│ 
│ Skip terraform init when using provider development overrides. It is not necessary and may error unexpectedly.
╵

╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider prefecthq/prefect: provider registry
│ registry.terraform.io does not have a provider named registry.terraform.io/prefecthq/prefect
│ 
│ All modules should specify their required_providers so that external consumers will get the correct providers when
│ using a module. To see which modules are currently depending on prefecthq/prefect, run the following command:
│     terraform providers

And terraform plan will also fail because there is no lock file, and therefore no versions for the non-overridden providers:

❯ terraform plan
╷
│ Warning: Provider development overrides are in effect
│ 
│ The following provider development overrides are set in the CLI configuration:
│  - prefecthq/prefect in /Users/tekumara/go/bin
│ 
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state
│ to become incompatible with published releases.
╵
╷
│ Error: Inconsistent dependency lock file
│ 
│ The following dependency selections recorded in the lock file are inconsistent with the current configuration:
│   - provider registry.terraform.io/hashicorp/aws: required by this configuration but no version is selected
│ 
│ To make the initial dependency selections that will initialize the dependency lock file, run:
│   terraform init

Seems like the only way to make this work is to override all providers or none?

terraform 1.3.7

[apparentlymart]

Yes, if you are developing with multiple providers at once then the easiest path is to make them all development overrides.

This feature is here primarily for testing the one provider you are actively working on, so it's assuming you'll be working with a minimal configuration that only includes exactly what you are testing and nothing else. It's in that case where my suggestion to skip running init will work.

If there are providers involved that are not the ones you are currently actively developing then that's outside the current intended scope for this feature, but you could work around it by pretending that you are developing those providers too and extracting their distribution packages onto your computer somewhere and adding dev_overrides entries for them.

[patmaddox]

I am running into an issue where I can't migrate state from local to S3 because I have a development override. I'm not sure if I should be using a development override - but it is the only way I've found to run a particular provider. Here's some info:

• terraform provider doesn't include a freebsd build, so terraform init fails (PR to include freebsd)
• it builds on FreeBSD, so I built it, tried to stick the binary in a path where TF would find it and not try to download it from registry. I never found anything that worked, so I used a development override, which does.
• terraform init -migrate-state fails with Could not retrieve the list of available versions for provider snowflake-labs/snowflake: no available releases match the given constraints │ ~> 0.64

For the non-development-override approach, I have placed the binary at .terraform/providers/registry.terraform.io/snowflake-labs/snowflake/0.64.0/freebsd_amd64/terraform-provider-snowflake_v0.64.0 but terraform errors with the same error.

So... I need to find some way that lets me run the terraform-provider binary that I built, without looking it up on the registry, and also migrate state to S3.

Any ideas?

[apparentlymart]

Hi @patmaddox,

Development overrides are definitely not the appropriate thing to be using if you are doing anything other than development and testing of a single provider.

For the situation you've described, the more appropriate configuration would be to either explicitly configure a local filesystem mirror or to place the provider package in one of the implied local mirror directories. It sounds like you've tried for the latter but note that .terraform/providers is not one of the documented paths for that. If you have trouble setting up a local mirror, please start a topic in the community forum where we can discuss this further.