Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

terraform-providers: Terraform 0.13 compatibility #99198

Merged
merged 7 commits into from Oct 8, 2020

Conversation

@timstott
Copy link
Contributor

@timstott timstott commented Sep 30, 2020

Motivation for this change

Terraform 0.13 introduces a new filesystem layout to store providers which is incompatible with Terraform 0.12. The new filesystem layout is informed by the Terraform Registry, a repository of providers and modules.

The objectives of this work are:

  • Make existing terraform providers available to Terraform 0.13 where possible
  • Change the update script to use the Terraform Registry

Modified

  • Added a provider-source-address attribute to providers to enable the creation of the filesystem layout.
  • The update provider script
    • Script uses Terraform Registry to fetch latest version
    • Script accepts and updates one provider at a time
  • Providers stored in providers.json instead of data.nix to use jq only instead of nix + jq combination
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Fixed #98652

@timstott timstott force-pushed the terraform-013-compatible-providers branch from edbd38e to 0860356 Sep 30, 2020
# Place provider in Terraform 0.13 search location
TF_13_PROVIDER_PATH=go/bin/${data.provider-source-address}/${version}/$(go env GOARCH)/
mkdir -p $TF_13_PROVIDER_PATH
ln -s go/bin/${repo}_v${version} $TF_13_PROVIDER_PATH
Copy link
Contributor Author

@timstott timstott Sep 30, 2020

I went with the symlink approach to avoid making two copies of the binary, but it doesn't work.

I also question whether it's toDrv responsibility to make the binary available in the correct location as opposed to only building. I'm open to recommendations 🙃.

Copy link
Member

@zimbatm zimbatm Oct 1, 2020

A wrapper script that execs the providers should be a good alternative. It's the approach we are taking in https://github.com/numtide/generate-terraform-provider-shim to download providers on demand and it works fine there.

"acme": {
"owner": "getstackhead",
"provider-source-address": "registry.terraform.io/getstackhead/acme",
"repo": "terraform-provider-acme",
"rev": "v1.5.0-patched",
"sha256": "1wdrjpd3l0xadsa3lqhsc9c57g8x2qkwb76q824sk8za1a7lapii",
"version": "1.5.0-patched"
},
Copy link
Contributor Author

@timstott timstott Sep 30, 2020

I have been using the acme provider to test. The remaining "provider-source-address" will be added within this PR.

Copy link
Member

@zimbatm zimbatm left a comment

Thanks, that's great!

The question is, how complete do you want the solution to be and this depends on the energy that you have available. The current PR is already a quite good solution. It could be made better by integrating with the updateScript framework, and by allowing to get new providers from the registry.

set -euo pipefail

# TODO: document usage
provider_name="$1"
Copy link
Member

@zimbatm zimbatm Oct 1, 2020

Do you see an easy way to list and/or update all of the providers from the registry?

Copy link
Contributor Author

@timstott timstott Oct 2, 2020

I'm inclined to provide two additional scripts, one to add a provider to providers.json from it's registry URL (as opposed to exhaustively add all providers). And one script to update all known providers.

# update-all-providers
jq 'keys[]' providers.json | xargs ./update-provider

I should be able to also re-use much of update-provider to create add-provider registry.terraform.io/some/provider

Any thoughts?

I will absolutely look at using update-provider with the updateScript framework ❤️

Copy link
Member

@zimbatm zimbatm Oct 2, 2020

Everything you said sounds good to me!

@timstott timstott force-pushed the terraform-013-compatible-providers branch 2 times, most recently from cc4a818 to ba4d7b2 Oct 6, 2020
@timstott timstott force-pushed the terraform-013-compatible-providers branch 3 times, most recently from ecdd9ac to 211f7bb Oct 6, 2020
@zimbatm
Copy link
Member

@zimbatm zimbatm commented Oct 8, 2020

Result of nixpkgs-review pr 99198 1

2 packages built:
  • terraform-providers.acme
  • terraform_plugins_test

Added provider source address to providers hosted in the same GitHub
repository. Other providers have moved repository and would require a
patch to the package path.
@timstott timstott force-pushed the terraform-013-compatible-providers branch from 9429fe9 to 172e74d Oct 8, 2020
@timstott timstott marked this pull request as ready for review Oct 8, 2020
zimbatm
zimbatm approved these changes Oct 8, 2020
Copy link
Member

@zimbatm zimbatm left a comment

excellent

toDrv = name: data:
let
fallbackProviderSourceAddress = "nixpkgs/${data.owner}/${name}";
providerSourceAddress = lib.attrByPath ["provider-source-address"] fallbackProviderSourceAddress data;
Copy link
Member

@zimbatm zimbatm Oct 8, 2020

this is a little bit cleaner, the or operator is triggered if the key doesn't exist.

Suggested change
providerSourceAddress = lib.attrByPath ["provider-source-address"] fallbackProviderSourceAddress data;
providerSourceAddress = data.provider-source-address or fallbackProviderSourceAddress;

Copy link
Contributor Author

@timstott timstott Oct 8, 2020

Much nicer ❤️. I've incorporated the change.

timstott added 2 commits Oct 8, 2020
Terraform 0.13 adopts a new filesystem layout for plugins (illustrated below).

Terraform 0.12 and earlier
`plugins-dir/terraform-provider-aws_v3.7.0`

Terraform 0.13
`plugins-dir/registry.terraform.io/hashicorp/aws/3.7.0/linux_amd64/terraform-provider-aws_v3.7.0`

To support all packaged Terraform versions a shim is created at both locations.

This approach was inspired by https://github.com/numtide/generate-terraform-provider-shim

Terraform 0.13 provider documentation https://www.terraform.io/upgrade-guides/0-13.html#new-filesystem-layout-for-local-copies-of-providers

layout terraform providers filesystem in withPlugins
@timstott timstott force-pushed the terraform-013-compatible-providers branch from 172e74d to b0a0ffb Oct 8, 2020
@timstott timstott requested a review from zimbatm Oct 8, 2020
@zimbatm zimbatm merged commit be6e9f1 into NixOS:master Oct 8, 2020
2 of 4 checks passed
2 of 4 checks passed
@github-actions[bot]
tests tests
Details
@github-actions[bot]
action
Details
@github-actions[bot]
Wait for ofborg This failed status will be cleared when ofborg finishes eval.
Details
@ofborg[bot]
grahamcofborg-eval Merging PR
Details
@zimbatm
Copy link
Member

@zimbatm zimbatm commented Oct 8, 2020

thanks!

@timstott timstott deleted the terraform-013-compatible-providers branch Oct 8, 2020
veprbl
Copy link
Member

veprbl commented on cd1b594 Oct 9, 2020

This broke eval @zimbatm @zowoq

zowoq
Copy link
Contributor

zowoq commented on cd1b594 Oct 9, 2020

Not sure why I'm being pinged for this?

veprbl
Copy link
Member

veprbl commented on cd1b594 Oct 9, 2020

@zowoq Sorry, thought this was relevant to the .github/workflows/pending-set.yml

doronbehar
Copy link
Contributor

doronbehar commented on cd1b594 Oct 9, 2020

I think the issues is that not all plugins or what ever have this passthru.

timstott
Copy link
Contributor

timstott commented on cd1b594 Oct 9, 2020

I think you're on the money. We need to make this conditional based on the presence of providerSourceAddress.

https://github.com/nixos/nixpkgs/blob/cd0065d18494afe01e42fb60ccbf6422ddb8c60b/pkgs/applications/networking/cluster/terraform/default.nix#L73-L77

timstott
Copy link
Contributor

timstott commented on cd1b594 Oct 9, 2020

@jonringer
Copy link
Contributor

@jonringer jonringer commented Oct 9, 2020

specifically cd1b594 broke it

veprbl
Copy link
Member

veprbl commented on cd1b594 Oct 9, 2020

This broke eval @zimbatm @zowoq

zowoq
Copy link
Contributor

zowoq commented on cd1b594 Oct 9, 2020

Not sure why I'm being pinged for this?

veprbl
Copy link
Member

veprbl commented on cd1b594 Oct 9, 2020

@zowoq Sorry, thought this was relevant to the .github/workflows/pending-set.yml

doronbehar
Copy link
Contributor

doronbehar commented on cd1b594 Oct 9, 2020

I think the issues is that not all plugins or what ever have this passthru.

timstott
Copy link
Contributor

timstott commented on cd1b594 Oct 9, 2020

I think you're on the money. We need to make this conditional based on the presence of providerSourceAddress.

https://github.com/nixos/nixpkgs/blob/cd0065d18494afe01e42fb60ccbf6422ddb8c60b/pkgs/applications/networking/cluster/terraform/default.nix#L73-L77

timstott
Copy link
Contributor

timstott commented on cd1b594 Oct 9, 2020

@zimbatm
Copy link
Member

@zimbatm zimbatm commented Oct 9, 2020

Sorry for the inconvenience. I will have a fix ready in 3h.

EDIT: #100072

@voobscout
Copy link
Contributor

@voobscout voobscout commented Nov 2, 2020

Please look into it again.
shell.nix:

let
  pkgs = import (builtins.fetchTarball "https://api.github.com/repos/nixos/nixpkgs/tarball/master") { };
in
pkgs.mkShell {
  buildInputs = [
    (pkgs.terraform_0_13.withPlugins (p: [
      p.null
      p.helm
      p.vultr
      p.digitalocean
      p.kubernetes
    ]))
  ];
}
nix-shell --run "terraform init"

produces:

Initializing the backend...

Initializing provider plugins...
- Using previously-installed hashicorp/null v2.1.2
- Using previously-installed hashicorp/helm v1.2.2
- Finding latest version of hashicorp/kubernetes...
- Finding latest version of hashicorp/digitalocean...
- Finding latest version of hashicorp/vultr...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
hashicorp/kubernetes: provider registry.terraform.io/hashicorp/kubernetes was
not found in any of the search locations

- /nix/store/6bgi4zxw3xvpni361hs8l7xf9d33cy3d-terraform-0.13.5/plugins


Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
hashicorp/digitalocean: provider registry.terraform.io/hashicorp/digitalocean
was not found in any of the search locations

- /nix/store/6bgi4zxw3xvpni361hs8l7xf9d33cy3d-terraform-0.13.5/plugins


Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
hashicorp/vultr: provider registry.terraform.io/hashicorp/vultr was not found
in any of the search locations

- /nix/store/6bgi4zxw3xvpni361hs8l7xf9d33cy3d-terraform-0.13.5/plugins

looking at plugins/registry.terraform.io, we find 2 dirs - hashicorp and nixpkgs.

ls /nix/store/6bgi4zxw3xvpni361hs8l7xf9d33cy3d-terraform-0.13.5/plugins/registry.terraform.io/
# dr-xr-xr-x 3 root 3 Jan  1  1970 helm/
# dr-xr-xr-x 3 root 3 Jan  1  1970 null/

Not found plugins ended up in plugins/registry.terraform.io/nixpkgs:

ls /nix/store/6bgi4zxw3xvpni361hs8l7xf9d33cy3d-terraform-0.13.5/plugins/registry.terraform.io/nixpkgs
# dr-xr-xr-x 3 root 3 Jan  1  1970 digitalocean/
# dr-xr-xr-x 3 root 3 Jan  1  1970 kubernetes/
# dr-xr-xr-x 3 root 3 Jan  1  1970 vultr/

@arcnmx
Copy link
Member

@arcnmx arcnmx commented Nov 2, 2020

Currently you need something like...

builtins.toJSON {
  terraform.required_providers = mapAttrs (name: plugin: {
    version = plugin.version;
    source = plugin.provider-source-address or "nixpkgs/${name}";
  }) { inherit (pkgs.terraform.plugins) null helm vultr digitalocean kubernetes; };
}

for providers without the source address set, or just:

terraform {
  required_providers {
    helm = {
      source = "nixpkgs/helm"
    }
  }
}

@voobscout
Copy link
Contributor

@voobscout voobscout commented Nov 3, 2020

attribute 'required_providers' missing, can you please supply a full example?

@timstott
Copy link
Contributor Author

@timstott timstott commented Nov 3, 2020

@voobscout the snippet below, to execute in the nix repl, will output the provider definition in nix and will have to be adapted to HCL in your main.tf

$ nix repl
:l https://api.github.com/repos/nixos/nixpkgs/tarball/master

providers = (lib.getAttrs
  ["null" "helm" "vultr" "digitalocean" "kubernetes"]
  pkgs.terraform-providers)

required_providers = lib.mapAttrs (name: plugin: {
  version = plugin.version;
  source = plugin.provider-source-address or "nixpkgs/${name}";
}) providers

:p { terraform = { inherit required_providers; }; }

This is highlighting an improvement I have been meaning to make, to include the provider declaration snippet in the package description (much like on the Terraform registry - screenshot below).

image

@voobscout
Copy link
Contributor

@voobscout voobscout commented Nov 4, 2020

That makes a lot of sense now, thank you @timstott!

@ivankovnatsky
Copy link
Contributor

@ivankovnatsky ivankovnatsky commented Nov 22, 2020

guys, maybe you can point me what i'm missing here:

- Using previously-installed nixpkgs/kubernetes v1.11.3
- Using previously-installed hashicorp/local v2.0.0
- Finding hashicorp/kubernetes versions matching "~> 1.10, >= 1.11.1"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
hashicorp/kubernetes: provider registry.terraform.io/hashicorp/kubernetes was
not found in any of the search locations

- /nix/store/cfldgk8hx5zfc3nzij52h149q6l1bnns-terraform-0.13.5/plugins

[terragrunt] 2020/11/22 18:25:40 Hit multiple errors:
exit status 1cat versions.tf
terraform {
  required_providers {
    helm = {
      source = "nixpkgs/kubernetes"
    }
  }
}

sorry for posting in merged pull request.

@arcnmx
Copy link
Member

@arcnmx arcnmx commented Nov 22, 2020

guys, maybe you can point me what i'm missing here:

    helm = {
      source = "nixpkgs/kubernetes"
    }

I'm pretty sure you want that to be kubernetes = {

@ivankovnatsky
Copy link
Contributor

@ivankovnatsky ivankovnatsky commented Nov 22, 2020

sorry to confuse, I had correct versions.tf, was just copy pasting from your comment before posting my comment:

- Finding hashicorp/kubernetes versions matching ">= 1.11.1"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
hashicorp/kubernetes: provider registry.terraform.io/hashicorp/kubernetes was
not found in any of the search locations

- /nix/store/cfldgk8hx5zfc3nzij52h149q6l1bnns-terraform-0.13.5/plugins

[terragrunt] 2020/11/22 19:58:44 Hit multiple errors:
exit status 1cat versions.tf
terraform {
  required_providers {
    kubernetes = {
      source = "nixpkgs/kubernetes"
    }
  }
}cat .terraform/plugins/registry.terraform.io/nixpkgs/kubernetes/1.11.3/linux_amd64/terraform-provider-kubernetes_v1.11.3
#!/nix/store/y0sw8sxkr7kwlsndwixyf4s4snlfhjh1-bash-4.4-p23/bin/bash
exec /nix/store/cx3k25sc0bvsw0im70gwl8m6yf8g9vng-terraform-provider-kubernetes-1.11.3/bin/terraform-provider-kubernetes_v1.11.3 "$@"

provider appears to be there, though i'm not sure yet why it does fail to initialize.

@ivankovnatsky
Copy link
Contributor

@ivankovnatsky ivankovnatsky commented Nov 23, 2020

so i did edit also eks module:

cat .terraform/modules/eks/versions.tf
terraform {
  required_providers {
    kubernetes = {
      source = "nixpkgs/kubernetes"
    }
  }
}

and did this trick:

terragrunt state replace-provider registry.terraform.io/hashicorp/kubernetes nixpkgs/kubernetes

which is incredibly ugly.

now it initiates fine:

Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Using previously-installed hashicorp/external v2.0.0
- terraform.io/builtin/terraform is built in to Terraform
- Using previously-installed hashicorp/template v2.2.0
- Using previously-installed hashicorp/local v2.0.0
- Using previously-installed hashicorp/random v3.0.0
- Using previously-installed nixpkgs/kubernetes v1.11.3
- Using previously-installed hashicorp/null v3.0.0
- Using previously-installed hashicorp/aws v2.65.0

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/external: version = "~> 2.0.0"
* hashicorp/local: version = "~> 2.0.0"
* hashicorp/null: version = "~> 3.0.0"
* hashicorp/random: version = "~> 3.0.0"
* hashicorp/template: version = "~> 2.2.0"

Terraform has been successfully initialized!

@grahamc
Copy link
Member

@grahamc grahamc commented Dec 2, 2020

I had a pretty hard time with this, and want to call out the replace-provider of the previous comment.

I had this in my .tf files:

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }

    local = {
      source = "hashicorp/local"
    }

    null = {
      source = "hashicorp/null"
    }
  }
}

I'd then run terraform init and it'd look fine:

$ terraform init ./terraform
Initializing modules...
- xxx in xxx/terraform/xxx

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/local...
- Finding latest version of hashicorp/null...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/local v1.4.0...
- Installed hashicorp/local v1.4.0 (unauthenticated)
- Installing hashicorp/null v2.1.2...
- Installed hashicorp/null v2.1.2 (unauthenticated)
- Installing hashicorp/aws v2.65.0...
- Installed hashicorp/aws v2.65.0 (unauthenticated)

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/aws: version = "~> 2.65.0"
* hashicorp/local: version = "~> 1.4.0"
* hashicorp/null: version = "~> 2.1.2"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

and then I'd go to do an apply, and it fails:

$ terraform apply \
	-state="./xxx/terraform.tfstate" \
	-input=false \
	-var-file="./xxx/terraform.tfvars.json" \
	./terraform

Error: Could not load plugin


Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

2 problems:

- Failed to instantiate provider "registry.terraform.io/-/aws" to obtain
schema: unknown provider "registry.terraform.io/-/aws"
- Failed to instantiate provider "registry.terraform.io/-/local" to obtain
schema: unknown provider "registry.terraform.io/-/local"

My shell.nix looked like this:

{ pkgs ? import <nixpkgs> { } }:
pkgs.mkShell {
  buildInputs = [
    (pkgs.terraform_0_13.withPlugins
      (p: [ p.aws p.template p.external p.null p.local ]))
  ];
}

The issue is actually a problem with the 0.12 to 0.13 migration, NOT a problem with this PR.

Our providers were considered "in-house" before, and have less information than terraform wants for the automatic upgrade. This means we need to follow this: https://www.terraform.io/upgrade-guides/0-13.html#in-house-providers

I needed to run:

$ terraform state replace-provider -state="./xxx/terraform.tfstate" 'registry.terraform.io/-/aws' 'hashicorp/aws'

$ terraform state replace-provider -state="./xxx/terraform.tfstate" 'registry.terraform.io/-/local' 'hashicorp/local'

and that fixed the issue with:

- Failed to instantiate provider "registry.terraform.io/-/aws" to obtain
schema: unknown provider "registry.terraform.io/-/aws"

@grahamc
Copy link
Member

@grahamc grahamc commented Mar 11, 2021

Note if you see an error like:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/aws...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
provider registry.terraform.io/hashicorp/aws was not found in any of the
search locations

- /nix/store/fam6azhh0wlpaxa27rbxkjyqzrla50xn-tf-plugin-env/bin

it might be because you're on 20.09 and 20.09 doesn't have this PR applied.

@ivankovnatsky
Copy link
Contributor

@ivankovnatsky ivankovnatsky commented Mar 11, 2021

sorry to bring it here, if you have something to say from your experience please share your thoughts here: https://discourse.nixos.org/t/public-cloud-infrastructure-people-whats-you-workflow-with-terraform-terragrunt-providers/11669/4

for a couple of months trying to use nixos provided terraform and providers i faced multiple issues contributing to many people infra projects, like:

  • figuring out how to configure tf in nixos to install only needed providers and not to install all of them
  • issues mentioned in this pr
  • figuring out how to pin the tf version (i tried to overlay 0.14.4 and it failed to build, i did not look into it too deeply)
  • nixos built providers, which might be not always up to date

also tf 0.14 introduced a lock file to keep providers versions/hashes consistent for all contributors in repo, in case of nixos providers these hashes would be different, meaning we can't use this lock file in the same repo, me NixOS user and other engineers.

again, i might using it all wrong, please share your workflow if my attitude towards nixos tf and providers are rather wrong.

and yeah, just wondering what is the use case to manage/build providers in nixos, rather than using hashicorp ones? (that might be due to nixos source model, if that's the case I could not agree more).

but for me, it brings much more inconvenience (maybe due to my workflow).

@anandsuresh
Copy link

@anandsuresh anandsuresh commented Jun 30, 2021

UPDATE: Fixed the issue. I was missing a postBuild script in the buildGoModule that was causing the compiled provider binary to NOT have the version number appended to the filename, as expected. Once I added that postBuild step, it works like a charm! Sorry for the noise.

I'm seeing this in nixpkgs v21.05. In my case, I'm building my TF modules from source, and they are being dropped into the /nix/store/qz2ahhd4g3nsrjlykwgs54ssb42lz8ck-terraform-0.15.4/plugins. I can see the compiled providers there, but I am still receiving an error.

$ terraform  init

Initializing the backend...

Initializing provider plugins...
- Finding cloudflare/cloudflare versions matching "2.23.0"...
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider cloudflare/cloudflare: provider registry.terraform.io/cloudflare/cloudflare was not found in any of
│ the search locations
│
│   - /nix/store/qz2ahhd4g3nsrjlykwgs54ssb42lz8ck-terraform-0.15.4/plugins

$ ls -la /nix/store/qz2ahhd4g3nsrjlykwgs54ssb42lz8ck-terraform-0.15.4/plugins
total 16
dr-xr-xr-x 7 anand admin 224 Dec 31  1969 .
dr-xr-xr-x 4 anand admin 128 Dec 31  1969 ..
dr-xr-xr-x 3 anand admin  96 Dec 31  1969 registry.terraform.io
-r-xr-xr-x 1 anand admin 201 Dec 31  1969 terraform-provider-cloudflare_v2.23.0
-r-xr-xr-x 1 anand admin 195 Dec 31  1969 terraform-provider-external_v2.1.0
-r-xr-xr-x 1 anand admin 187 Dec 31  1969 terraform-provider-null_v3.1.0
-r-xr-xr-x 1 anand admin 193 Dec 31  1969 terraform-provider-proxmox_v2.7.2

$ terraform providers

Providers required by configuration:
.
└── provider[registry.terraform.io/cloudflare/cloudflare]

The suggested fix to run terraform state replace-provider does not apply to my case since this is a new codebase and there is no .tfstate file at this stage. Has anyone else encountered this issue before?

Update: If I switch my configuration to to use nixpkgs/cloudflare as opposed to cloudflare/cloudflare as the source in the required_providers block, the terraform init succeeds. However, all subsequent commands fail with the same error as documented above by @grahamc.

$ terraform  init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of nixpkgs/cloudflare...
- Installing nixpkgs/cloudflare v2.23.0...
- Installed nixpkgs/cloudflare v2.23.0 (unauthenticated)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ terraform validate
╷
│ Error: Could not load plugin
│
│
│ Plugin reinitialization required. Please run "terraform init".
│
│ Plugins are external binaries that Terraform uses to access and manipulate
│ resources. The configuration provided requires plugins which can't be located,
│ don't satisfy the version constraints, or are otherwise incompatible.
│
│ Terraform automatically discovers provider requirements from your
│ configuration, including providers used in child modules. To see the
│ requirements and constraints, run "terraform providers".
│
│ failed to instantiate provider "registry.terraform.io/nixpkgs/cloudflare" to obtain schema: Unrecognized remote plugin message:
│
│ This usually means that the plugin is either invalid or simply
│ needs to be recompiled to support the latest protocol.
│

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.