Terraform v0.10.0 3rd party plugin location incorrectly stated

[codycushing]

Terraform Version

v0.10.0

Terraform Configuration Files

NA

Debug Output

NA

Panic Output

NA

Expected Behavior

The previous model for specifying provider location should work (.terraformrc).
The correct path for my system would be outputted.
Sufficient migration documentation would be available to figure this out without searching through code

Actual Behavior

Incorrect path was outputted for where to locate plugin

Steps to Reproduce

1. Update terraform to v.0.10.0 without changing legacy plugin organization
2. run terraform init
3. Observe inaccurate message on where 3rd party plugins should live:

In the latter case, the plugin must be installed manually by locating and
downloading a suitable distribution package and placing the plugin's executable
file in the following directory:
    terraform.d/plugins/darwin_amd64

Actual path that works is ~/.terraform.d/plugins/

Important Factoids

.terraformrc config approach was nice for development because it could point to an arbitrary location like the provider projects /bin folder for development work.

References

NA

[codycushing]

For Windows, 3rd party plugin location is:
%APPDATA%/terraform.d/plugins/

[jbardin]

Hi @codycushing,

Sorry this issue slipped through. It was intended that the legacy .terraformrc providers config get read for plugin discovery as well as .terraform.d, but it seems that mechanism isn't working.

The suggested directory of terraform.d/plugins/darwin_amd64 is correct, and is the intended location to override providers with vendored versions. If that directory doesn't work, then that is a bug.

There is also the -plugin-dir option to init if you want to override all automatic downloading of plugins, and rely solely on the versions provided.

[jbardin]

(sorry, that go-getter issue was mistakingly linked here for a moment, hopefully it will "un-reference" itself eventually)

[tehmaspc]

Have tried a bunch of options to no avail. @codycushing - can you provide your setup again?
I'm trying to setup the BMC TF provider :)

Thanks!

[ghost]

The only way I am able to get third party providers to work:

1. run terraform init
2. copy third part over existing provider tp .terraform provider folder and rename file with v_0.0.1_x4 suffix
3. run terraform init on .terraform provider folder

terraform init -plugin-dir="~/projects/terraform/projectx/.terraform/plugins/darwin_amd64"

this properly registers my softlayer provider override. I would rather it honor the .terraformrc file and use the proper binary and move it and register it.

[apparentlymart]

Hi all,

It sounds like there are some weird behaviors here and Terraform is not behaving exactly as designed, but I want to explain the intended behavior here to help us figure out which parts are not working as expected.

First, the .terraformrc file is meant to still be working as it always did, but it has an additional constraint compared to before: it can only represent "unversioned" providers, and so it doesn't work to use a version constraint (using the new provider block version meta-attribute) with a provider declared in this way. ("Unversioned" internally means version 0.0.0, which is considered "older" than all other versions but should be accepted in the absence of a version constraint that excludes it.)

In the absense of a .terraformrc path override, Terraform then looks for locally-installed plugins in the following locations:

• The current working directory. (Supported for compatibility with 0.9 and earlier, but no longer suggested since it can't represent operating system and architecture.)
• The directory where the terraform executable is installed.
• terraform.d/plugins/OS_ARCH under the current working directory (as indicated in the error message)
• The .terraform/plugins/OS_ARCH directory that the auto-installer writes to (this is not intended to be manipulated manually, since terraform init will overwrite/delete files in here)
• The global configuration directory for the current user, which varies in location depending on OS:
  • The subdirectory terraform.d/plugins of the user's Application Data directory on Windows.
  • The subdirectory .terraform.d of the user's home directory on non-Windows.

The -plugin-dir option overrides all of the above and restricts Terraform to looking only in the specified directory. It can be used multiple times to look in multiple places. This option was provided to satisfy concerns from organizations that want to carefully control which provider plugins are available, and so we suggest using it only for that use-case.

For third-party providers to be used in conjunction with official releases from the HashiCorp repository, placing the file in one of the above directories and naming it with a version suffix like _v0.0.1 is the recommended approach. The x4 suffix won't hurt, but it shouldn't be needed and doesn't do anything for local plugins. (It's used only during auto-installation.)

For most cases I recommend placing custom providers in the same directory as the terraform binary, assuming that the same set of plugins can be used across all configurations you work with. If instead you wish to maintain separate third-party plugins for each configuration, the terraform.d/plugins/OS_ARCH directory is the recommended one, which is why it's in the init output.

Making use of these directories is recommended over the .terraformrc file specifications because, as noted above, the rc file is unable to represent provider version numbers (due to its historical syntax) and so it effectively disables the provider versioning capability.

---

With all of this said, it does seem like there are some bugs here. The two discrepancies I'm seeing in the comments here are:

• Plugins in the terraform.d/plugins/OS_ARCH directory aren't being found.
• Plugins explicitly declared in .terraformrc are no longer being used in preference to auto-discovery.

Does that match what you all are seeing here? Are there other discrepancies not accounted for in this list?

The verbose log output produced by running Terraform with the TF_LOG=debug environment variable includes (amongst lots of other output) log lines of this form that describe where Terraform is looking, aside from the .terraformrc file (which is handled separately):

2017/08/07 12:38:18 [DEBUG] checking for provider in "."
...
2017/08/07 12:38:18 [DEBUG] checking for provider in "/usr/local/bin"
...
2017/08/07 12:38:18 [DEBUG] checking for provider in ".terraform/plugins/darwin_amd64"
...

If any of you are able to share these lines from your debug output (redacting any sensitive information the paths might disclose) that'd be helpful to try to figure out what's going on here. Note that Terraform does not produce any output if one of the directories in the list doesn't exist at all, which is why not all of the directories I listed are shown in the above example output.

We also left a particularly-verbose log line in 0.10.0 that indicates all of the files the discovery process is evaluating, and whether Terraform considers them to be plugins based on the filenames:

2017/08/07 12:38:18 [DEBUG] checking for provider in ".terraform/plugins/darwin_amd64"
2017/08/07 12:38:18 [DEBUG] skipping "lock.json", not a provider
2017/08/07 12:38:18 [DEBUG] found provider "terraform-provider-aws_v0.1.3_x4"
2017/08/07 12:38:18 [DEBUG] found provider "terraform-provider-random_v0.1.0_x4"
2017/08/07 12:38:18 [DEBUG] found provider "terraform-provider-template_v0.1.1_x4"
2017/08/07 12:38:18 [DEBUG] found valid plugin: "random"
2017/08/07 12:38:18 [DEBUG] found valid plugin: "template"
2017/08/07 12:38:18 [DEBUG] found valid plugin: "aws"

(We're going to remove the "skipping ... " message in 0.10.1 since it's overly verbose for most purposes, but it happens to be quite handy here for understanding what files Terraform is finding during the discovery process.)

[ehough]

Plugins in the terraform.d/plugins/OS_ARCH directory aren't being found.

That's what I'm experiencing.

If any of you are able to share these lines from your debug output

Sure! Just FYI I've aliased my terraform binary to tf. i.e.

> which tf
/usr/local/bin/tf

> ls -alh /usr/local/bin/tf
lrwxrwxrwx 1 root staff 9 Jul 24 23:50 /usr/local/bin/tf -> terraform

> ls -alh /usr/local/bin/terraform
-rwxrwxr-x 1 ehough ehough 50M Aug  2 14:06 /usr/local/bin/terraform

> tf version
Terraform v0.10.0

> uname -a
Linux blue 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux

Running plan in a project after upgrading to v0.10, I get the expected message:

> tf plan
Plugin reinitialization required. Please run "terraform init".
Reason: Could not satisfy plugin requirements.

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

4 error(s) occurred:

* provider.aws: no suitable version installed
  version requirements: "(any version)"
  versions installed: none
* provider.cloudflare: no suitable version installed
  version requirements: "(any version)"
  versions installed: none
* provider.template: no suitable version installed
  version requirements: "(any version)"
  versions installed: none
* provider.terraform: no suitable version installed
  version requirements: "(any version)"
  versions installed: none

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints from each module, run "terraform providers".

error satisfying plugin requirements

I ran init, then moved the newly-downloaded plugins to ~/.terraform.d/plugins/linux_amd64 because I want to reuse them across all of my terraform projects. Hope that's the right thing to do?

> ls -alh ~/.terraform.d/plugins/linux_amd64 
total 211M
drwxr-xr-x 2 ehough ehough 4.0K Aug  6 18:24 .
drwxr-xr-x 4 ehough ehough 4.0K Aug  6 18:24 ..
-rwxr-xr-x 1 ehough ehough  328 Aug  5 12:19 lock.json
-rwxr-xr-x 1 ehough ehough  87M Aug  5 12:19 terraform-provider-aws_v0.1.3_x4
-rwxr-xr-x 1 ehough ehough  17M Aug  5 12:19 terraform-provider-cloudflare_v0.1.0_x4
-rwxr-xr-x 1 ehough ehough  17M Aug  5 12:19 terraform-provider-template_v0.1.1_x4
-rwxr-xr-x 1 ehough ehough  91M Aug  5 12:19 terraform-provider-terraform_v0.1.0_x4

Then I try re-running plan, which should find the plugins in my home directory, right?

> TF_LOG=debug tf plan
2017/08/08 12:04:41 [INFO] Terraform version: 0.10.0  2041053ee9444fa8175a298093b55a89586a1823
2017/08/08 12:04:41 [INFO] Go runtime version: go1.8
2017/08/08 12:04:41 [INFO] CLI args: []string{"/usr/local/bin/terraform", "plan"}
2017/08/08 12:04:41 [DEBUG] Attempting to open CLI config file: /home/ehough/.terraformrc
2017/08/08 12:04:41 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2017/08/08 12:04:41 [INFO] CLI command args: []string{"plan"}
2017/08/08 12:04:41 [INFO] command: empty terraform config, returning nil
2017/08/08 12:04:41 [DEBUG] command: no data state file found for backend config
2017/08/08 12:04:41 [DEBUG] New state was assigned lineage "3914e4f9-15d7-400c-a385-228d8160c2b7"
2017/08/08 12:04:41 [INFO] command: backend initialized: <nil>
2017/08/08 12:04:41 [DEBUG] checking for provider in "."
2017/08/08 12:04:41 [DEBUG] skipping "alb.tf", not a provider
2017/08/08 12:04:41 [DEBUG] ...
2017/08/08 12:04:41 [DEBUG] checking for provider in "/usr/local/bin"
2017/08/08 12:04:41 [DEBUG] skipping "docker-compose", not a provider
2017/08/08 12:04:41 [DEBUG] ...
2017/08/08 12:04:41 [DEBUG] checking for provider in "/home/ehough/.terraform.d/plugins"
2017/08/08 12:04:41 [DEBUG] skipping "darwin_amd64", not a provider
2017/08/08 12:04:41 [DEBUG] skipping "linux_amd64", not a provider
2017/08/08 12:04:41 [DEBUG] checking for provisioner in "."
2017/08/08 12:04:41 [DEBUG] skipping "alb.tf", not a provisioner
2017/08/08 12:04:41 [DEBUG] ...
2017/08/08 12:04:41 [DEBUG] checking for provisioner in "/usr/local/bin"
2017/08/08 12:04:41 [DEBUG] skipping "docker-compose", not a provisioner
2017/08/08 12:04:41 [DEBUG] ...
2017/08/08 12:04:41 [DEBUG] checking for provisioner in "/home/ehough/.terraform.d/plugins"
2017/08/08 12:04:41 [DEBUG] skipping "darwin_amd64", not a provisioner
2017/08/08 12:04:41 [DEBUG] skipping "linux_amd64", not a provisioner
2017/08/08 12:04:41 [INFO] Failed to read plugin lock file .terraform/plugins/linux_amd64/lock.json: open .terraform/plugins/linux_amd64/lock.json: no such file or directory
2017/08/08 12:04:41 [INFO] command: backend <nil> is not enhanced, wrapping in local
2017/08/08 12:04:41 [INFO] backend/local: starting Plan operation
2017/08/08 12:04:41 [DEBUG] plugin: waiting for all plugin processes to complete...
Plugin reinitialization required. Please run "terraform init".
Reason: Could not satisfy plugin requirements.

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

4 error(s) occurred:

* provider.aws: no suitable version installed
  version requirements: "(any version)"
  versions installed: none
* provider.cloudflare: no suitable version installed
  version requirements: "(any version)"
  versions installed: none
* provider.template: no suitable version installed
  version requirements: "(any version)"
  versions installed: none
* provider.terraform: no suitable version installed
  version requirements: "(any version)"
  versions installed: none

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints from each module, run "terraform providers".

error satisfying plugin requirements

It looks to me like it's looking for the providers in ~/.terraform.d/plugins instead of ~/.terraform.d/plugins/linux_amd64. I should note that if I leave the plugins in place (don't move them to my home directory), everything works perfectly.

Hope this is useful.

[jbardin]

Just to fill in one piece of the puzzle, I bet I know the change leading to the legacy path ~/.terraform.d/plugins being checked without the OS_ARCH subdirectory, so I can at least understand that behavior.

I am fairly certain that terraform.d/plugins/OS_ARCH work as expected though, as I just tested it and it was confirmed in at least one other issue here. I think people may be getting confused because of the similarity of ~/.terraform.d/plugins, ./.terraform/plugins, and ./terraform.d/plugins

[tehmaspc]

@apparentlymart

FYI - going to retest my 3rd party provider again - but the following piece of info was not available to me - wondering if you can do better output to users when providers are not found in the error message.

For third-party providers to be used in conjunction with official releases from the HashiCorp
repository, placing the file in one of the above directories and naming it with a version suffix like
_v0.0.1 is the recommended approach. The x4 suffix won't hurt, but it shouldn't be needed and
doesn't do anything for local plugins. (It's used only during auto-installation.)

[apparentlymart]

Hi @tehmaspc! Sorry this isn't better documented.

At the moment the best reference to this is in the 0.10 upgrade guide, but indeed we should find a more permanent place for it to live in the docs that describe how to build third-party providers.

[tehmaspc]

@jbardin @apparentlymart

guys - there are bugs here

i have now finally got a terraform init to work but I had to do the following:

- append a version suffix to my provider binary
- copy the provider binary to the local terraform.d/plugins/OS_ARCH (~/.terraform.d/plugins/...) did NOT work

root@ubuntu-xenial:/vagrant/test# terraform init
Downloading modules...
Get: REDACTED
Get: file:///vagrant
Get: file:///vagrant
Get: file:///vagrant

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.baremetal: version = "~> 1.0"
* provider.template: version = "~> 0.1"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
root@ubuntu-xenial:/vagrant/test# tree -f terraform.d/
terraform.d
├── terraform.d/checkpoint_cache
├── terraform.d/checkpoint_signature
└── terraform.d/plugins
    └── terraform.d/plugins/linux_amd64
        └── terraform.d/plugins/linux_amd64/terraform-provider-baremetal_v1.0.16

2 directories, 3 files
root@ubuntu-xenial:/vagrant/test# uname -a
Linux ubuntu-xenial 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[jbardin]

• append a version suffix to my provider binary

Sorry, this is not strictly required, and Terraform will pick up an unversioned plugin from the directories listed. It is recommended though to start versioning plugins, so naming the file accordingly may be a good idea.

• copy the provider binary to the local terraform.d/plugins/OS_ARCH (~/.terraform.d/plugins/...) did NOT work

This is fixed by PR 15769 referenced above.

[tehmaspc]

@jbardin - sorry but w/o a version - nothing was being picked up for the BMC terraform provider I was testing with. Maybe I was doing something wrong but I spent quite a bit of time trying every combination of directory and file setup. I basically followed @Itsguy approach.

[jbardin]

@tehmaspc: Let me know if that's something you can reproduce, because it shouldn't currently be the case.

$ ls terraform.d/plugins/darwin_amd64/
terraform-provider-test

$ TF_LOG=trace terraform init
<...>
2017/08/10 11:21:17 [DEBUG] checking for provider in "terraform.d/plugins/darwin_amd64"
2017/08/10 11:21:17 [WARNING] found legacy provider "terraform-provider-test"
<...>

The log message is [WARNING] because it's not versioned, but otherwise it's treated as a normal plugin with version 0.0.0.

Thanks!

[tehmaspc]

retested - I can confirm now that w/ or w/o the version suffix the local terraform.d/plugins/OS_ARCH does work. The user ~/.terraform.d/plugins does NOT but that is what #15769 fixes. Thus - once #15769 is pushed out to a release stuff will be better. Thanks.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.