Update Extending Terraform documentation for 0.13 plugin directory structure

[rgl]

I'm trying to use nbering/terraform-provider-ansible by placing it in ~/.terraform.d/plugins/linux_amd64/terraform-provider-ansible, but terraform init never seems to actually find it (NB this works fine in 0.12.x). Why is that?

Can 0.13 actually load plugins from local file-system directories (as described in the documentation)? Or does the plugin need to be in a registry?

Can 0.13 init handle mixed plugins locations? that is, from registry and from local file-system plugin-locations?

Using strace I can see terraform find the ~/.terraform.d/plugins/linux_amd64/terraform-provider-ansible binary, but it never executes is (at least is does not appear in TG_LOG).

NB This is somewhat related to nbering/terraform-provider-ansible#29.

[alisdair]

Thanks for reporting this!

Can 0.13 actually load plugins from local file-system directories (as described in the documentation)?

Yes. But the problem you're having is due to the path used. The plugin directory layout has changed, in order to support the new provider source feature. See this section of the upgrade guide for more details.

Can 0.13 init handle mixed plugins locations? that is, from registry and from local file-system plugin-locations?

Yes, that works fine.

I'm leaving this bug open because we should fix the misleading docs on the page you linked.

[rgl]

Ah thank you!

I've got it working in rgl/terraform-ansible-azure-vagrant@2dfbcba :-)

[siassaj]

@alisdair I've been trying to find a relevant issue for a similar problem, I can not for the life of my figure out how to bundle a custom provider binary (in this case https://github.com/mrolla/terraform-provider-circleci) and configure the required_providers block such that Terraform Cloud will locate & use the custom provider.

Should I raise a new issue?

[rgl]

@siassaj there are two options to do this locally (not sure on Terraform Cloud):

The author of https://github.com/mrolla/terraform-provider-circleci publishes the provider to https://registry.terraform.io/ and he lets you known how to use it.

Or you manually download the provider from https://github.com/mrolla/terraform-provider-circleci/releases/download/v0.3.0/terraform-provider-circleci-linux-amd64 and save it in ~/.terraform.d/plugins/registry.github.com.local/mrolla/circleci/0.3.0/linux_amd64/terraform-provider-circleci_v0.3.0 (or at the machine level, /usr/local/share/terraform/plugins/registry.github.com.local/mrolla/circleci/0.3.0/linux_amd64/terraform-provider-circleci_v0.3.0) then use it as:

terraform {
  required_providers {
    circleci = {
      source = "registry.github.com.local/mrolla/circleci"
      version = "~> 0.3.0"
    }

This is similar to what I did in rgl/terraform-ansible-azure-vagrant@2dfbcba to use https://github.com/nbering/terraform-provider-ansible.

[siassaj]

@rgl Thank you for that clarification. We have a bunch of workspaces using 0.13 on Terraform Cloud where we'd like to use the custom providers. Guess I'll have to keep looking...

[alisdair]

I've been trying to find a relevant issue for a similar problem, I can not for the life of my figure out how to bundle a custom provider binary (in this case https://github.com/mrolla/terraform-provider-circleci) and configure the required_providers block such that Terraform Cloud will locate & use the custom provider.

Should I raise a new issue?

@siassaj This isn't a bug in Terraform, so please don't raise an issue here. Instead, if you ask a question on the Terraform discussion forum, hopefully someone will be able to help out. As a starting point, check out the documentation for using custom providers with Terraform Cloud, which has not yet been updated for the new filesystem structure.

[taiidani]

Related question -- is there a way to have the local plugin installation directory versionless? I don't see any examples of this in the docs.

The use case being for local development in feature branches where you have not yet tagged a new version. Right now I've been faking Terraform out by hardcoding a version that the required_providers line allows, but that requires committing a dummy version to my makefile and keeping it up to date as new versions are tagged.

[taiidani]

It looks like Terraform does support version less directories, assuming the Writing Custom Providers tutorial has been updated for 0.13. That uses a hashicorp.com/edu/hashicups namespace.

Just poking at it though, I wasn't able to get it to pick up my provider binary. Hmm. Would love for the documentation to assist with this use case.

EDIT: Well that's misleading! The terminal output from make install in the tutorial (versionless) doesn't match the actual committed makefile (hardcoded version requiring updating). So now I'm not sure again!

[alisdair]

@taiidani Thanks for reporting that issue! I've reached out to the Learn team to update the tutorial.

You're right to conclude that we currently don't have any way to have Terraform use unversioned providers. If you have time, perhaps you could submit an enhancement issue to explain your ideal workflow for developing a provider?

[taiidani]

@alisdair No problem! Enhancement request submitted alongside a couple potential proposals: hashicorp/terraform#25906

[siassaj]

For anyone interested on 0.13.0 & 0.13.1 (and maybe other 0.13.x) Terraform Cloud will find your Providers under

<working directory>/terraform.d/plugins/<some custom registry hostname>/<some namespace>/<provider name>/<provider version>/linux_amd64/terraform-provider-<provider-name>_v<provider version>

A concrete example in a monorepository:

# main.tf

terraform {
  backend "remote" {
    hostname     = "app.terraform.io"
    organization = "my-org"

    workspaces {
      prefix = "some-prefix"
    }
  }

  required_providers {
    circleci = {
      source  = "registry.local/mrolla/circleci"
      version = "0.3.0"
    }
  }
}

# terraform repository `tree` command output

.
├── locals.tf
├── main.tf
├── terraform.d
│   └── plugins
│       └── registry.local
│           └── mrolla
│               └── circleci
│                   ├── 0.3.0
│                   │   └── linux_amd64
│                   │       └── terraform-provider-circleci_v0.3.0
│                   └── README.md

A concrete example in a mixed env repository:

Terraform Cloud also allows submodules (to prevent the large provider binaries from blowing up your main terraform repo) and will follow symlinks.

Just make sure to bundle up the provider binary in your submodule repo in a way that makes it easy to set up the correct directory structure in the terraform repo.

# main.tf

terraform {
  backend "remote" {
    hostname     = "app.terraform.io"
    organization = "my-org"

    workspaces {
      prefix = "some-prefix"
    }
  }

  required_providers {
    circleci = {
      source  = "registry.local/mrolla/circleci"
      version = "0.3.0"
    }
  }
}

# .gitmodules

[submodule "plugins/registry.local/mrolla/circleci"]
	path = plugins/registry.local/mrolla/circleci
	url = https://bitbucket.org/my-org/terraform-provider-circleci

# terraform repository `tree` command output

.
├── stg
│   ├── main.tf
│   └── terraform.d
│       └── plugins -> ../../plugins #symlink 
├── prd
│   ├── main.tf
│   └── terraform.d
│       └── plugins -> ../../plugins #symlink 
├── plugins
│   └── registry.local
│       └── mrolla
│           └── circleci
│               ├── 0.3.0 # this is the root of the git submodule
│               │   └── linux_amd64
│               │       └── terraform-provider-circleci_v0.3.0
│               └── README.md

# `tree` output for the terraform provider repo that containes the provider binary & is used as a submodule

.
├── 0.3.0
│   └── linux_amd64
│       └── terraform-provider-circleci_v0.3.0
└── README.md

Just make sure to check Include submodules on clone in Settings -> Version Control

[Setheck]

Another example that doesn't make sense to me
Looking at plugins that can be downloaded from
https://releases.hashicorp.com/terraform-provider-aws/3.15.0/terraform-provider-aws_3.15.0_darwin_amd64.zip
https://releases.hashicorp.com/terraform-provider-docker/2.7.2/terraform-provider-docker_2.7.2_darwin_amd64.zip

However, they need to have the following structure to work with terraform 0.13 in a local plugin-dir plugin scenario

└── registry.terraform.io
    ├── hashicorp
    │   └── aws
    │       └── 3.15.0
    │           └── darwin_amd64
    │               └── terraform-provider-aws_v3.15.0_x5
    └── terraform-providers
        └── docker
            └── 2.7.2
                └── darwin_amd64
                    └── terraform-provider-docker_v2.7.2_x4

Originally I wanted to programatically download and setup a local plugin dir, but now i'm not sure how I decide if it needs to go into hashicorp or terraform-providers ?

[apparentlymart]

The forthcoming Terraform v0.14 release (expected in a few weeks) includes a new CLI configuration block that allows configuring Terraform to ignore copies of a particular provider installed in the usual way and instead force use of a particular local directory. This mechanism is separate from the normal provider installer and so it can disregard concerns like version numbers and target platforms.

provider_installation {

  # Use /home/developer/tmp/terraform-null as an overridden package directory
  # for the hashicorp/null provider. This disables the version and checksum
  # verifications for this provider and forces Terraform to look for the
  # null provider plugin in the given directory.
  dev_overrides {
    "hashicorp/null" = "/home/developer/tmp/terraform-null"
  }

  # For all other providers, install them directly from their origin provider
  # registries as normal. If you omit this, Terraform will _only_ use
  # the dev_overrides block, and so no other providers will be available.
  direct {}
}

go build -o /home/developer/tmp/terraform-null/terraform-provider-null

For more information on what this will look like in Terraform 0.14, you can refer to the draft of the new CLI Configuration configuration, which is the content that will be published on the CLI Configuration page on the website once we release v0.14.0 final. (Internal links in our documentation pages are written for the URL structure on the published website, so they typically won't function correctly in GitHub's rendering of the content.)

From reading the history of this discussion, it seemed like this issue ended up representing the task of updating the "Extending Documentation" section of the website to reflect the new approach. It seems like that's still to do, and I'd recommend updating it to just link to the relevant sections of the CLI configuration rather than duplicating the information, because the CLI configuration reference is what we'll update with additional information in future.

The "Extending Terraform" section of the website is maintained by the Terraform SDK team rather than the Terraform Core team, and that content lives in the hashicorp/terraform-website repository rather than the hashicorp/terraform repository. To reflect that, I'm going to transfer this issue into the website repository and mention @hashicorp/terraform-enablement-sdk to draw attention to it.