-
Notifications
You must be signed in to change notification settings - Fork 9.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support alternate plugin release download hosts #15252
Comments
Hi @adamdecaf! Thanks for this feedback. We're still bringing together all the documentation for this change, so apologies that things are not super clear right now. We are not planning to support an alternate host for downloads in the first release, because we want to introduce such complexity gradually so we can make sure this feature is robust before making it super-generic, but we will continue to support the previous mechanisms for manually installing providers locally, along with some new but similar conventions, and thus enable skipping the auto-download mechanism altogether when running in a production environment. More details on this will come soon, as we finish up the documentation for the release. |
Duplicate of #15997 ? |
Hi all! It's been a while, so I just wanted to share an update on the situation here. As noted before, using the existing infrastructure on Knowing that reaching out to a remote service would not be acceptable for all users, we implemented the At the time of my original comments on this subject there were no specific plans to talk about. In the mean time, we have made a plan to define a protocol that Terraform will expect for a "provider registry", similar in principle to the one already defined for module registries. With that done, we will then change the HashiCorp-distributed providers to be available via that protocol, and invite others to run their own registry servers for internal use. As well as allowing alternative sources for the HashiCorp-distributed providers, this will also allow automatic installation of third-party providers. The Terraform Core team is currently focused on improvements to the configuration language and so will not be able to execute on this plan immediately. However, we have begun conversations with other teams within HashiCorp to figure out all the details of implementing and deploying the new distribution infrastructure and plan to roll this out at some point after the current configuration language work is concluded. I'm sorry that this is a bit of a "no-op" update, but I just wanted to be explicit that this is still a goal and merely a matter of resources/prioritization rather than us not intending to address this. In the mean time, I hope the |
Sounds good. |
Pull requests welcome? |
Thanks for the update @apparentlymart ... This is currently a big deal at my enterprise day-job where there's next to zero chance of us getting a firewall/proxy hole to releases.hashicorp.com Ideally we'd proxy this through something like artifactory or similar caching-repo interface, so we can also leverage security and checksum scanning and the likes. Is there anything resembling a rough ETD/release-target on this? Currently looking at a option to use |
Unfortunately this is one situation where the blocker is not implementation effort but rather design effort and coordination with other teams, since we need to define something that we're going to be able to support in the long term after introducing new features such as third-party plugin installation. Therefore I would ask community members not to propose PRs for this feature for the moment until we're able to better-define the requirements/protocol. Our iterative development style means we're not able to predict the timing of anything other than our current tasks, but the configuration work will take at least a few more months which hopefully gives a sense of the soonest possible time to start this work, which of course does not consider the amount of time it would take to actually implement it since we cannot know that until the design is fully complete. For situations where an explicit
This implicit approach requires careful use of the provider version constraints feature to ensure that selected versions remain consistent with what is available locally, which may or may not be more convenient than adding |
Seems like a reasonable intermediate step to allow the override of the |
Hello! The terraform 0.12 release (anticipated early next year) will include support for terraform providers in the registry. In the first release this will support our official providers, (those hosted on releases.hashicorp.com), and over time we will add support for community providers and alternate release hosts or sources (the details have yet to be determined). Cheers! |
Hi all, Has there ever been a discussion about implementing a
This pattern would allow to have custom providers (we use some very specific providers internally) installed by same mechanism which are used when using 'standard' providers without having additional infra (such as a registry, since s3 for example would do). EDIT: I got a bit off topic here, I'll open up a separate issue... |
Just being able to override the base URL would be sufficient here. I don't particularly want/need to spin up a private registry as I already have releases.hashicorp.com set up as a generic repository in Artifactory so it's effectively a pull-through proxy/cache. Being able to override http(s)://releases.hashicorp.com/ with https://artifactory.example.com/path/to/repository/ would be enough to get things working. |
Would it be possible to get an update or additional clarification on the state of this work? Last update was 5 months ago. We're now into the better part of Q2. More specifically, would be interested in suggestions on how to actually make use of -plugin-dir from end-to-end. To simply say "put the custom provider in this dir" is a bit...vague. Are there any recommendations on how to make this approach salable? Are people just pre-install their private provider on docker containers? or some other strategy? What about using S3 buckets? etc. |
If someone were to submit a PR that allowed optionally overriding the releaseHost var with an environment variable, would that PR have a chance of being accepted? Our use case is simple, we would just like to point this at our internal mirror of releases.hashicorp.com. |
Terraform v0.12 (and thus current master) no longer uses However, if your requirement is just to override the location of the primary "registry" to point at a local proxy, Terraform v0.12 will already support this via a host "registry.terraform.io" {
services = {
"modules.v1": "https://your-mirror.example.com/v1/modules/",
"providers.v1": "https://your-mirror.example.com/v1/providers/",
}
} The Note that the registry is just an index, not the hostname where the providers archives are distributed from. For the moment, that'll still be This issue is, for us, representing the capability to install from third-party registries at different hostnames, rather than overriding the meaning of |
Thanks for the updates @apparentlymart! |
@apparentlymart with v0.12 pointing to As part of the |
The server-side implementation and signature-checking for third-party providers on The work for this generalization is primarily being driven by the team that maintains Terraform Registry (rather than the Terraform Core team) so I don't know when it will be completed in terms of Terraform Core release versions, but I can say that the work is planned out in terms of high-level steps and this initial release supporting only the HashiCorp-distributed providers is one of the first steps in the plan. |
As per @ cullenmcdermott:
It is such a pity that simply because we cannot use the internal (no direct internet) mirror, we need take Terraform off from the tool selection. |
My bad - that just changes the Index URL. There is no way to override the actual base URL where binaries are pulled from. |
So I have been looking at hosting a custom registry for a few internal providers. After digging pretty deep into how its fully implemented and what is required. I stumbled onto this issue #20527 Which requested that the gpg key that originally was pinned inside the bin was reinstated. That was completed in this PR #20543 As it currently stands no one except HashiCorp public key can publish towards any registry including custom ones because of the gpg signing. Is there any goals to allow even a |
Why not a config to specify a different public key? |
@adamdecaf Just allowing a different key inherently means that you have to retrieve and download every provider and then resign with that key. I personally would like to only have to host and sign the providers I maintain. The real goal is establishing trust with the registry (public/private) and confirming that its not been compromised. |
@mildwonkey any updates? |
@circa10a - fair point. This thread is going on 3 years old with a fairly reasonable request |
👋 Hi everyone! I finally have an update on this long-awaited feature for y'all. Back in January we announced our plans for the provider "source" attribute: https://www.hashicorp.com/blog/adding-provider-source-to-hashicorp-terraform And last week, we announced our v0.13 beta release: https://discuss.hashicorp.com/t/terraform-v0-13-0-beta-program/9066 This new source attribute will allow you to specify alternate registry hosts for any given provider. I also see some questions regarding the GPG signatures. The registry response includes a GPG public key and Terraform will use that key to verify providers which aren't signed by HashiCorp. Terraform will display a warning notifying you that the provider binary is self-signed. We are expecting to ship beta 1 on June 3rd, and I'd like to encourage y'all to give it a try (on non-production infrastructure, please!) and let us know what you think and if you run across any issues. Thanks for you patience - it's been a long project, and there's work still to be done for future releases, but I'm excited to finally get to share what I and my team have been working on for the past year. |
@mildwonkey When can we expect more documentation on this? Also I noticed the new format has been backported to 0.12. Will 0.12 actually be able to utilize other sources or is this just for code compatibility? |
Both good questions! This feature will not be available in terraform v0.12: we slowly started introducing the new code in v0.12, but terraform did not (and will not) start parsing source attributes until v0.13. The documentation will get deployed to our website when we release Terraform v0.13 (General Release), tentatively scheduled for 6 weeks after the first beta. More information and documentation will be shared as we get closer to the beta release. |
So I dug into the code and really enjoy what I see 22ef5cc I am curious if we will be able to pin a (Internal/Company) GPG key and disable self-signed packages. The only reason this functionality didn't make it into 0.12.x was because the registry was the source of truth for both package and registry. The Concern in #20527 is still valid for a MITM attacks. For example I modify a provider to target a specific resource. The only difference to an end user is a warning about a provider not being signed. |
If it's not officially supported you could only run TF from CI and have the CI fail if it sees that warning in the plan. Obviously native support would be better though. |
Hi all, There's a lot of discussion here from along the way while this was under design and development, but now that the first iteration of this feature is merged I'm going to close this out so we can fan-out further discussion and feedback into more focused separate issues. If you have feedback after you try out this feature either by compiling from source right now or by using the forthcoming beta releases, please feel free to open an new specific issue and then we'll be better able to engage with each conversation separately. Thanks! |
Terraform 0.13.0 beta 1 launched today, and I'd love to get feedback from people who have been following this issue. I hope you'll try it! |
I noticed in reading the v0.10.0-beta changelog that plugins are now downloaded from a remote source when
terraform init
is ran. I have a concern in that in the context which the team I'm on is running terraform we need to keep a copy of those plugins internal to our networks. (Similar to adding an internal apt cache/proxy.)I'd like to suggest adding some flag to use a different
releaseHost
inplugin/discovery/get.go
.I could think of a few options right now:
terraform
provider. (similar toterraform.required_version
-var
/env variable (TF_PLUGIN_RELEASE_HOST
)The text was updated successfully, but these errors were encountered: