Use provider within the registry of Terraform hashicorp.

[paultyng]

👋 Hi, I'm on the Terraform Providers team at HashiCorp. With the release of the Terraform 0.13 beta, users can now download and install community providers from the registry. We are inviting provider authors (especially those for popular community providers) to publish their providers in a closed beta.

To get invited to the closed beta, please email terraform-registry-beta@hashicorp.com. We need:

• A list of GitHub usernames to add to the beta (you and any team members who will publish)
• List of provider repositories you plan to publish
• A GPG public key in ASCII-armor format, which you will be using to sign your provider releases

You can use one key for all of your providers, or separate keys if you prefer. If you are publishing from an organization, this key or keys will be associated with that namespace. Once in the beta, you can manage personal keys in the UI as well.

[MalloZup]

@paultyng hi thx for feedback. We will come to you on this soon

[limburgher]

Hi! terraform 0.13 was just released publicly, and I'm hitting this. I'm trying to add a source entry to use the local binary but I can't work out the syntax. Do you anticipate this being published soon?

[MalloZup]

@limburgher hi ! yes I plan to update the provider to TF.0 13 soonish. We plan to release a first a fixes releases then the next one will be the TF 0.13.. stay tuned 🌻

[paultyng]

👍 Let me know if there is anything I can do to help. This should be self-serve now in the registry UI, you just need to create releases that conform to the publishing criteria (we have a goreleaser config file that can do it).

[wenerme]

@MalloZup also blocked by this, don't find a way to add locally built provider

[apparentlymart]

Hi all!

It looks like this provider will not be straightforward to publish via the registry, because unlike most providers it has a separate build for each Linux distribution, presumably because it's dynamically linking against other libraries that have different ABIs in different distributions. Terraform Registry supports only one package per OS+arch, and so all of the release packages would come under the same linux_amd64 target. Therefore it will probably take some thinking to get this provider set up to auto-install with Terraform v0.13.

To take pressure off the provider maintainers a bit and hopefully get you all moving again, I wanted to let you all know how you can update your locally-installed copies of this provider to work with Terraform v0.13 so that you don't need to be blocked until the provider is published in the public registry.

From Terraform's perspective, a provider that isn't listed in a registry comes under the same guidance as for so-called "In-house Providers". The upgrade guide includes some general guidance for these, but since here we're talking specifically about the libvirt provider I can fill in the gaps for you all a bit more by giving some specific examples.

First, I'm going to make the assumption that this provider will eventually be published in the Terraform Registry as dmacvicar/libvirt, because that's how the naming convention would treat this repository's current location. That address is itself a shorthand for registry.terraform.io/dmacvicar/libvirt, which is the full address we'll use to set this up locally.

Next we'll need to identify a directory to install the plugin into. The installer is configurable for different directories, but for simplicity here I'll use the primary recommended location. This provider only has releases for Linux, and on Linux Terraform v0.13 searches for plugins in ~/.local/share/terraform/plugins by default. If you don't already have that directory, you can create it:

mkdir -p ~/.local/share/terraform/plugins

Now you can download an appropriate package for provider version v0.6.2, or choose a different version and adjust the version references that follow. Extract your package into the following directory:

~/.local/share/terraform/plugins/registry.terraform.io/dmacvicar/libvirt/0.6.2/linux_amd64

In that directory you'll have just the executable file from the archive you downloaded, which should be called terraform-provider-libvirt.

(If you are using an s390x build then I imagine you'd need to use linux_s390x as that final directory instead, but I'm not 100% sure because Terraform doesn't officially support that architecture.)

With the local plugin package in place, the final step is to add a provider requirement to each of the modules in your configuration to state which provider you mean when you say "libvirt" elsewhere in the module:

terraform {
  required_providers {
    libvirt = {
      source  = "dmacvicar/libvirt"
      version = "0.6.2"
    }
  }
}

With all of that in place, terraform init should see that the configuration depends on dmacvicar/libvirt, and see that you have that provider installed on local disk, and so it should "install" the local package instead of trying to download one from the registry.

If you already have a terraform-provider-libvirt binary on your system that you were using with Terraform 0.12, it should still be compatible with Terraform 0.13 as long as you move it into the new-style directory structure I showed above, which allows Terraform to understand that you intend it as a local copy of registry.terraform.io/dmacvicar/libvirt.

I hope that helps!

[MalloZup]

@apparentlymart thx for the infos 👍

[MalloZup]

ALLL: so there is here the documentation for migrating to TF13. Examples are updated

https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/docs/migration-13.md

[limburgher]

Thank you! I may be doing it wrong, as I'm still having issues. I created ~/.local/share/terraform/plugins/registry.terraform.io/dmacvicar/libvirt/0.6.2/linux_amd64, and put 0.6.2 of terraform-provider-libvirt there. Then I added the above block to my providers.tf. terraform init then ends with:

• Finding latest version of -/libvirt...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider -/libvirt:
provider registry registry.terraform.io does not have a provider named
registry.terraform.io/-/libvirt

But it did symlink .terraform/plugins/registry.terraform.io/dmacvicar/libvirt/0.6.2/linux_amd64/ in my workspace to the above directory.

[invidian]

@limburgher if this is existing code/state, you may need to rename existing provider using terraform state replace-provider -- -/libvirt dmacvicar/libvirt command.

[limburgher]

Ah, thank you! Now init finishes, and I can refresh, etc.

[Balajigit2019]

Hi Team,

i our environment we are using centos 7 os , i tried the steps what you have told here. but i am still facing the issue while execute terraform init.

Initializing the backend...

Initializing provider plugins...

• Finding dmacvicar/libvirt versions matching "0.6.2"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
dmacvicar/libvirt: provider registry registry.terraform.io does not have a
provider named registry.terraform.io/dmacvicar/libvirt

Below steps i done.
1.installed the terraform.
2. download the provider plugin from(https://github.com/dmacvicar/terraform-provider-libvirt/releases/download/v0.6.2/terraform-provider-libvirt-0.6.2+git.1585292411.8cbe9ad0.Fedora_28.s390x.tar.gz)
3,created the directory~/.local/share/terraform/plugins
4. extract the terraform-provider-libvirt-0.6.2+git.1585292411.8cbe9ad0.Fedora_28.s390x.tar.gz and move the terraform-provider-libvirt into ~/.local/share/terraform/plugins
5. also created registry.terraform.io/dmacvicar/libvirt/0.6.2/linux_amd64
6. created provder.tf and the above mentioned syntax and placed that file in registry.terraform.io/dmacvicar/libvirt/0.6.2/linux_amd64

same issue please help me to solve this i want to do some automation using terraform in my enviroment

[Balajigit2019]

Validating provider dmacvicar/libvirt v0.6.2 failed: read
/root/.local/share/terraform/plugins/registry.terraform.io/dmacvicar/libvirt/0.6.2/linux_amd64/.terraform/plugins/registry.terraform.io/dmacvicar/libvirt/0.6.2/linux_amd64:
is a directory

[paultyng]

Its not ideal, but if you did want to publish this in the Registry still, you could fork to repos like dmacvicar/terraform-provider-libvirt-fedora, etc. and publish those flavors independently. But would probably require an annoying amount of tooling to automate. For documentation, we require the full source on the tag/release in GitHub, so you'd need to replicate the entire git history in each repo.

[Balajigit2019]

Thank you i solved the issue, could you please share any docs for provisioning the vm's using terraform in libvirt

[TheSeubert]

Ran into this myself and a reminder for all to follow the above and if you are on MacOS the locations are at:

~/.terraform.d/plugins/registry.terraform.io/

Specifically you also need to change the architecture folder and place the binary there:

~/.terraform.d/plugins/registry.terraform.io/dmacvicar/libvirt/0.6.2/darwin_amd64/

[MalloZup]

@dude051 and others feel free to improve the documentation here:
https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/docs/migration-13.md#migrate-provider-to-terraform-v13

I was thinking it was obvious that one need to change the arch accordingly but maybe is not that clear enough, we could state to change it. Remember that this doc is just a work-around, until we will pubish the RPM/DEB etc packages which will do this for the Linux distros. (see later)

@Balajigit2019 happy worked for you? what was the issue? for docs checkout:
https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/website/docs/r/domain.html.markdown

https://github.com/dmacvicar/terraform-provider-libvirt/tree/master/examples/v0.13

Also as Info:

we plan to ship the packages rpm and others in future accordingly RPMS/Debian to the new changes, so that when the user will install them, they location will be changed. All this manual work will be removed

I think personally as @apparentlymart stated, the libvirt provider per se is complex enough regarding the linking issue so isn't that easy as other clouds to publish to the registry. Since each LInux distro might have a different libvirt version etc etc.

Allthough withe the new packages we will publish, the user experience is just to put

  required_providers {
    libvirt = {
      source  = "dmacvicar/libvirt"
      version = "0.6.2"
    }
  }
}```

 on their hcl files.

[MalloZup]

I just changed the issue name, because the provider itself can be already used withing terraform V13. Allthough the issue is a research one , how the provider can be published there, so I will let this open.

[shubgwa]

I end up with the below error .I guess i am doing some mistake . Can someone please check and let me know what is the exact issue here .

No state file was found!

State management commands require a state file. Run this command
in a directory where Terraform has been run or use the -state flag
to point the command to a specific state location.

[TheSeubert]

@shubgwa your error does not seem related to this issue.

If this was a working Terraform deployment before and you are receiving this error, something is not correct with your configuration or your state file is no longer present. This would need to be sorted out and should be separate from this provider in regards to the issues that error would cause.

Verify that you have a terraform.tfstate in the directory you are running your commands from. Verify that the .terraform directory is present as well. If needed, locate and restore your state backup if it is present within the directory with the terraform.tfstate.backup file. Lastly, verify that you are not using a remote state in your Terraform configuration as that would require additional and different troubleshooting.

If you really believe this is due to the libvirt provider, I would open a new issue and provide the exact Terraform command(s) you are running.

[shubgwa]

Thank you Justin sir for your prompt response .

I followed the procedure given by Fabian Lee ( https://fabianlee.org/2020/02/22/kvm-terraform-and-cloud-init-to-create-local-kvm-resources/ ) but get stuck when i performed terraform init . And i was getting below error :

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider -/libvirt:
provider registry registry.terraform.io does not have a provider named
registry.terraform.io/-/libvirt

I saw the resolution given by Mateusz :

terraform state replace-provider -- -/libvirt dmacvicar/libvirt

But i am getting state file error .

[shubgwa]

I checked using locate command and try to find terraform.tfstate but it does not exist .

[invidian]

@shubgwa do you have following block configured in the directory where you run Terraform commands?

terraform {
  required_providers {
    libvirt = {
      source  = "dmacvicar/libvirt"
      version = "0.6.2"
    }
  }
}

Without that, Terraform won't be able to pull the provider.

No state file was found!

I guess this happens because you start in fresh Terraform environment, meaning there is no resources in state? If that's the case, you can skip terraform state replace-provider -- -/libvirt dmacvicar/libvirt part.

[shubgwa]

Yes i have configured this in my simple1.tf file . I have attached screenshot.

Yes i start in a fresh terraform environment .

But still how will i resolve the error i am getting .

infinity@infinity-ThinkPad-T440p:~/.terraform.d/plugins/terraform-libvirt-ubuntu-examples/simple$ terraform init

Initializing the backend...

Initializing provider plugins...

• Using previously-installed hashicorp/template v2.1.2
• Finding dmacvicar/libvirt versions matching "0.6.2"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
dmacvicar/libvirt: provider registry registry.terraform.io does not have a
provider named registry.terraform.io/dmacvicar/libvirt

[invidian]

@shubgwa and did you follow #747 (comment) to download the provider and to put it in the right place?

To be honest, I had really weird issues with locating locally built providers myself with Terraform 0.13. I usually put them in ~/.local/share/terraform/plugins/registry.terraform.io/ and in ~/.terraform.d/plugin-cache/registry.terraform.io/ with cache directory configured as:

$ cat ~/.terraformrc
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"

And it works.

[MalloZup]

thx @invidian, @dude051 and others for support to others.
I do think the issue from @shubgwa is due perhaps that he didn't put the provider to the right path.

To all: I do think here we are pretty much deviating to the main existence of the issue, which is:
how to bring the provider libvirt into the registry from terrraform, since it is a linked binary to each Linux libvirt
details: #747 (comment)

that is the problem of the issue.

If you have other issue with terraformV13 migration, either write on the chat (see readme.md) or open another issue.

Otherwise I do think we will deviate to much from the main point. ( @shubgwa if the comment of @invidian doesn't work, create an issue with all the steps in details you did, where is the provider etc, so we can follow-up etc).

thx guys! 🌻

[apparentlymart]

Hi again all!

I see the earlier comments about the possibility of distributing this provider as .rpm and .deb packages for specific distributions. That does seem like a plausible way to navigate the problem that each build of the provider must be tailored to the ABI of a specific distribution, and I expect it would also simplify installation because the packages could depend on whatever other distro packages contain a compatible libvirt and any other dynamic dependencies.

Possibly you saw this already and that's where the idea came from, but in case not I just wanted to say that on Linux Terraform implements the XDG base directory specification for locating provider plugins, so although in my earlier comment I showed installation into a user's home directory there are also two default locations that can be more appropriate for a system-wide package:

• /usr/share/terraform/plugins
• /usr/local/share/terraform/plugins

Other directories may be supported on certain distributions if they customize the XDG base directory environment variables.

By making an OS package install to one of these default search locations, terraform init should (as long as there's no special CLI configuration overriding the defaults) automatically find the provider and make use of it, as long as the source address in required_providers is correct.

If this is the strategy you intend to take then it may be a little weird to use registry.terraform.io/dmacvicar/ as the namespace prefix, because the provider will presumably not be available there. However, it won't cause any technical problem as long as there's never a real provider published at that address... the problem is only one of potential confusion for people who are bringing their experience with other Terraform providers.

[MalloZup]

Hi all and @apparentlymart,

thx for information and collaboration,

so let me summarize the status like following:

1)future 0.7.0 version and terraform registry
2) present: the current 0.6.3 version yesterday released behaviors

1) 0.7.0 Version (next)

So it is our current priority and for the next milestone release (0.7.0) to accommodate the situation within the terraform registry and bring the libvirt provider to the Terraform official registry, so that the normal user experience will be the same as the rest of provider like aws/azure/gcp etc so using use registry.terraform.io/dmacvicar/ as the namespace prefix

We are currently researching how to make this happen together @dmacvicar and we plan it for the 0.7.0.
( see outlook 0.6.3 notes )

2) 0.6.3 Version (current)

The current strategy for the 0.6.3 is that we use for the packages the suggested system wide paths.
So basically:

1. for the Distro Like installation, the user need only to update the package
2. for manual installation of binary, user can follow the documentation already written at https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/docs/migration-13.md.

The current 0.6.3 version is already packaged by openSUSE distros with RPM and binary are provided also for Fedora and Ubuntu.

• Archlinux has already rebuild the binary : see archlinux/svntogit-community@f4f19cc#diff-3e341d2d9c67be01819b25b25d5e53ea3cdf3a38d28846cda85a195eb9b7203a

I am for sure missing maybe other distros who have already other pkgs ❤️

[dmacvicar]

I have added this to the 0.7.0 release plan.

The main challenge (//cc @paultyng ) is that the registry does not allow for dependencies, external programs and cgo. Our provider is linked with libvirt, and that makes the provider binary be specific to different Linux distributions.

I plan to attack this by trying different approaches:

1.- Create a pure-go version of the provider. This has already been started in a local branch, using go-libvirt.
The main risks are:

• libvirt changing the protocol later
• The reliability of the library
• There are missing features, like polkit authentication

I am still hopeful this path may work, and I have a good strategy how to get there. We also have the testsuite to check everything is working. The library also has some history and looks maintained.

2.- Create a statically linked version of the provider, and distribute that in the registry, while still using packages like we do today.
I have only done minimal exploration here.

[gibsonje]

I'm able to get terraform init to work following the various comments in this thread. But I can't get a plan or apply to work.

I have ~/.local/share/terraform/plugins/registry.terraform.io/dmacvicar/libvirt/0.6.3/linux_amd64 with terraform-provider-libvirt in it.

  required_providers {
    libvirt = {
      source  = "dmacvicar/libvirt"
      version = "0.6.3"
    }
  }

Terraform init results in:

Initializing provider plugins...
- Finding dmacvicar/libvirt versions matching "0.6.3"...
- Installing dmacvicar/libvirt v0.6.3...
- Installed dmacvicar/libvirt v0.6.3 (unauthenticated)

Terraform has been successfully initialized!

But plan fails with:

Initializing provider plugins...
- Finding dmacvicar/libvirt versions matching "0.6.3"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
dmacvicar/libvirt: provider registry registry.terraform.io does not have a
provider named registry.terraform.io/dmacvicar/libvirt

Output of terraform --version

terraform --version
Terraform v0.13.4
+ provider registry.terraform.io/dmacvicar/libvirt v0.6.3

[MalloZup]

HI @gibsonje, please have a look on this doc: https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/docs/migration-13.md

if you still have issues feel free to ask in the chat: https://gitter.im/terraform-provider-libvirt/Lobby

Let's try to keep this issue focused on "moving" the provider to registry. TIA 🌹

[dmacvicar]

Hey @paultyng we got this done!

[limburgher]

Thank you so much!!!