Error: Failed to query available provider packages

[lVlario0O0o]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

cdktf & Language Versions

$ cdktf --version
0.0.19-pre.153

$ terraform -version
Terraform v0.14.3

$ node --version
v15.4.0

$ yarn --version
1.22.5

$ cdktf init --template=typescript --local

Affected Resource(s)

Not able to deploy with the following command:

$ cdktf deploy

Debug Output

https://gist.github.com/lVlario0O0o/b0d528d646319052326ada38cb510f6d

Expected Behavior

https://gist.github.com/lVlario0O0o/a494f055443cfef0d042e1b6f23f5071

Actual Behavior

see the Debug Output

Steps to Reproduce

Follow the tutorial for the terraform cdk at the following link

Important Factoids

nothing atypical about my account

References

No other issues that i did not found.

• #0000

[lVlario0O0o]

I've used the 'main.ts' from the typescript docker example in this repo. It's probably due to the fact that there's changes in the way that terraform 0.12, 0.13 and up deal differently with providers?

[lVlario0O0o]

With this version of the 'main.ts' file, the deploy command works fine. I've also respond to the feedback comment on the hashicorp web site that it should be in sync with the actual repo!

[lVlario0O0o]

When I'm removing the line 26

// new DockerProvider(this, 'provider', {})

in this example it reproduce the error.

[jsteinich]

I'm not certain if implicit providers used to work and they broke or if there is just an error with that tutorial. @skorfmann Do you know the intended functionality?

[lVlario0O0o]

With the terraform v0.12, the terraform get and the terraform deploy commands works find like in the tutorial.
Proof : look at the User-Agent at the last line.

I guess there are mistake in the tutorial since it is written that it works with the terraform version 0.12 and UP.

In the generated file, cdk.out/cdk.tf.json, by the cdk does not include the required provider

We may follow these steps from a repo that I used to test the above.

[jsteinich]

@lVlario0O0o thanks for clarifying.
This issue could either be taken as a bug report about that tutorial or as a feature request to make that work with newer versions of Terraform.

[lukep-coxauto]

EDIT: I realized that the emit problem I was having was related to improper initialization of the provider I was using, and not to the code generation, sorry for the pings. You can reference a provider, generate the code, and then use objects within the provider and everything will build and synth, but will not deploy because the provider initialization (new someProvider()) is not included in the context. This was my error, but if this could somehow be shifted to the cdktf portion to error during build, it would be better. Disregard the below.

PR #285 appears to be structured to fix this issue, and implies that this docker provider with the hierarchical path should function properly (specifically this commit). Here is the file that was introduced to split the provider name and emit these hierarchical providers.

@skorfmann since you worked on this PR, perhaps you can weigh in if we are not using this correctly or if this is some kind of regression? Will help however I can, as this is a blocker for our cdktf POC (we can work around it by adding the required providers manually, but will need this to function properly for the POC so we do not have to edit the cdk.tf.json file as it is generated code).

[skorfmann]

@lVlario0O0o Thanks for the detailed bug report! As @jsteinich said, we'll fix that in the tutorial.

@jsteinich

I'm not certain if implicit providers used to work and they broke or if there is just an error with that tutorial. @skorfmann Do you know the intended functionality?

Implicit providers used to work and should still work, when they are under the hashicorp namespace as far as I remember.

@lukep-coxauto

This was my error, but if this could somehow be shifted to the cdktf portion to error during build, it would be better.

Could you elaborate on what you mean? Other than that, everything works for you - is that correct?

[lukep-coxauto]

Yes, I'm good now for my issue, but I'll try to explain what I mean in more detail and why I was confused.

// cdktf.json
{
  "language": "typescript",
  "app": "npm run --silent compile && node main.js",
  "terraformProviders": ["terraform-providers/docker@~> 2.0"],
  "context": {
    "excludeStackIdFromLogicalIds": "true",
    "allowSepCharsInLogicalIds": "true"
  }
}

// main.ts
import { Construct } from "constructs";
import { App, TerraformStack } from "cdktf";
import { Container } from "./.gen/providers/docker";

class MyStack extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name);
    new Container(this, "my-container", {
      image: "ubuntu",
      name: "ubuntu-test",
    });
  }
}

const app = new App();
new MyStack(app, "cdktf-test-docker");

app.synth();

This has no build issues and runs through synth and creates a full output file set, but it is not terraform applyable or cdktf deployable because Container has a dependency on a Provider being defined before its called, but there is no enforcement of that, which results in the runtime error when trying to run cdktf deploy

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
hashicorp/docker: provider registry registry.terraform.io does not have a
provider named registry.terraform.io/hashicorp/docker

This makes one assume that the problem is related to namespacing of providers but it's really not. I assumed that this was an issue with the required providers property not being included in the output for third party providers since it happened to the docker provider as well as the one I was originally testing with, but the issue was actually that I didn't create an instance of the provider within the context.

I feel like this file should not have been "synthed" because it is not runnable. In my mind, cdktf synth should be throwing an error when new Container() is called and the context does not have a corresponding provider already defined. The error doesn't occur until Terraform is trying to resolve the providers, which results in the unclear error message from Terraform which was really an issue in the synth step. What do you think? Is there a use case where this should be allowed?

Here is the cdk.tf.json output file (notice it doesn't have required providers at all. Originally I was using aws and docker, but I only had the aws provider defined correctly, so I thought the issue was because of the third party provider, rather than the fact that I didn't use the providers properly).

{
  "//": {
    "metadata": {
      "version": "0.0.19-pre.155",
      "stackName": "cdktf-test-docker"
    }
  },
  "resource": {
    "docker_container": {
      "my-container": {
        "image": "ubuntu",
        "name": "ubuntu-test",
        "//": {
          "metadata": {
            "path": "cdktf-test-docker/my-container",
            "uniqueId": "my-container",
            "stackTrace": [
              "...omitted..."
            ]
          }
        }
      }
    }
  }
}

[skorfmann]

I feel like this file should not have been "synthed" because it is not runnable. In my mind, cdktf synth should be throwing an error when new Container() is called and the context does not have a corresponding provider already defined.

Ahh, I see. So far synth isn't doing any validation whatsoever, it's all delegated to the Terraform CLI. However, I can see the value of doing a few basic checks like this.

@lukep-coxauto would you mind creating a new issue for this?

[skorfmann]

I'm not certain if implicit providers used to work and they broke or if there is just an error with that tutorial. @skorfmann Do you know the intended functionality?

Implicit providers used to work and should still work, when they are under the hashicorp namespace as far as I remember.

I looked it up again - Providers need to explicitly define their source location since Terraform 0.13. We do this by explicitly creating a provider object in the stack. This wasn't necessary before, but it was totally valid to do so anyway.

Submitted a PR to the Learn guide. Thanks again for the bug report and the follow up issue 👍

[github-actions]

I'm going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.