[Orc8r] Error on update when running terraform init --upgrade #5435

BioZrod · 2021-03-11T14:09:43Z

BioZrod
Mar 11, 2021

Hello I am trying to update my orchestrator from version 1.2 to 1.3 I have updated the docker hub registry and and helm repository successfully. After I updated my terraform file following upgrade steps and then I ran terraform init --upgrade and got the following error:

biozrod@Rodrigos-MacBook-Pro eks % terraform init --upgrade
Upgrading modules...
Downloading github.com/magma/magma?ref=v1.3 for orc8r...
- orc8r in .terraform/modules/orc8r/orc8r/cloud/deploy/terraform/orc8r-aws
Downloading github.com/magma/magma?ref=v1.3 for orc8r-app...
- orc8r-app in .terraform/modules/orc8r-app/orc8r/cloud/deploy/terraform/orc8r-helm-aws
Error: Error accessing remote module registry
Failed to retrieve available versions for module "eks"
(.terraform/modules/orc8r/orc8r/cloud/deploy/terraform/orc8r-aws/eks.tf:27)
from registry.terraform.io: Failed to request discovery document: Get
"https://registry.terraform.io/.well-known/terraform.json": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while awaiting
headers).
Error: Error accessing remote module registry
Failed to retrieve available versions for module "vpc"
(.terraform/modules/orc8r/orc8r/cloud/deploy/terraform/orc8r-aws/vpc.tf:14)
from registry.terraform.io: Failed to request discovery document: Get
"https://registry.terraform.io/.well-known/terraform.json": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while awaiting
headers).
Error: Error accessing remote module registry
Failed to retrieve available versions for module "eks"
(.terraform/modules/orc8r/orc8r/cloud/deploy/terraform/orc8r-aws/eks.tf:27)
from registry.terraform.io: Failed to request discovery document: Get
"https://registry.terraform.io/.well-known/terraform.json": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while awaiting
headers).
Error: Error accessing remote module registry
Failed to retrieve available versions for module "eks"
(.terraform/modules/orc8r/orc8r/cloud/deploy/terraform/orc8r-aws/eks.tf:27)
from registry.terraform.io: Failed to request discovery document: Get
"https://registry.terraform.io/.well-known/terraform.json": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while awaiting
headers).
Error: Error accessing remote module registry
Failed to retrieve available versions for module "vpc"
(.terraform/modules/orc8r/orc8r/cloud/deploy/terraform/orc8r-aws/vpc.tf:14)
from registry.terraform.io: Failed to request discovery document: Get
"https://registry.terraform.io/.well-known/terraform.json": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while awaiting
headers).
Error: Error accessing remote module registry
Failed to retrieve available versions for module "eks"
(.terraform/modules/orc8r/orc8r/cloud/deploy/terraform/orc8r-aws/eks.tf:27)
from registry.terraform.io: Failed to request discovery document: Get
"https://registry.terraform.io/.well-known/terraform.json": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while awaiting
headers).
Error: Error accessing remote module registry
Failed to retrieve available versions for module "vpc"
(.terraform/modules/orc8r/orc8r/cloud/deploy/terraform/orc8r-aws/vpc.tf:14)
from registry.terraform.io: Failed to request discovery document: Get
"https://registry.terraform.io/.well-known/terraform.json": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while awaiting
headers).

Answered by karthiksubraveti

Mar 17, 2021

@BioZrod Can you try to set the cluster version to latest cluster version and see if the issue gets fixed ? If not can you explicitly set the values of
https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/8.2.0?tab=inputs.
Meanwhile can you also check what's the ID set for this value in terraform tfstate ?

variable "worker_ami_owner_id" {
description = "The ID of the owner for the AMI to use for the AWS EKS workers. Valid values are an AWS account ID, 'self' (the current account), or an AWS owner alias (e.g. 'amazon', 'aws-marketplace', 'microsoft')."
type = string
default = "961992271922" // The ID of the owner of the official AWS EKS AMIs.
}

variable "worker_ami_owner_id_…

View full answer

hcgatewood · 2021-03-11T20:36:26Z

hcgatewood
Mar 11, 2021

Hi @BioZrod, can you include the output from terraform version? We haven't noticed this issue before, it's possible the resolution is upgrading your Terraform version. It's also possible the connection to the public registry may be getting blocked by firewall or VPN misconfiguration.

*If you are running an older version of Terraform, tfenv is an easy way to upgrade

If these don't fix, we can move forward with helping debug

10 replies

sudhikan Mar 12, 2021

@BioZrod - any possibility of a DNS issue on your machine? I am thinking something similar to the comments mentioned here. Can you just put https://registry.terraform.io/.well-known/terraform.json in the browser and see if it gives you a valid response?

BioZrod Mar 12, 2021
Author

@sudhikan when opening this link into my browser I get a blank page with the following message:

{"modules.v1":"/v1/modules/","providers.v1":"/v1/providers/"}

Is that the expected output?

sudhikan Mar 12, 2021

looks fine... just going on a hunch, can you run sudo terraform init on your deployment? Just reading around the internet on how people have gone around this problem... running the init with TF_LOG will help quite a bit as @hcgatewood mentioned above.

BioZrod Mar 12, 2021
Author

@sudhikan It seems your hunch was right as I replaced my ISP default DNS with google one and terraform init --upgrade ran successfully with the following output:

biozrod@Rodrigos-MacBook-Pro eks % terraform init --upgrade
Upgrading modules...
Downloading github.com/magma/magma?ref=v1.3 for orc8r...
- orc8r in .terraform/modules/orc8r/orc8r/cloud/deploy/terraform/orc8r-aws
Downloading terraform-aws-modules/eks/aws 8.2.0 for orc8r.eks...
- orc8r.eks in .terraform/modules/orc8r.eks
- orc8r.eks.node_groups in .terraform/modules/orc8r.eks/modules/node_groups
Downloading terraform-aws-modules/vpc/aws 2.17.0 for orc8r.vpc...
- orc8r.vpc in .terraform/modules/orc8r.vpc
Downloading github.com/magma/magma?ref=v1.3 for orc8r-app...
- orc8r-app in .terraform/modules/orc8r-app/orc8r/cloud/deploy/terraform/orc8r-helm-aws
Initializing the backend...
Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Finding hashicorp/random versions matching ">= 2.1.0, ~> 2.1"...
- Finding hashicorp/local versions matching ">= 1.2.0"...
- Finding hashicorp/kubernetes versions matching ">= 1.6.2, ~> 1.10.0"...
- Finding hashicorp/helm versions matching "~> 1.0"...
- Finding hashicorp/null versions matching ">= 2.1.0"...
- Finding hashicorp/tls versions matching "~> 2.1"...
- Finding hashicorp/template versions matching "~> 2.0, >= 2.1.0"...
- Finding hashicorp/aws versions matching ">= 2.6.0, >= 2.44.0"...
- Installing hashicorp/kubernetes v1.10.0...
- Installed hashicorp/kubernetes v1.10.0 (signed by HashiCorp)
- Installing hashicorp/helm v1.3.2...
- Installed hashicorp/helm v1.3.2 (signed by HashiCorp)
- Installing hashicorp/null v3.1.0...
- Installed hashicorp/null v3.1.0 (signed by HashiCorp)
- Installing hashicorp/tls v2.2.0...
- Installed hashicorp/tls v2.2.0 (signed by HashiCorp)
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)
- Installing hashicorp/aws v3.31.0...
- Installed hashicorp/aws v3.31.0 (signed by HashiCorp)
- Installing hashicorp/random v2.3.1...
- Installed hashicorp/random v2.3.1 (signed by HashiCorp)
- Installing hashicorp/local v2.1.0...
- Installed hashicorp/local v2.1.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Warning: Version constraints inside provider configuration blocks are deprecated
  on main.tf line 16, in provider "aws":
  16:   version = ">= 2.6.0"
Terraform 0.13 and earlier allowed provider version constraints inside the
provider configuration block, but that is now deprecated and will be removed
in a future version of Terraform. To silence this warning, move the provider
version constraint into the required_providers block.
(and 10 more similar warnings elsewhere)
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary

I was wondering how I can fix those warnings as it says it will be deprecated in the future.

Also I ran into a new issue when running terraform plan, it outputs a lot of refreshing state... and at the end it gives me this warnings and error:

Warning: Deprecated Resource
The null_data_source was historically used to construct intermediate values to
re-use elsewhere in configuration, the same can now be achieved using locals
Warning: Version constraints inside provider configuration blocks are deprecated
  on main.tf line 16, in provider "aws":
  16:   version = ">= 2.6.0"
Terraform 0.13 and earlier allowed provider version constraints inside the
provider configuration block, but that is now deprecated and will be removed
in a future version of Terraform. To silence this warning, move the provider
version constraint into the required_providers block.
(and 10 more similar warnings elsewhere)
Error: Your query returned no results. Please change your search criteria and try again.
  on .terraform/modules/orc8r.eks/data.tf line 36, in data "aws_ami" "eks_worker_windows":
  36: data "aws_ami" "eks_worker_windows" {

Please let me know if you know how I can solve that issue and if I need to open another discussion for this specific error so it has it's own thread in case someone else in the future runs into it. Also if you still want me to run terraform init with TF_LOG let me know.

karthiksubraveti Mar 12, 2021

The warnings on null_data_source is fine and version constraints inside provider configuration blocks should be harmless. We should clean this up going forward, but this shouldn't affect your deployment. The newer error might be a bug in eks module. terraform-aws-modules/terraform-aws-eks#824. Still looking and trying to understand this error. Will update if i find something here.

karthiksubraveti · 2021-03-16T13:03:52Z

karthiksubraveti
Mar 16, 2021

@BioZrod Just checking if you were still blocked here ?

4 replies

BioZrod Mar 16, 2021
Author

Yes I could not figure out what is the issue, I copy the data.tf file that has the issue if that helps you see if there is any problem there.

locals {
  worker_ami_name_filter = var.worker_ami_name_filter != "" ? var.worker_ami_name_filter : "amazon-eks-node-${var.cluster_version}-v*"

  # Windows nodes are available from k8s 1.14. If cluster version is less than 1.14, fix ami filter to some constant to not fail on 'terraform plan'.
  worker_ami_name_filter_windows = (var.worker_ami_name_filter_windows != "" ?
    var.worker_ami_name_filter_windows : "Windows_Server-2019-English-Core-EKS_Optimized-${tonumber(var.cluster_version) >= 1.14 ? var.cluster_version : 1.14}-*"
  )
}

data "aws_iam_policy_document" "workers_assume_role_policy" {
  statement {
    sid = "EKSWorkerAssumeRole"

    actions = [
      "sts:AssumeRole",
    ]

    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

data "aws_ami" "eks_worker" {
  filter {
    name   = "name"
    values = [local.worker_ami_name_filter]
  }

  most_recent = true

  owners = [var.worker_ami_owner_id]
}

data "aws_ami" "eks_worker_windows" {
  filter {
    name   = "name"
    values = [local.worker_ami_name_filter_windows]
  }

  filter {
    name   = "platform"
    values = ["windows"]
  }

  most_recent = true

  owners = [var.worker_ami_owner_id_windows]
}


data "aws_iam_policy_document" "cluster_assume_role_policy" {
  statement {
    sid = "EKSClusterAssumeRole"

    actions = [
      "sts:AssumeRole",
    ]

    principals {
      type        = "Service"
      identifiers = ["eks.amazonaws.com"]
    }
  }
}

data "template_file" "kubeconfig" {
  count    = var.create_eks ? 1 : 0
  template = file("${path.module}/templates/kubeconfig.tpl")

  vars = {
    kubeconfig_name           = local.kubeconfig_name
    endpoint                  = aws_eks_cluster.this[0].endpoint
    cluster_auth_base64       = aws_eks_cluster.this[0].certificate_authority[0].data
    aws_authenticator_command = var.kubeconfig_aws_authenticator_command
    aws_authenticator_command_args = length(var.kubeconfig_aws_authenticator_command_args) > 0 ? "        - ${join(
      "\n        - ",
      var.kubeconfig_aws_authenticator_command_args,
      )}" : "        - ${join(
      "\n        - ",
      formatlist("\"%s\"", ["token", "-i", aws_eks_cluster.this[0].name]),
    )}"
    aws_authenticator_additional_args = length(var.kubeconfig_aws_authenticator_additional_args) > 0 ? "        - ${join(
      "\n        - ",
      var.kubeconfig_aws_authenticator_additional_args,
    )}" : ""
    aws_authenticator_env_variables = length(var.kubeconfig_aws_authenticator_env_variables) > 0 ? "      env:\n${join(
      "\n",
      data.template_file.aws_authenticator_env_variables.*.rendered,
    )}" : ""
  }
}

data "template_file" "aws_authenticator_env_variables" {
  count = length(var.kubeconfig_aws_authenticator_env_variables)

  template = <<EOF
        - name: $${key}
          value: $${value}
EOF


  vars = {
    value = values(var.kubeconfig_aws_authenticator_env_variables)[count.index]
    key   = keys(var.kubeconfig_aws_authenticator_env_variables)[count.index]
  }
}

data "template_file" "userdata" {
  count = var.create_eks ? local.worker_group_count : 0
  template = lookup(
    var.worker_groups[count.index],
    "userdata_template_file",
    file(
      lookup(var.worker_groups[count.index], "platform", local.workers_group_defaults["platform"]) == "windows"
      ? "${path.module}/templates/userdata_windows.tpl"
      : "${path.module}/templates/userdata.sh.tpl"
    )
  )

  vars = merge({
    platform            = lookup(var.worker_groups[count.index], "platform", local.workers_group_defaults["platform"])
    cluster_name        = aws_eks_cluster.this[0].name
    endpoint            = aws_eks_cluster.this[0].endpoint
    cluster_auth_base64 = aws_eks_cluster.this[0].certificate_authority[0].data
    pre_userdata = lookup(
      var.worker_groups[count.index],
      "pre_userdata",
      local.workers_group_defaults["pre_userdata"],
    )
    additional_userdata = lookup(
      var.worker_groups[count.index],
      "additional_userdata",
      local.workers_group_defaults["additional_userdata"],
    )
    bootstrap_extra_args = lookup(
      var.worker_groups[count.index],
      "bootstrap_extra_args",
      local.workers_group_defaults["bootstrap_extra_args"],
    )
    kubelet_extra_args = lookup(
      var.worker_groups[count.index],
      "kubelet_extra_args",
      local.workers_group_defaults["kubelet_extra_args"],
    )
    },
    lookup(
      var.worker_groups[count.index],
      "userdata_template_extra_args",
      local.workers_group_defaults["userdata_template_extra_args"]
    )
  )
}

data "template_file" "launch_template_userdata" {
  count = var.create_eks ? local.worker_group_launch_template_count : 0
  template = lookup(
    var.worker_groups_launch_template[count.index],
    "userdata_template_file",
    file(
      lookup(var.worker_groups_launch_template[count.index], "platform", local.workers_group_defaults["platform"]) == "windows"
      ? "${path.module}/templates/userdata_windows.tpl"
      : "${path.module}/templates/userdata.sh.tpl"
    )
  )

  vars = merge({
    platform            = lookup(var.worker_groups_launch_template[count.index], "platform", local.workers_group_defaults["platform"])
    cluster_name        = aws_eks_cluster.this[0].name
    endpoint            = aws_eks_cluster.this[0].endpoint
    cluster_auth_base64 = aws_eks_cluster.this[0].certificate_authority[0].data
    pre_userdata = lookup(
      var.worker_groups_launch_template[count.index],
      "pre_userdata",
      local.workers_group_defaults["pre_userdata"],
    )
    additional_userdata = lookup(
      var.worker_groups_launch_template[count.index],
      "additional_userdata",
      local.workers_group_defaults["additional_userdata"],
    )
    bootstrap_extra_args = lookup(
      var.worker_groups_launch_template[count.index],
      "bootstrap_extra_args",
      local.workers_group_defaults["bootstrap_extra_args"],
    )
    kubelet_extra_args = lookup(
      var.worker_groups_launch_template[count.index],
      "kubelet_extra_args",
      local.workers_group_defaults["kubelet_extra_args"],
    )
    },
    lookup(
      var.worker_groups_launch_template[count.index],
      "userdata_template_extra_args",
      local.workers_group_defaults["userdata_template_extra_args"]
    )
  )
}

data "aws_iam_role" "custom_cluster_iam_role" {
  count = var.manage_cluster_iam_resources ? 0 : 1
  name  = var.cluster_iam_role_name
}

data "aws_iam_instance_profile" "custom_worker_group_iam_instance_profile" {
  count = var.manage_worker_iam_resources ? 0 : local.worker_group_count
  name = lookup(
    var.worker_groups[count.index],
    "iam_instance_profile_name",
    local.workers_group_defaults["iam_instance_profile_name"],
  )
}

data "aws_iam_instance_profile" "custom_worker_group_launch_template_iam_instance_profile" {
  count = var.manage_worker_iam_resources ? 0 : local.worker_group_launch_template_count
  name = lookup(
    var.worker_groups_launch_template[count.index],
    "iam_instance_profile_name",
    local.workers_group_defaults["iam_instance_profile_name"],
  )
}

sudhikan Mar 16, 2021

where is this data.tf coming from? Also, why windows?

BioZrod Mar 16, 2021
Author

from .terraform/modules/orc8r.eks/data.tf I am using a MAC and I am trying to terraform my deployment on AWS that is why it is strange for me that it shows windows.

EKS cluster nodes are linux AMI's

BioZrod Mar 16, 2021
Author

The error comes from running terraform plan, it outputs a lot of refreshing state... and at the end it gives me this warnings and error:

Warning: Deprecated Resource
The null_data_source was historically used to construct intermediate values to
re-use elsewhere in configuration, the same can now be achieved using locals
Warning: Version constraints inside provider configuration blocks are deprecated
  on main.tf line 16, in provider "aws":
  16:   version = ">= 2.6.0"
Terraform 0.13 and earlier allowed provider version constraints inside the
provider configuration block, but that is now deprecated and will be removed
in a future version of Terraform. To silence this warning, move the provider
version constraint into the required_providers block.
(and 10 more similar warnings elsewhere)
Error: Your query returned no results. Please change your search criteria and try again.
  on .terraform/modules/orc8r.eks/data.tf line 36, in data "aws_ami" "eks_worker_windows":
  36: data "aws_ami" "eks_worker_windows" {

karthiksubraveti · 2021-03-17T03:03:12Z

karthiksubraveti
Mar 17, 2021

@BioZrod Can you try to set the cluster version to latest cluster version and see if the issue gets fixed ? If not can you explicitly set the values of
https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/8.2.0?tab=inputs.
Meanwhile can you also check what's the ID set for this value in terraform tfstate ?

variable "worker_ami_owner_id" {
description = "The ID of the owner for the AMI to use for the AWS EKS workers. Valid values are an AWS account ID, 'self' (the current account), or an AWS owner alias (e.g. 'amazon', 'aws-marketplace', 'microsoft')."
type = string
default = "961992271922" // The ID of the owner of the official AWS EKS AMIs.
}

variable "worker_ami_owner_id_windows" {
description = "The ID of the owner for the AMI to use for the AWS EKS Windows workers. Valid values are an AWS account ID, 'self' (the current account), or an AWS owner alias (e.g. 'amazon', 'aws-marketplace', 'microsoft')."
type = string
default = "016951021795" // The ID of the owner of the official AWS EKS Windows AMIs.
}

5 replies

BioZrod Mar 17, 2021
Author

I just managed to run terraform apply successfully by changing

locals {
  worker_ami_name_filter = var.worker_ami_name_filter != "" ? var.worker_ami_name_filter : "amazon-eks-node-${var.cluster_version}-v*"

  # Windows nodes are available from k8s 1.14. If cluster version is less than 1.14, fix ami filter to some constant to not fail on 'terraform plan'.
  worker_ami_name_filter_windows =  "Windows_Server-2019-English-Core-EKS_Optimized-1.16-*"
  )
}

being 1.16 my current cluster version.

I am unsure what caused the issue and why this works.

karthiksubraveti Mar 17, 2021

What did you change here ? Did you set cluster_version to 1.16 ?

BioZrod Mar 17, 2021
Author

Original value was:

locals {
  worker_ami_name_filter = var.worker_ami_name_filter != "" ? var.worker_ami_name_filter : "amazon-eks-node-${var.cluster_version}-v*"

  # Windows nodes are available from k8s 1.14. If cluster version is less than 1.14, fix ami filter to some constant to not fail on 'terraform plan'.
  worker_ami_name_filter_windows = (var.worker_ami_name_filter_windows != "" ?
    var.worker_ami_name_filter_windows : "Windows_Server-2019-English-Core-EKS_Optimized-${tonumber(var.cluster_version) >= 1.14 ? var.cluster_version : 1.14}-*"
  )
}

It says to change it to a constant if cluster version is less than 1.14 which is not my case but after reading the links you posted last week I couldn't figure out anything else, so I changed it to a random constant, this did not work as it still gave the same issue.

Playing around with this value my last attempt was changing it to and it worked:

worker_ami_name_filter_windows = "Windows_Server-2019-English-Core-EKS_Optimized-1.16-*"

Important things to note on my setup is that cluster is 1.16 and nodes are still on 1.14 because I haven't figure it out how to update the nodes. Also once this worked when running terraform apply I did have to change some other files because I got the error that it is not possible to upgrade from cluster version 1.16 to 1.14 and also not possible to upgrade from postgres 9.6.20 to 9.6.18 which is similar to ##3714

After fixing those upgrade issues I successfully went from orchestrator 1.12 to 1.13.

Still unsure why or how this windows_ami value worked but hopefully is someone else gets this error can fix it like this.

karthiksubraveti Mar 17, 2021

What's the cluster_version on the EKS cluster though ? Most likely the older AMI wasn't available.
https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest

BioZrod Mar 17, 2021
Author

Cluster version is 1.16.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Orc8r] Error on update when running terraform init --upgrade #5435

{{title}}

Replies: 3 comments 19 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

[Orc8r] Error on update when running terraform init --upgrade #5435

Replies: 3 comments · 19 replies

BioZrod Mar 12, 2021 Author

BioZrod Mar 12, 2021 Author

BioZrod Mar 16, 2021 Author

BioZrod Mar 16, 2021 Author

BioZrod Mar 16, 2021 Author

BioZrod Mar 17, 2021 Author

BioZrod Mar 17, 2021 Author

BioZrod Mar 17, 2021 Author

Replies: 3 comments 19 replies

BioZrod Mar 12, 2021
Author

BioZrod Mar 12, 2021
Author

BioZrod Mar 16, 2021
Author

BioZrod Mar 16, 2021
Author

BioZrod Mar 16, 2021
Author

BioZrod Mar 17, 2021
Author

BioZrod Mar 17, 2021
Author

BioZrod Mar 17, 2021
Author