feat: Replace the local-exec script with a http datasource for waiting cluster #1339

barryib · 2021-05-05T22:42:15Z

PR o'clock

Description

In this PR we drop the usage of local-exec. But we need hashicorp/terraform-provider-http#29.

We decided to fork the terraform-provider-http provider (the PR is almost a year) into the terraform-aws-modules organization for internal usage only. The provider is published at https://registry.terraform.io/providers/terraform-aws-modules/http/latest

Related also to #1253

Test

This is still a proof of concept. I tested it locally and so far so good. I've got this kind of output during my tests.

module.eks.aws_eks_cluster.this[0]: Creation complete after 10m20s [id=test-eks-kvnu9oYg]
module.eks.data.http.wait_for_cluster[0]: Reading...
module.eks.data.template_file.userdata[0]: Reading...
module.eks.data.template_file.userdata[1]: Reading...
module.eks.aws_iam_role.workers[0]: Creating...
module.eks.data.template_file.userdata[1]: Read complete after 0s [id=3ffad6f89915b07a9027e89ab9f36248b75b36c289057f594e79061ec808d3fb]
module.eks.data.template_file.userdata[0]: Read complete after 0s [id=3ffad6f89915b07a9027e89ab9f36248b75b36c289057f594e79061ec808d3fb]
module.eks.local_file.kubeconfig[0]: Creating...
module.eks.local_file.kubeconfig[0]: Creation complete after 0s [id=55314f623fda1f56705845677c8bbddf79c83f4e]
module.eks.aws_iam_role.workers[0]: Creation complete after 2s [id=test-eks-kvnu9oYg2021050708275921740000000c]
module.eks.aws_iam_role_policy_attachment.workers_AmazonEC2ContainerRegistryReadOnly[0]: Creating...
module.eks.aws_iam_role_policy_attachment.workers_AmazonEKS_CNI_Policy[0]: Creating...
module.eks.aws_iam_role_policy_attachment.workers_AmazonEKSWorkerNodePolicy[0]: Creating...
module.eks.aws_iam_instance_profile.workers[0]: Creating...
module.eks.aws_iam_instance_profile.workers[1]: Creating...
module.eks.aws_iam_role_policy_attachment.workers_AmazonEC2ContainerRegistryReadOnly[0]: Creation complete after 1s [id=test-eks-kvnu9oYg2021050708275921740000000c-2021050708280162970000000f]
module.eks.aws_iam_role_policy_attachment.workers_AmazonEKS_CNI_Policy[0]: Creation complete after 1s [id=test-eks-kvnu9oYg2021050708275921740000000c-20210507082801643400000010]
module.eks.aws_iam_role_policy_attachment.workers_AmazonEKSWorkerNodePolicy[0]: Creation complete after 1s [id=test-eks-kvnu9oYg2021050708275921740000000c-20210507082801988200000011]
module.eks.aws_iam_instance_profile.workers[1]: Creation complete after 2s [id=test-eks-kvnu9oYg2021050708280112110000000e]
module.eks.aws_iam_instance_profile.workers[0]: Creation complete after 2s [id=test-eks-kvnu9oYg2021050708280112100000000d]
module.eks.aws_launch_configuration.workers[0]: Creating...
module.eks.aws_launch_configuration.workers[1]: Creating...
module.eks.data.http.wait_for_cluster[0]: Still reading... [10s elapsed]
module.eks.aws_launch_configuration.workers[0]: Still creating... [10s elapsed]
module.eks.aws_launch_configuration.workers[1]: Still creating... [10s elapsed]
module.eks.aws_launch_configuration.workers[1]: Creation complete after 16s [id=test-eks-kvnu9oYg-worker-group-220210507082804636500000013]
module.eks.aws_launch_configuration.workers[0]: Creation complete after 16s [id=test-eks-kvnu9oYg-worker-group-120210507082804589800000012]
module.eks.random_pet.workers[0]: Creating...
module.eks.random_pet.workers[1]: Creating...
module.eks.random_pet.workers[1]: Creation complete after 0s [id=relaxed-woodcock]
module.eks.random_pet.workers[0]: Creation complete after 0s [id=exact-locust]
module.eks.aws_autoscaling_group.workers[1]: Creating...
module.eks.aws_autoscaling_group.workers[0]: Creating...
module.eks.data.http.wait_for_cluster[0]: Still reading... [20s elapsed]
module.eks.aws_autoscaling_group.workers[0]: Still creating... [10s elapsed]
module.eks.aws_autoscaling_group.workers[1]: Still creating... [10s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [30s elapsed]
module.eks.aws_autoscaling_group.workers[1]: Still creating... [20s elapsed]
module.eks.aws_autoscaling_group.workers[0]: Still creating... [20s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [40s elapsed]
module.eks.aws_autoscaling_group.workers[0]: Still creating... [30s elapsed]
module.eks.aws_autoscaling_group.workers[1]: Still creating... [30s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [50s elapsed]
module.eks.aws_autoscaling_group.workers[1]: Creation complete after 40s [id=test-eks-kvnu9oYg-worker-group-220210507082818853700000014]
module.eks.aws_autoscaling_group.workers[0]: Still creating... [40s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [1m0s elapsed]
module.eks.aws_autoscaling_group.workers[0]: Creation complete after 45s [id=test-eks-kvnu9oYg-worker-group-120210507082818856900000015]
module.eks.data.http.wait_for_cluster[0]: Still reading... [1m10s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [1m20s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [1m30s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [1m40s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [1m50s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [2m0s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [2m10s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [2m20s elapsed]
module.eks.data.http.wait_for_cluster[0]: Still reading... [2m30s elapsed]
module.eks.data.http.wait_for_cluster[0]: Read complete after 2m32s [id=https://xxxxxxx.xxx.xxx.eks.amazonaws.com/healthz]
data.aws_eks_cluster.cluster: Reading...

It took 2m32s for module.eks.data.http.wait_for_cluster[0] to clomplete.

Todo

Add terraform-aws-module/terraform-prodiver-http in examples
Document why we fork the http provider
Publish provider into terraform registry

barryib · 2021-05-07T08:21:34Z

@shoekstra @daroga0002 @jlpettersson can you please help me to test this ?

barryib · 2021-05-07T09:54:19Z

Maybe @xavfernandez could help to review and test this ?

barryib · 2021-05-07T11:10:37Z

README.md

@@ -221,6 +219,7 @@ MIT Licensed. See [LICENSE](https://github.com/terraform-aws-modules/terraform-a
 | [aws_iam_policy_document.workers_assume_role_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
 | [aws_iam_role.custom_cluster_iam_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_role) | data source |
 | [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
+| [http_http.wait_for_cluster](https://registry.terraform.io/providers/hashicorp/http/latest/docs/data-sources/http) | data source |


How to change this url ?

daroga0002 · 2021-05-10T10:47:25Z

Tested IRSA example with adding some explicit kubernetes resource creation:

...........
module.eks.aws_eks_cluster.this[0]: Still creating... [9m20s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [9m30s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [9m40s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [9m50s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [10m0s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [10m10s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [10m20s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [10m30s elapsed]
module.eks.aws_eks_cluster.this[0]: Still creating... [10m40s elapsed]
module.eks.aws_eks_cluster.this[0]: Creation complete after 10m44s [id=test-eks-irsa]
module.eks.data.http.wait_for_cluster[0]: Reading...
module.eks.data.template_file.userdata[0]: Reading...
module.eks.aws_iam_openid_connect_provider.oidc_provider[0]: Creating...
module.iam_assumable_role_admin.data.aws_iam_policy_document.assume_role_with_oidc[0]: Reading...
module.eks.aws_iam_role.workers[0]: Creating...
module.eks.data.template_file.userdata[0]: Read complete after 0s [id=5467189cf9996685ed6585631c69c9d829c9fb16b55b2e8f02752a661d8421bc]
module.iam_assumable_role_admin.data.aws_iam_policy_document.assume_role_with_oidc[0]: Read complete after 0s [id=2170324641]
module.eks.local_file.kubeconfig[0]: Creating...
module.iam_assumable_role_admin.aws_iam_role.this[0]: Creating...
.........

I spin cluster few times and each time created with success.

stevehipwell · 2021-05-11T07:06:52Z

@barryib would it be possible to expose the new wait_for_cluster variable as an output along the lines of control_plane_ready?

barryib · 2021-05-11T22:45:18Z

@barryib would it be possible to expose the new wait_for_cluster variable as an output along the lines of control_plane_ready?

@stevehipwell What do you mean by that ? Do you want me to add a new output for data.wait_for_cluster ?

stevehipwell · 2021-05-12T06:01:33Z

@barryib yes that's what I mean. Although in the context of the module it's the control plane not the cluster that's ready which is why I suggested the output name be changed.

versions.tf

barryib · 2021-05-17T08:04:45Z

@barryib yes that's what I mean. Although in the context of the module it's the control plane not the cluster that's ready which is why I suggested the output name be changed.

Normally when you use cluster_id or kubeconfig outputs, you don't need bother yourself with another control_plane_ready because those outputs depend already on data.http.wait_for_cluster.

stevehipwell · 2021-05-17T08:19:27Z

@barryib we don't use the kubeconfig output and prefer to use aws_eks_cluster_auth to get a token. We need to know when the control plane is ready as we need to remove config (due to aws/containers-roadmap#923 not being completed yet) before the workers are started. We were using curl and now have a beta version of our internal implementation using your http datasource. An output would be a tidier implementation.

stevehipwell · 2021-05-17T08:20:50Z

I guess we could use cluster_id to achieve this.

barryib · 2021-05-17T20:25:25Z

This now shipped in v16.0.0

nauxliu · 2021-05-18T14:35:56Z

@barryib the Forked http module does not have a darwin_arm64 release, I can't use this module on my M1 MacBook, I've created a PR to build releases with Go 1.16. Please take a look terraform-aws-modules/terraform-provider-http#5

…g cluster (terraform-aws-modules#1339) NOTES: Using the [terraform-aws-modules/http](https://registry.terraform.io/providers/terraform-aws-modules/http/latest) provider is a more platform agnostic way to wait for the cluster availability than using a local-exec. With this change we're able to provision EKS clusters and manage the `aws_auth` configmap while still using the `hashicorp/tfc-agent` docker image.

rkinganduril · 2021-05-21T18:49:55Z

I was using the wait for cluster cmd for Windows compatibility. Now that the variable is removed I cannot finish my terraform with the new http module. Upon removing the wait for cluster command in favor of the new method.

ivialex-mcd · 2021-05-21T19:41:59Z

I was using the wait for cluster cmd for Windows compatibility. Now that the variable is removed I cannot finish my terraform with the new http module. Upon removing the wait for cluster command in favor of the new method.

I have the same problem.

…g cluster (terraform-aws-modules#1339) NOTES: Using the [terraform-aws-modules/http](https://registry.terraform.io/providers/terraform-aws-modules/http/latest) provider is a more platform agnostic way to wait for the cluster availability than using a local-exec. With this change we're able to provision EKS clusters and manage the `aws_auth` configmap while still using the `hashicorp/tfc-agent` docker image.

github-actions · 2022-11-14T02:31:31Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

This was referenced May 5, 2021

improvement: Use time_sleep instead of local-exec #1253

Closed

Add server certificate validation with Root CA and insecure option hashicorp/terraform-provider-http#29

Closed

feat: Use http datasource instead of local-exec for waiting cluster

ddb43fa

barryib force-pushed the tba/http-datasource branch from 57e5cbd to ddb43fa Compare May 7, 2021 08:20

barryib marked this pull request as ready for review May 7, 2021 09:51

barryib commented May 7, 2021

View reviewed changes

kubeconfig should depends on cluster ready

4870784

BARRY Thierno Ibrahima (Canal Plus Prestataire) added 3 commits May 12, 2021 00:54

update docs

925a153

fixup

a125989

fixup

d9e3969

barryib requested a review from antonbabenko May 11, 2021 23:02

antonbabenko approved these changes May 12, 2021

View reviewed changes

versions.tf Outdated Show resolved Hide resolved

pin version

2773d1b

barryib changed the title ~~feat: Use http datasource instead of local-exec for waiting cluster~~ feat: Replace the local-exec script with a http datasource for waiting cluster May 17, 2021

barryib merged commit e3bf48b into terraform-aws-modules:master May 17, 2021

barryib deleted the tba/http-datasource branch May 17, 2021 08:11

barryib mentioned this pull request May 17, 2021

fix: Remove null provider from required providers #1357

Merged

2 tasks

barryib mentioned this pull request May 18, 2021

fix: Bump terraform-aws-modules/http provider version to support darwin arm64 release #1369

Merged

2 tasks

ivialex-mcd mentioned this pull request May 21, 2021

Error tls: crypto/x509: system root pool is not available on Windows #1384

Closed

barryib mentioned this pull request May 25, 2021

fix: Add back depends_on for data.wait_for_cluster #1389

Merged

1 task

mantoine96 mentioned this pull request Jun 29, 2021

r/aws_emrcontainers_virtual_cluster - new resource hashicorp/terraform-provider-aws#20003

Merged

tei-k mentioned this pull request Jul 13, 2021

Fix to use http resource instead of local-exec for eks health check scalar-labs/scalar-terraform#321

Merged

github-actions bot locked as resolved and limited conversation to collaborators Nov 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Replace the local-exec script with a http datasource for waiting cluster #1339

feat: Replace the local-exec script with a http datasource for waiting cluster #1339

barryib commented May 5, 2021 •

edited

barryib commented May 7, 2021 •

edited

barryib commented May 7, 2021

barryib May 7, 2021

daroga0002 commented May 10, 2021

stevehipwell commented May 11, 2021

barryib commented May 11, 2021

stevehipwell commented May 12, 2021

barryib commented May 17, 2021

stevehipwell commented May 17, 2021

stevehipwell commented May 17, 2021

barryib commented May 17, 2021

nauxliu commented May 18, 2021

rkinganduril commented May 21, 2021

ivialex-mcd commented May 21, 2021

github-actions bot commented Nov 14, 2022

feat: Replace the local-exec script with a http datasource for waiting cluster #1339

feat: Replace the local-exec script with a http datasource for waiting cluster #1339

Conversation

barryib commented May 5, 2021 • edited

PR o'clock

Description

Test

Todo

barryib commented May 7, 2021 • edited

barryib commented May 7, 2021

barryib May 7, 2021

Choose a reason for hiding this comment

daroga0002 commented May 10, 2021

stevehipwell commented May 11, 2021

barryib commented May 11, 2021

stevehipwell commented May 12, 2021

barryib commented May 17, 2021

stevehipwell commented May 17, 2021

stevehipwell commented May 17, 2021

barryib commented May 17, 2021

nauxliu commented May 18, 2021

rkinganduril commented May 21, 2021

ivialex-mcd commented May 21, 2021

github-actions bot commented Nov 14, 2022

barryib commented May 5, 2021 •

edited

barryib commented May 7, 2021 •

edited