Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform: update GKE cluster version, use locals and lookup to set default values #1482

Merged

Conversation

aLekSer
Copy link
Collaborator

@aLekSer aLekSer commented Apr 20, 2020

Add new variable to select Kubernetes major and minor version.
google_container_engine_versions data source added, which allows to select proper node and master versions which applicable for every zone in the region if it is used with a regional cluster.

What type of PR is this?
/kind feature

What this PR does / Why we need it:
For #1478 .

It would allow proper version selection by specifying Kubernetes version prefix and cluster location. I.e. for regional clusters, us-west2 could be used, for zonal clusters us-west2-c

Which issue(s) this PR fixes:

Special notes for your reviewer:
Useful documents:
https://www.terraform.io/docs/providers/google/d/google_container_engine_versions.html#latest_master_version
https://www.terraform.io/docs/providers/google/r/container_cluster.html#min_master_version
https://www.terraform.io/docs/providers/google/r/container_cluster.html#node_version

@aLekSer aLekSer added the area/operations Installation, updating, metrics etc label Apr 20, 2020
@aLekSer aLekSer force-pushed the terraform/update-gke-version branch from 1d60c07 to 93c6e50 Compare April 20, 2020 11:37
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: d39a606a-1028-41d5-b12a-f8bf0912eaaa

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1482/head:pr_1482 && git checkout pr_1482
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-1d60c07

@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: b282d962-994a-42db-9d94-ec37bf0197c8

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@aLekSer
Copy link
Collaborator Author

aLekSer commented Apr 20, 2020

An error in C# SDK build step:

includes/sdk.mk:107: recipe for target 'build-build-sdk-image' failed
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
E: Failed to fetch https://packages.microsoft.com/ubuntu/18.04/prod/pool/main/d/dotnet-sdk-2.2/dotnet-sdk-2.2.402-x64.deb  GnuTLS recv error (-54): Error in the pull function.
Fetched 44.1 MB in 3s (12.2 MB/s)
  GnuTLS recv error (-54): Error in the pull function.
Err:9 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 dotnet-sdk-2.2 amd64 2.2.402-1

@aLekSer aLekSer force-pushed the terraform/update-gke-version branch from 93c6e50 to 6eccce5 Compare April 20, 2020 11:50
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 17f54d6b-cd4c-4d11-8b8d-4d05f5e2727f

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1482/head:pr_1482 && git checkout pr_1482
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-6eccce5

build/includes/terraform.mk Show resolved Hide resolved
@@ -97,17 +97,21 @@ variable "feature_gates" {
default = ""
}

variable "kubernetes_version" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming we're choosing 1.15 to start work on #1478 ?

Copy link
Collaborator Author

@aLekSer aLekSer Apr 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable was used here so that we can make changes to this file more rare. And imagine the situation when between Agones releases Kubernetes version would be deprecated but with Terraform we are cloning the Agones repo (as in docs) only at release points. Another thing is if we want to compare Terraform configs of Agones at version 1.6.0 and 1.7.0 to perform regression testing for example, then we can select the same Kubernetes version to compare at the same environment.
This would be useful when we select Kubernetes patch version (say 1.14.8), and then there is a new patch with a fix, so we can adjust the docs and switch to a new one.

@@ -65,6 +64,10 @@ variable "feature_gates" {
default = ""
}

variable "kubernetes_version" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 do we want to be doing this here, or do we want to be advocating that users should defer to the Kubernetes cluster we set the default to? (basically do a "trust us Agones developers, we know best"?)

@roberthbailey WDYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having an override is useful (and we may end up wanting to use it for skew testing in the future) but the docs are pretty clear that we only test/support one version. So we should make sure that any places where we expose this to users have the same warning as we have on https://agones.dev/site/docs/installation/.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree having an override is useful, I feel like we shouldn't be showing it in the example, the example should be best practice - in which case, the best practice will be to default to the kubernetes version that is applied as the default in the module.

WDYT?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, I will not use this param in examples.

install/terraform/modules/gke/variables.tf Show resolved Hide resolved
@aLekSer aLekSer force-pushed the terraform/update-gke-version branch 3 times, most recently from e64182d to 866e20b Compare May 4, 2020 22:22
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 5b491126-0303-476d-89be-02c12d8a6017

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1482/head:pr_1482 && git checkout pr_1482
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-8ccebf4

@aLekSer aLekSer force-pushed the terraform/update-gke-version branch from 866e20b to 9b00da8 Compare May 4, 2020 22:28
@agones-bot
Copy link
Collaborator

Build Failed 😱

Build Id: b6a66ab9-fd1e-4c28-958b-7b6f79d1967b

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 81ffe9c1-1318-4141-98c4-38c38189e764

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1482/head:pr_1482 && git checkout pr_1482
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-e64182d

@aLekSer
Copy link
Collaborator Author

aLekSer commented May 4, 2020

Flaky SDK test in C# make tests step:

/usr/share/dotnet/sdk/2.2.402/NuGet.targets(123,5): error : The file '/go/src/agones.dev/agones/sdks/csharp/sdk/obj/project.assets.json' already exists. [/go/src/agones.dev/agones/sdks/csharp/test/csharp-sdk-test.csproj]
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/transport/timeout_encoding.o
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/transport/transport.o
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/transport/transport_op_string.o
includes/sdk.mk:89: recipe for target 'run-sdk-command' failed
make[1]: *** [run-sdk-command] Error 1
includes/sdk.mk:85: recipe for target 'run-sdk-command-csharp' failed
make: *** [run-sdk-command-csharp] Error 2
make: *** Waiting for unfinished jobs....

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 58e3af84-5e56-42f5-9e54-acf519e803d4

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1482/head:pr_1482 && git checkout pr_1482
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-9b00da8

@aLekSer aLekSer force-pushed the terraform/update-gke-version branch from 864ebfc to 0489b59 Compare May 4, 2020 22:49
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: bd16aa78-3168-4f28-ba3c-61c99b4cad18

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1482/head:pr_1482 && git checkout pr_1482
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-864ebfc

@aLekSer aLekSer force-pushed the terraform/update-gke-version branch from 0489b59 to 9b00da8 Compare May 4, 2020 23:08
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 99f34c62-ab9b-4477-8dfd-421ec3583138

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1482/head:pr_1482 && git checkout pr_1482
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-0489b59

@aLekSer
Copy link
Collaborator Author

aLekSer commented May 4, 2020

@markmandel This Pull Request is ready for review. Note that there is an additional requirement to use var terraform destroy -var project=<GCP_ProjectID>, without it we got the following:

google_container_engine_versions.kubernetes: Refreshing state...                                                                                                                   
                                                                                                                                                                                                           
Error: project: required field is not set                                                                                                                                                                  
                                                                                                                                                                                                           
  on ../../../install/terraform/modules/gke/cluster.tf line 34, in data "google_container_engine_versions" "kubernetes":                                                                                   
  34: data "google_container_engine_versions" "kubernetes" {   

Optionally we can add next workaround in this place:

data "google_container_engine_versions" "kubernetes" {
  # if there is no project specified, for example in destroy
  project        = local.project == "" ? "agones" : local.project 

As per docs this field is totally Optional, however it seems not (provider does not set project either):
https://www.terraform.io/docs/providers/google/d/google_container_engine_versions.html#project

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 62d864f1-3cc0-4fb5-89c9-552e868ea0d3

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1482/head:pr_1482 && git checkout pr_1482
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-9b00da8

@aLekSer
Copy link
Collaborator Author

aLekSer commented May 7, 2020

Well, it seems that I could also test all GKE Terraform related changes using this https://github.com/googleforgames/cloud-game-servers-examples/tree/master/terraform-4-cluster configs for backward compatibility.

@aLekSer aLekSer force-pushed the terraform/update-gke-version branch from 9b00da8 to 839bba1 Compare May 12, 2020 11:53
@aLekSer
Copy link
Collaborator Author

aLekSer commented May 12, 2020

Updating PR, removed data source. Agree with you it was too many changes for one PR. Now this PR looks much more safe.

@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 7ee43c83-69fc-4e4a-984d-a1452d99b49a

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1482/head:pr_1482 && git checkout pr_1482
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-839bba1

@aLekSer aLekSer changed the title Terraform: update GKE cluster version, use data source Terraform: update GKE cluster version, use locals and lookup to set default values May 12, 2020
@aLekSer
Copy link
Collaborator Author

aLekSer commented May 12, 2020

By the way current version without data source lead to the same version as with it - 1.15.11-gke.12 .

@@ -19,37 +19,49 @@ terraform {

data "google_client_config" "default" {}

# A list of all parameters used in interpolation var.cluster
# Set values to default if not key was not set in original map
locals {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chrisst you are our resident TF expert. This look good to you? If so, I'm happy to approve this PR 👍

Copy link
Collaborator Author

@aLekSer aLekSer May 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that I have checked that removing kubernetesVersion from the cluster map
does not affect the result, as it was before.
https://github.com/aLekSer/agones/blob/1f83575dc0f0a7150a9231d9a5402c2d935f1aac/examples/terraform-submodules/gke/module.tf#L74-L81

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@aLekSer aLekSer force-pushed the terraform/update-gke-version branch from 839bba1 to 922d049 Compare May 14, 2020 17:43
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 89f1c007-5203-46df-95aa-b775eaa39688

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1482/head:pr_1482 && git checkout pr_1482
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-922d049

Add Kubernetes version prefix parameter.
`google_container_engine_versions` allows to select proper node and master
version which applicable for every zone in the region.
Avoid situation when missing one key could break deployment.
Move data source to select versions of GKE into a separate review.
@aLekSer aLekSer force-pushed the terraform/update-gke-version branch from 922d049 to 32fbb78 Compare May 14, 2020 18:07
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: 3b55f498-8e72-43f6-922b-2d1bc0f2b236

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1482/head:pr_1482 && git checkout pr_1482
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-32fbb78

Copy link
Member

@markmandel markmandel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!

@google-oss-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aLekSer, markmandel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-robot
Copy link

New changes are detected. LGTM label has been removed.

@markmandel markmandel added the kind/feature New features for Agones label May 14, 2020
@agones-bot
Copy link
Collaborator

Build Succeeded 👏

Build Id: c3672879-0b0c-4a6a-9a6a-dc645dc40170

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1482/head:pr_1482 && git checkout pr_1482
  • helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-a2aa5b8

@markmandel markmandel merged commit da3bc27 into googleforgames:master May 14, 2020
@markmandel markmandel added this to the 1.6.0 milestone May 14, 2020
@aLekSer aLekSer deleted the terraform/update-gke-version branch May 15, 2020 06:10
ilkercelikyilmaz pushed a commit to ilkercelikyilmaz/agones that referenced this pull request Oct 23, 2020
… default values (googleforgames#1482)

* Terraform: update cluster version, use data source

Add Kubernetes version prefix parameter.
`google_container_engine_versions` allows to select proper node and master
version which applicable for every zone in the region.

* Add locals and lookup to handle missing keys

Avoid situation when missing one key could break deployment.

* Remove latest version to default version selection

Move data source to select versions of GKE into a separate review.

Co-authored-by: Mark Mandel <markmandel@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved area/operations Installation, updating, metrics etc kind/feature New features for Agones size/M
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants