Kubernetes Update template, Release template, and site content change for multiple k8s version support

.github/ISSUE_TEMPLATE/kubernetes_update.md

@@ -1,38 +1,51 @@
 ---
-name: Upgrade Kubernetes Version
-about: Issue for updating the Kubernetes version (usually decided in community meetings).
-title: 'Update Kubernetes to {version}'
+name: Upgrade Supported Kubernetes Versions
+about: Issue for updating the supported Kubernetes versions.
+title: 'Update Supported Kubernetes to {version_1} {version_2} {version_3}'
 labels: kind/operations, kind/breaking
 assignees: ''
 
 ---
 
-List of items to do for upgrading to {version}:
+Agones supports and is tested against 3 releases of Kubernetes, targeting the newest version as being the [default version in the GKE Rapid channel](https://cloud.google.com/kubernetes-engine/docs/release-notes#current_versions). The vendored version of client-go will be aligned with the middle of the three supported Kubernetes versions ({version_2}). All the example clusters will use the middle of the three supported Kubernetes versions ({version_2}).
 
-- [ ] Update terraform submodules
-    - [ ] GKE
-    - [ ] Azure
-    - [ ] EKS
-- [ ] Update e2e cluster
-    - [ ] Recreate cluster with new scripts: `cd build/terraform/e2e; terraform apply -var project=agones-images`
-- [ ] Update kubectl in dev tooling
+List of items to do for upgrading to {version_1} {version_2} {version_3}
+
+- [ ] Update the cluster version of terraform submodules in `install/terraform/modules`
+    - [ ] Update Kubernetes version of GKE cluster to {version_2}
+    - [ ] Update Kubernetes version of AKS to the newest supported version in {version_1} {version_2} {version_3}
+    - [ ] Update Kubernetes version of EKS to the newest supported version in {version_1} {version_2} {version_3}
+- [ ] Update kubectl in dev tooling to {version_2}
     - [ ] Update kubectl in `build/build-image/Dockerfile`
     - [ ] Update kubectl in `build/e2e-image/Dockerfile`
-- [ ] Update documentation for creating clusters
-    - [ ] Config.toml `supported_k8s` and related (do `dev_` before main)
-- [ ] Update the dev tooling to create clusters
-    - [ ] Minikube
-    - [ ] Kind
-- [ ] Update the k8s image used in the helm [pre-delete-hook](https://github.com/googleforgames/agones/blob/main/install/helm/agones/templates/hooks/pre_delete_hook.yaml)
-- [ ] Update client-go
-- [ ] Update CRD API reference
+- [ ] Update the Kubernetes version of the below test clusters to {version_2}
+    - [ ] Minikube (Get the patch version [here](https://github.com/kubernetes/minikube/blob/master/CHANGELOG.md))
+    - [ ] Kind (Confirm {version_2} is supported and get the patch version [here](https://github.com/kubernetes-sigs/kind/releases))
+- [ ] Update the k8s image used in the helm [pre-delete-hook](https://github.com/googleforgames/agones/blob/main/install/helm/agones/templates/hooks/pre_delete_hook.yaml) to {version_2}
+- [ ] Update client-go in `go.mod` and `test/terraform/go.mod` to {version_2} and re-run `go mod tidy` and `go mod vendor`
+- [ ] Update CRD API reference to {version_2}
     - [ ] Update links to k8s documentation in `site/assets/templates/crd-doc-config.json`
     - [ ] Regenerate crd api reference docs - `make gen-api-docs`
     - [ ] Regenerate crd client libraries - `make gen-crd-client`
 - [ ] Regenerate Kubernetes resource includes (e.g. ObjectMeta, PodTemplateSpec)
-    - [ ] Start a cluster with `make gcloud-test-cluster`, uninstall agones using `helm uninstall agones -n agones-system`, and then run  `make gen-embedded-openapi` and `make gen-install`
+    - [ ] Start a cluster with `make gcloud-test-cluster` (this cluster will use Kubernetes {version_2}), uninstall agones using `helm uninstall agones -n agones-system`, and then run  `make gen-embedded-openapi` and `make gen-install`
+- [ ] Update documentation for creating clusters and k8s API references to align with the above clusters versions and the k8s API version
+    - [ ] `site/config.toml`
+        - [ ] `dev_supported_k8s`, which are {version_1} {version_2} {version_3}
+        - [ ] `dev_k8s_api_version`, which is {version_2}
+        - [ ] `dev_gke_example_cluster_version`, which is {version_2}
+        - [ ] `dev_aks_example_cluster_version`, which is the newest AKS supported version in {version_1} {version_2} {version_3}
+        - [ ] `dev_eks_example_cluster_version`, which is the newest EKS supported version in {version_1} {version_2} {version_3}
+        - [ ] `dev_minikube_example_cluster_version`, which is {version_2} with the supported patch version
 - [ ] If client-go pulled in a new version of gRPC, then also
     - [ ] Update the SDK [base image grpc version](https://github.com/googleforgames/agones/blob/main/build/includes/sdk.mk#L30) and rebuild the image. Note that this can take a while and in the past we have had to manually push it to gcr because cloud build doesn't like how long it takes.
     - [ ] Regenerate allocated API endpoints: [make gen-allocation-grpc](https://github.com/googleforgames/agones/blob/main/build/includes/allocation.mk#L55)
     - [ ] Regenerate all client sdks: [make gen-all-sdk-grpc](https://github.com/googleforgames/agones/blob/main/build/README.md#make-gen-all-sdk-grpc)
     - [ ] Update the version number in C++ Cmake scripts [here](https://github.com/googleforgames/agones/blob/main/sdks/cpp/CMakeLists.txt#L100) and [here](https://github.com/googleforgames/agones/blob/main/sdks/cpp/cmake/prerequisites.cmake#L34)
+- [ ] Confirm the update works as expected by running e2e tests
+    - [ ] Update the Kubernetes version of the e2e clusters
+        - [ ] In `terraform/e2e/module.tf`, update variable `kubernetes_versions_standard` and `kubernetes_versions_autopilot` to the new versions to be supported
+        - [ ] Recreate cluster with new scripts: `cd build; make GCP_PROJECT=agones-images gcloud-e2e-test-cluster`
+    - [ ] Update the Cloud Build configuration to run e2e test on the new created clusters
+        - [ ] Update the `versionsAndRegions` variable to reflect new versions in `cloudbuild.yaml` `submit-e2e-test-cloud-build` step
+        - [ ] Submit a PR to trigger the e2e tests and verfiy they all pass

docs/governance/kubernetes_update_process.md

@@ -0,0 +1,10 @@
+
+# Kubernetes versions update policy
+
+We will support 3 releases of Kubernetes, targeting the newest version as being the [default version in the GKE Rapid channel](https://cloud.google.com/kubernetes-engine/docs/release-notes#current_versions) (since tests run on GKE we can't support versions that GKE doesn't support). However, we will ensure that at least one of the 3 versions chosen for each Agones release is supported by each of the major cloud providers (EKS and AKS). The vendored version of client-go will be aligned with the middle of the three supported Kubernetes versions.
+
+# Kubernetes versions update process
+
+1. Create a Issue from the [kubernetes update issue template](../../.github/ISSUE_TEMPLATE/kubernetes_update.md) with the newly supported versions.
+2. Complete all items in the issue checklist.
+3. Close the issue.

docs/governance/templates/release_issue.md

@@ -47,16 +47,19 @@ and copy it into a release issue. Fill in relevant values, found inside {}
     - [ ] Make a `tag` with the release version.
 - [ ] Site updated
   - [ ] Copy the draft release content into a new `/site/content/en/blog/releases` content (this will be what you send via email).
-  - [ ] Add the Agones release version and its supported Kubernetes version to the version matrix in `site/content/en/docs/Installation/_index.md #agones-and-kubernetes-supported-versions`.
+  - [ ] In `site/content/en/docs/Installation/_index.md #agones-and-kubernetes-supported-versions`, for the current version, replace `{{% k8s-version %}}` with hardcoded Kubernetes versions supported by the current version. And add a row for the Agones release version with `{{% k8s-version %}}` as its supported Kubernetes versions.
   - [ ] Review all `link_test` and `data-proofer-ignore` attributes and remove for link testing
   - [ ] Review and remove all instances of the `feature` shortcode
   - [ ] Add a link to previous version's documentation to nav dropdown.
   - [ ] config.toml updates:
     - [ ] Update `release_branch` to the new release branch for {version}.
     - [ ] Update `release-version` with the new release version {version}.
     - [ ] Copy `dev_supported_k8s` to `supported_k8s`.
-    - [ ] Copy `dev_aks_minor_supported_k8s` to `aks_minor_supported_k8s`.
-    - [ ] Copy `dev_minikube_minor_supported_k8s` to `minikube_minor_supported_k8s`.
+    - [ ] Copy `dev_k8s_api_version` to `k8s_api_version`.
+    - [ ] Copy `dev_gke_example_cluster_version` to `gke_example_cluster_version`.
+    - [ ] Copy `dev_aks_example_cluster_version` to `aks_example_cluster_version`.
+    - [ ] Copy `dev_eks_example_cluster_version` to `eks_example_cluster_version`.
+    - [ ] Copy `dev_minikube_example_cluster_version` to `minikube_example_cluster_version`.
     - [ ] Update documentation with updated example images tags.
 - [ ] Create PR with these changes, and merge them with an approval.
 - [ ] Run `git remote update && git checkout main && git reset --hard upstream/main` to ensure your code is in line

site/config.toml

@@ -87,13 +87,19 @@ release_branch = "release-1.29.0"
 release_version = "1.29.0"
 
 # shown for production
-supported_k8s = "1.24"
-aks_minor_supported_k8s = "6"
-minikube_minor_supported_k8s = "9"
-# shown in development (or the next version that will be supported)
-dev_supported_k8s = "1.24"
-dev_aks_minor_supported_k8s = "6"
-dev_minikube_minor_supported_k8s = "9"
+supported_k8s = ["1.24"]
+k8s_api_version = "1.24"
+gke_example_cluster_version = "1.24"
+aks_example_cluster_version = "1.24.6"
+eks_example_cluster_version = "1.24"
+minikube_example_cluster_version = "1.24.9"
+# shown in development (or the next versions that will be supported)
+dev_supported_k8s = ["1.23", "1.24", "1.25"]
+dev_k8s_api_version = "1.24"
+dev_gke_example_cluster_version = "1.24"
+dev_aks_example_cluster_version = "1.24.6"
+dev_eks_example_cluster_version = "1.24"
+dev_minikube_example_cluster_version = "1.24.9"
 
 # example tag
 example_image_tag = "us-docker.pkg.dev/agones-images/examples/simple-game-server:0.14"

site/content/en/docs/Advanced/limiting-resources.md

@@ -19,7 +19,7 @@ Kubernetes documentation for more details on "requests" and "limits" to both CPU
 
 ## GameServers
 
-Since the `GameServer` specification provides a full [`PodSpecTemplate`]({{% k8s-api href="#podtemplatespec-v1-core" %}}),
+Since the `GameServer` specification provides a full [`PodSpecTemplate`]({{% k8s-api-version href="#podtemplatespec-v1-core" %}}),
 we can take advantage of both resource limits and requests in our `GameServer` configurations.
 
 For example, to set a CPU limit on our `GameServer` configuration of `250m/0.25` of a CPU,

site/content/en/docs/FAQ/_index.md

@@ -147,7 +147,7 @@ preferentially over the `ExternalIP` node record.
 ### How is traffic routed from the allocated Port to the GameServer container?
 
 Traffic is routed to the GameServer Container utilising the `hostPort` field on a 
-[Pod's Container specification]({{< k8s-api href="#containerport-v1-core" >}}).
+[Pod's Container specification]({{< k8s-api-version href="#containerport-v1-core" >}}).
 
 This opens a port on the host Node and routes traffic to the container 
 via [iptables](https://en.wikipedia.org/wiki/Iptables) or 

site/content/en/docs/Installation/Creating Cluster/aks.md

@@ -28,8 +28,8 @@ az group create --name $AKS_RESOURCE_GROUP --location $AKS_LOCATION
 
 # Create the AKS cluster - this might take some time. Type 'az aks create -h' to see all available options
 
-# The following command will create a four Node AKS cluster. Node size is Standard A1 v1 and Kubernetes version is {{% k8s-version %}}.{{% aks-k8s-minor-version %}}. Plus, SSH keys will be generated for you, use --ssh-key-value to provide your values
-az aks create --resource-group $AKS_RESOURCE_GROUP --name $AKS_NAME --node-count 4 --generate-ssh-keys --node-vm-size Standard_A4_v2 --kubernetes-version {{% k8s-version %}}.{{% aks-k8s-minor-version %}} --enable-node-public-ip
+# The following command will create a four Node AKS cluster. Node size is Standard A1 v1 and Kubernetes version is {{% aks-example-cluster-version %}}. Plus, SSH keys will be generated for you, use --ssh-key-value to provide your values
+az aks create --resource-group $AKS_RESOURCE_GROUP --name $AKS_NAME --node-count 4 --generate-ssh-keys --node-vm-size Standard_A4_v2 --kubernetes-version {{% aks-example-cluster-version %}} --enable-node-public-ip
 
 # Install kubectl
 sudo az aks install-cli

site/content/en/docs/Installation/Creating Cluster/eks.md

@@ -18,7 +18,7 @@ Possible steps are the following:
 ```bash
 eksctl create cluster \
 --name prod \
---version {{% k8s-version %}} \
+--version {{% eks-example-cluster-version %}} \
 --nodegroup-name standard-workers \
 --node-type t3.medium \
 --nodes 3 \

site/content/en/docs/Installation/Creating Cluster/gke.md

@@ -83,7 +83,7 @@ gcloud compute firewall-rules create game-server-firewall \
 A [cluster][cluster] consists of at least one *control plane* machine and multiple worker machines called *nodes*. In Google Kubernetes Engine, nodes are [Compute Engine virtual machine][vms] instances that run the Kubernetes processes necessary to make them part of the cluster.
 
 ```bash
-gcloud container clusters create [CLUSTER_NAME] --cluster-version={{% k8s-version %}} \
+gcloud container clusters create [CLUSTER_NAME] --cluster-version={{% gke-example-cluster-version %}} \
   --tags=game-server \
   --scopes=gke-default \
   --num-nodes=4 \

site/content/en/docs/Installation/Creating Cluster/minikube.md

@@ -24,7 +24,7 @@ Optionally, we also recommend starting with an `agones` profile, using `-p` to k
 clusters you may have running with Minikube.
 
 ```bash
-minikube start --kubernetes-version v{{% k8s-version %}}.{{% minikube-k8s-minor-version %}} -p agones
+minikube start --kubernetes-version v{{% minikube-example-cluster-version %}} -p agones
 ```
 
 Check the official [minikube start](https://minikube.sigs.k8s.io/docs/commands/start/) reference for more options that

site/content/en/docs/Installation/_index.md

@@ -22,9 +22,7 @@ description: >
 - Game Servers must have the [game server SDK]({{< ref "/docs/Guides/Client SDKs/_index.md"  >}}) integrated, to manage Game Server state, health checking, etc.
 
 {{< alert title="Warning" color="warning">}}
-Later versions of Kubernetes may work, but this project is tested against {{% k8s-version %}}, and is therefore the supported version.
-Agones will update its support to the n-1 version of what is available across the majority of major cloud providers - GKE, EKS and
-AKS, while also ensuring that all Cloud providers can support that version.
+This release has been tested against Kubernetes versions {{% k8s-version %}}. Other versions may work, but are unsupported.
 {{< /alert >}}
 
 ## Supported Container Architectures
@@ -45,17 +43,17 @@ Each version of Agones supports a specific version of Kubernetes. When a new ver
 
 The following table lists recent Agones versions and their corresponding required Kubernetes versions:
 
-| Agones version | Kubernetes version |
-| -------------- | ------------------ |
-| 1.29           | 1.24               |
-| 1.28           | 1.23               |
-| 1.27           | 1.23               |
-| 1.26           | 1.23               |
-| 1.25           | 1.22               |
-| 1.24           | 1.22               |
-| 1.23           | 1.22               |
-| 1.22           | 1.21               |
-| 1.21           | 1.21               |
+| Agones version | Kubernetes version   |
+| -------------- | ------------------   |
+| 1.29           | {{% k8s-version %}}  |
+| 1.28           | 1.23                 |
+| 1.27           | 1.23                 |
+| 1.26           | 1.23                 |
+| 1.25           | 1.22                 |
+| 1.24           | 1.22                 |
+| 1.23           | 1.22                 |
+| 1.22           | 1.21                 |
+| 1.21           | 1.21                 |
 
 ## Best Practices
 

site/content/en/docs/Reference/fleet.md

@@ -16,7 +16,7 @@ A full `Fleet` specification is available below and in the {{< ghlink href="exam
 apiVersion: "agones.dev/v1"
 kind: Fleet
 # Fleet Metadata
-# {{< k8s-api href="#objectmeta-v1-meta" >}}
+# {{< k8s-api-version href="#objectmeta-v1-meta" >}}
 metadata:
   name: fleet-example
 spec:

site/content/en/docs/Reference/fleetautoscaler.md

@@ -13,7 +13,7 @@ A full `FleetAutoscaler` specification is available below and in the
 apiVersion: "autoscaling.agones.dev/v1"
 kind: FleetAutoscaler
 # FleetAutoscaler Metadata
-# {{< k8s-api href="#objectmeta-v1-meta" >}}
+# {{< k8s-api-version href="#objectmeta-v1-meta" >}}
 metadata:
   name: fleet-autoscaler-example
 spec:

site/content/en/docs/Reference/gameserver.md

@@ -13,7 +13,7 @@ A full GameServer specification is available below and in the {{< ghlink href="e
 apiVersion: "agones.dev/v1"
 kind: GameServer
 # GameServer Metadata
-# {{< k8s-api href="#objectmeta-v1-meta" >}}
+# {{< k8s-api-version href="#objectmeta-v1-meta" >}}
 metadata:
   # generateName: "gds-example" # generate a unique name, with the given prefix
   name: "gds-example" # set a fixed name
@@ -77,7 +77,7 @@ spec:
   #   # set this GameServer's initial player capacity
   #   initialCapacity: 10
   # Pod template configuration
-  # {{< k8s-api href="#podtemplate-v1-core" >}}
+  # {{< k8s-api-version href="#podtemplate-v1-core" >}}
   template:
     # pod metadata. Name & Namespace is overwritten
     metadata:
@@ -125,7 +125,7 @@ The `spec` field is the actual GameServer specification and it is composed as fo
   - `grpcPort` the port that the SDK Server binds to for gRPC connections
   - `httpPort` the port that the SDK Server binds to for HTTP gRPC gateway connections
 - `players` (Alpha, behind "PlayerTracking" feature gate), sets this GameServer's initial player capacity
-- `template` the [pod spec template]({{% k8s-api href="#podtemplatespec-v1-core" %}}) to run your GameServer containers, [see](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/#pod-templates) for more information.
+- `template` the [pod spec template]({{% k8s-api-version href="#podtemplatespec-v1-core" %}}) to run your GameServer containers, [see](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/#pod-templates) for more information.
 
 {{< alert title="Note" color="info">}}
 The GameServer resource does not support updates. If you need to make regular updates to the GameServer spec, consider using a [Fleet]({{< ref "/docs/Reference/fleet.md" >}}).

site/layouts/shortcodes/aks-example-cluster-version.html

@@ -0,0 +1,5 @@
+{{- if or (eq (getenv "HUGO_ENV") "production") (eq (getenv "HUGO_ENV") "snapshot")}}
+{{- $.Page.Site.Params.aks_example_cluster_version }}
+{{- else }}
+{{- $.Page.Site.Params.dev_aks_example_cluster_version }}
+{{- end -}}

site/layouts/shortcodes/eks-example-cluster-version.html

@@ -0,0 +1,5 @@
+{{- if or (eq (getenv "HUGO_ENV") "production") (eq (getenv "HUGO_ENV") "snapshot")}}
+{{- $.Page.Site.Params.eks_example_cluster_version }}
+{{- else }}
+{{- $.Page.Site.Params.dev_eks_example_cluster_version }}
+{{- end -}}

site/layouts/shortcodes/gke-example-cluster-version.html

@@ -0,0 +1,5 @@
+{{- if or (eq (getenv "HUGO_ENV") "production") (eq (getenv "HUGO_ENV") "snapshot")}}
+{{- $.Page.Site.Params.gke_example_cluster_version }}
+{{- else }}
+{{- $.Page.Site.Params.dev_gke_example_cluster_version }}
+{{- end -}}

site/layouts/shortcodes/k8s-api.html → site/layouts/shortcodes/k8s-api-version.html

@@ -1,8 +1,8 @@
 <!-- $version must be assigned outside of if statement for variable scope purposes.
-  Default to "supported_k8s" version. -->
-{{- $version := $.Page.Site.Params.supported_k8s }}
+  Default to "k8s_api_version" version. -->
+{{- $version := $.Page.Site.Params.k8s_api_version }}
 {{- if and (ne (getenv "HUGO_ENV") "production") (ne (getenv "HUGO_ENV") "snapshot")}}
-{{- $version = $.Page.Site.Params.dev_supported_k8s }}
+{{- $version = $.Page.Site.Params.dev_k8s_api_version }}
 {{- end}}
 {{- $prefix := replace $version "." "-" }}https://v{{- $prefix
 }}.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v{{- $version }}/{{- .Get "href" -}}

site/layouts/shortcodes/k8s-version.html

@@ -1,5 +1,15 @@
+{{- $index := 0}}
+{{- $versionsList := slice }}
 {{- if or (eq (getenv "HUGO_ENV") "production") (eq (getenv "HUGO_ENV") "snapshot")}}
-{{- $.Page.Site.Params.supported_k8s }}
+{{- $versionsList = $.Page.Site.Params.supported_k8s}}
 {{- else }}
-{{- $.Page.Site.Params.dev_supported_k8s }}
-{{- end -}}
+{{- $versionsList = $.Page.Site.Params.dev_supported_k8s }}
+{{- end}}
+{{- $length := len $versionsList}}
+{{- range $versionsList}}
+    {{- (index $versionsList $index) }}
+    {{- if (ne $index (sub $length 1))}}
+    {{- ", "}}
+    {{- end}}
+    {{- $index = (add $index 1) }}
+{{- end}}