Skip to content

GCP: delete N1 nodepools for the CI clusters #8225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 24, 2025
Merged

GCP: delete N1 nodepools for the CI clusters #8225

merged 1 commit into from
Jun 24, 2025

Conversation

ameukam
Copy link
Member

@ameukam ameukam commented Jun 24, 2025 •
edited
Loading

The node pools were reduced to 0 nodes

closes #2438

@ameukam
GCP: delete N1 nodepools for the CI clusters
7823351 
The nodepools were reduced to 0 nodes

Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 24, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ameukam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. area/infra Infrastructure management, infrastructure design, code in infra/ labels Jun 24, 2025
@ameukam
Copy link
Member Author

ameukam commented Jun 24, 2025

cc @dims @upodroid

@k8s-ci-robot k8s-ci-robot added area/infra/gcp Issues or PRs related to Kubernetes GCP infrastructure area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters labels Jun 24, 2025
@k8s-ci-robot k8s-ci-robot requested review from cjwagner and puerco June 24, 2025 09:32
@k8s-ci-robot k8s-ci-robot added area/terraform Terraform modules, testing them, writing more of them, code in infra/gcp/clusters/ sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 24, 2025
@k8s-infra-ci-robot
Copy link
Contributor

Ran Plan for 2 projects:

  1. dir: infra/gcp/terraform/k8s-infra-prow-build-trusted workspace: default
  2. dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

1. dir: infra/gcp/terraform/k8s-infra-prow-build-trusted workspace: default

Show Output 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
- destroy

Terraform will perform the following actions:

  # module.prow_build_nodepool.google_container_node_pool.node_pool will be destroyed
  # (because google_container_node_pool.node_pool is not in configuration)
- resource "google_container_node_pool" "node_pool" {
      - cluster                     = (sensitive value) -> null
      - id                          = "projects/k8s-infra-prow-build-trusted/locations/us-central1/clusters/prow-build-trusted/nodePools/trusted-pool1-20250619150831625500000001" -> null
      - initial_node_count          = 0 -> null
      - instance_group_urls         = [
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build-trusted/zones/us-central1-a/instanceGroupManagers/gke-prow-build-trust-trusted-pool1-20-3cab2aee-grp",
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build-trusted/zones/us-central1-b/instanceGroupManagers/gke-prow-build-trust-trusted-pool1-20-dba540f6-grp",
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build-trusted/zones/us-central1-f/instanceGroupManagers/gke-prow-build-trust-trusted-pool1-20-52e4044d-grp",
        ] -> null
      - location                    = (sensitive value) -> null
      - managed_instance_group_urls = [
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build-trusted/zones/us-central1-a/instanceGroups/gke-prow-build-trust-trusted-pool1-20-3cab2aee-grp",
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build-trusted/zones/us-central1-b/instanceGroups/gke-prow-build-trust-trusted-pool1-20-dba540f6-grp",
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build-trusted/zones/us-central1-f/instanceGroups/gke-prow-build-trust-trusted-pool1-20-52e4044d-grp",
        ] -> null
      - name                        = "trusted-pool1-20250619150831625500000001" -> null
      - name_prefix                 = "trusted-pool1-" -> null
      - node_count                  = 0 -> null
      - node_locations              = [
          - "us-central1-a",
          - "us-central1-b",
          - "us-central1-f",
        ] -> null
      - project                     = "k8s-infra-prow-build-trusted" -> null
      - version                     = "1.32.4-gke.1353003" -> null

      - autoscaling {
          - location_policy      = "BALANCED" -> null
          - max_node_count       = 0 -> null
          - min_node_count       = 0 -> null
          - total_max_node_count = 0 -> null
          - total_min_node_count = 0 -> null
        }

      - management {
          - auto_repair  = true -> null
          - auto_upgrade = true -> null
        }

      - node_config {
          - disk_size_gb                = 200 -> null
          - disk_type                   = "pd-ssd" -> null
          - effective_taints            = [] -> null
          - enable_confidential_storage = false -> null
          - image_type                  = "UBUNTU_CONTAINERD" -> null
          - labels                      = {} -> null
          - local_ssd_count             = 0 -> null
          - logging_variant             = "DEFAULT" -> null
          - machine_type                = "n1-highmem-8" -> null
          - metadata                    = {
              - "disable-legacy-endpoints" = "true"
            } -> null
          - oauth_scopes                = [
              - "https://www.googleapis.com/auth/cloud-platform",
            ] -> null
          - preemptible                 = false -> null
          - resource_labels             = {
              - "goog-gke-node-pool-provisioning-model" = "on-demand"
            } -> null
          - resource_manager_tags       = {} -> null
          - service_account             = "gke-nodes-prow-build-trusted@k8s-infra-prow-build-trusted.iam.gserviceaccount.com" -> null
          - spot                        = false -> null
          - storage_pools               = [] -> null
          - tags                        = [] -> null
            # (5 unchanged attributes hidden)

          - kubelet_config {
              - allowed_unsafe_sysctls                 = [] -> null
              - container_log_max_files                = 0 -> null
              - cpu_cfs_quota                          = false -> null
              - image_gc_high_threshold_percent        = 0 -> null
              - image_gc_low_threshold_percent         = 0 -> null
              - insecure_kubelet_readonly_port_enabled = "TRUE" -> null
              - pod_pids_limit                         = 0 -> null
                # (5 unchanged attributes hidden)
            }

          - shielded_instance_config {
              - enable_integrity_monitoring = true -> null
              - enable_secure_boot          = false -> null
            }

          - windows_node_config {
                # (1 unchanged attribute hidden)
            }

          - workload_metadata_config {
              - mode = "GKE_METADATA" -> null
            }
        }

      - upgrade_settings {
          - max_surge       = 1 -> null
          - max_unavailable = 0 -> null
          - strategy        = "SURGE" -> null
        }
    }

Plan: 0 to add, 0 to change, 1 to destroy.
  • ▶️ To apply this plan, comment: 
    atlantis apply -d infra/gcp/terraform/k8s-infra-prow-build-trusted
  • 🚮 To delete this plan and lock, click here
  • 🔁 To plan this project again, comment: 
    atlantis plan -d infra/gcp/terraform/k8s-infra-prow-build-trusted

Plan: 0 to add, 0 to change, 1 to destroy.

2. dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default

Show Output 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
~ update in-place
- destroy

Terraform will perform the following actions:

  # google_iam_workload_identity_pool_provider.eks_kops will be created
+ resource "google_iam_workload_identity_pool_provider" "eks_kops" {
      + attribute_mapping                  = {
          + "google.subject" = "assertion.sub"
        }
      + display_name                       = "kops"
      + id                                 = (known after apply)
      + name                               = (known after apply)
      + project                            = "k8s-infra-prow-build"
      + state                              = (known after apply)
      + workload_identity_pool_id          = "prow-eks"
      + workload_identity_pool_provider_id = "kops"

      + oidc {
          + allowed_audiences = [
              + "sts.googleapis.com",
            ]
          + issuer_uri        = "https://oidc.eks.us-east-2.amazonaws.com/id/7283E85C59E9C4129CFD07BAC9378D44"
        }
    }

  # google_vmwareengine_network_peering.gvce_peering will be updated in-place
~ resource "google_vmwareengine_network_peering" "gvce_peering" {
      ~ export_custom_routes_with_public_ip = false -> true
        id                                  = "projects/k8s-infra-prow-build/locations/global/networkPeerings/peer-with-gcve-project"
      ~ import_custom_routes_with_public_ip = false -> true
        name                                = "peer-with-gcve-project"
        # (13 unchanged attributes hidden)
    }

  # module.iam.google_project_iam_binding.project_iam_authoritative["default--roles/secretmanager.secretAccessor"] will be updated in-place
~ resource "google_project_iam_binding" "project_iam_authoritative" {
        id      = "k8s-infra-prow-build/roles/secretmanager.secretAccessor"
      ~ members = [
          - "principal://iam.googleapis.com/projects/180382678033/locations/global/workloadIdentityPools/k8s-infra-prow-build-trusted.svc.id.goog/subject/ns/external-secrets/sa/external-secrets",
            # (2 unchanged elements hidden)
        ]
        # (3 unchanged attributes hidden)
    }

  # module.prow_build_nodepool_c4_highmem_8_localssd.google_container_node_pool.node_pool will be updated in-place
~ resource "google_container_node_pool" "node_pool" {
        id                          = "projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool6-20250327232037500200000001"
        name                        = "pool6-20250327232037500200000001"
        # (10 unchanged attributes hidden)

      ~ autoscaling {
          ~ max_node_count       = 25 -> 80
            # (4 unchanged attributes hidden)
        }

        # (3 unchanged blocks hidden)
    }

  # module.prow_build_nodepool_c4d_highmem_8_localssd.google_container_node_pool.node_pool will be updated in-place
~ resource "google_container_node_pool" "node_pool" {
        id                          = "projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool7-20250528124554315100000001"
        name                        = "pool7-20250528124554315100000001"
        # (10 unchanged attributes hidden)

      ~ autoscaling {
          ~ min_node_count       = 15 -> 10
            # (4 unchanged attributes hidden)
        }

        # (3 unchanged blocks hidden)
    }

  # module.prow_build_nodepool_n1_highmem_8_localssd.google_container_node_pool.node_pool will be destroyed
  # (because google_container_node_pool.node_pool is not in configuration)
- resource "google_container_node_pool" "node_pool" {
      - cluster                     = (sensitive value) -> null
      - id                          = "projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool5-20210928124956061000000001" -> null
      - initial_node_count          = 0 -> null
      - instance_group_urls         = [
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build/zones/us-central1-c/instanceGroupManagers/gke-prow-build-pool5-2021092812495606-e8f905a4-grp",
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build/zones/us-central1-f/instanceGroupManagers/gke-prow-build-pool5-2021092812495606-72cb7850-grp",
          - "https://www.googleapis.com/compute/v1/projects/k8s-infra-prow-build/zones/us-central1-b/instanceGroupManagers/gke-prow-build-pool5-2021092812495606-3a8095df-grp",
        ] -> null
      - location                    = (sensitive value) -> null
      - managed_instance_group_urls = [
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build/zones/us-central1-c/instanceGroups/gke-prow-build-pool5-2021092812495606-e8f905a4-grp",
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build/zones/us-central1-f/instanceGroups/gke-prow-build-pool5-2021092812495606-72cb7850-grp",
          - "https://www.googleapis.com/compute/beta/projects/k8s-infra-prow-build/zones/us-central1-b/instanceGroups/gke-prow-build-pool5-2021092812495606-3a8095df-grp",
        ] -> null
      - name                        = "pool5-20210928124956061000000001" -> null
      - name_prefix                 = "pool5-" -> null
      - node_count                  = 0 -> null
      - node_locations              = [
          - "us-central1-b",
          - "us-central1-c",
          - "us-central1-f",
        ] -> null
      - project                     = "k8s-infra-prow-build" -> null
      - version                     = "1.32.4-gke.1353003" -> null

      - autoscaling {
          - location_policy      = "BALANCED" -> null
          - max_node_count       = 0 -> null
          - min_node_count       = 0 -> null
          - total_max_node_count = 0 -> null
          - total_min_node_count = 0 -> null
        }

      - management {
          - auto_repair  = true -> null
          - auto_upgrade = true -> null
        }

      - node_config {
          - disk_size_gb                = 100 -> null
          - disk_type                   = "pd-standard" -> null
          - effective_taints            = [] -> null
          - enable_confidential_storage = false -> null
          - image_type                  = "UBUNTU_CONTAINERD" -> null
          - labels                      = {} -> null
          - local_ssd_count             = 0 -> null
          - logging_variant             = "DEFAULT" -> null
          - machine_type                = "n1-highmem-8" -> null
          - metadata                    = {
              - "disable-legacy-endpoints" = "true"
            } -> null
          - oauth_scopes                = [
              - "https://www.googleapis.com/auth/cloud-platform",
            ] -> null
          - preemptible                 = false -> null
          - resource_labels             = {} -> null
          - resource_manager_tags       = {} -> null
          - service_account             = "gke-nodes-prow-build@k8s-infra-prow-build.iam.gserviceaccount.com" -> null
          - spot                        = false -> null
          - storage_pools               = [] -> null
          - tags                        = [] -> null
            # (5 unchanged attributes hidden)

          - ephemeral_storage_config {
              - local_ssd_count = 2 -> null
            }

          - shielded_instance_config {
              - enable_integrity_monitoring = true -> null
              - enable_secure_boot          = false -> null
            }

          - workload_metadata_config {
              - mode = "GKE_METADATA" -> null
            }
        }

      - upgrade_settings {
          - max_surge       = 1 -> null
          - max_unavailable = 0 -> null
          - strategy        = "SURGE" -> null
        }
    }

Plan: 1 to add, 4 to change, 1 to destroy.
  • ▶️ To apply this plan, comment: 
    atlantis apply -d infra/gcp/terraform/k8s-infra-prow-build
  • 🚮 To delete this plan and lock, click here
  • 🔁 To plan this project again, comment: 
    atlantis plan -d infra/gcp/terraform/k8s-infra-prow-build

Plan: 1 to add, 4 to change, 1 to destroy.

Plan Summary

2 projects, 2 with changes, 0 with no changes, 0 failed

  • ⏩ To apply all unapplied plans from this Pull Request, comment: 
    atlantis apply
  • 🚮 To delete all plans and locks from this Pull Request, comment: 
    atlantis unlock

@ameukam
Copy link
Member Author

ameukam commented Jun 24, 2025

/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 24, 2025
@upodroid
Copy link
Member

The layer has some diffs, lets merge the PR and we can delete the nodepool via the console

I'm working on it

upodroid
upodroid reviewed Jun 24, 2025
View reviewed changes
Copy link
Member

@upodroid upodroid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 24, 2025
@ameukam
Copy link
Member Author

ameukam commented Jun 24, 2025

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 24, 2025
@k8s-ci-robot k8s-ci-robot merged commit 3884ae2 into kubernetes:main Jun 24, 2025
8 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.34 milestone Jun 24, 2025
@k8s-infra-ci-robot
Copy link
Contributor

Locks and plans deleted for the projects and workspaces modified in this pull request:

  • dir: infra/gcp/terraform/k8s-infra-prow-build workspace: default
  • dir: infra/gcp/terraform/k8s-infra-prow-build-trusted workspace: default

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@upodroid upodroid upodroid left review comments

@cjwagner cjwagner Awaiting requested review from cjwagner

@puerco puerco Awaiting requested review from puerco

Assignees

@upodroid upodroid

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/infra/gcp Issues or PRs related to Kubernetes GCP infrastructure area/infra Infrastructure management, infrastructure design, code in infra/ area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters area/terraform Terraform modules, testing them, writing more of them, code in infra/gcp/clusters/ cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
v1.34
Development

Successfully merging this pull request may close these issues.

Change instance type for GKE build clusters
4 participants
@ameukam @k8s-ci-robot @k8s-infra-ci-robot @upodroid