Add a new private kubernetes cluster in the new sponsored azure subscription

[smerle33]

Service(s)

Azure, infra.ci.jenkins.io

Summary

as per #3918 (comment)

• "Using subscription credits instead of CDF money" following [Sponsorships] Setup the secondary Azure subscription to consume the sponsor credits #3818 and Migrate ci.jenkins.io to the sponsored subscription #3913
• "Continue work about infra.ci on arm64" as part of infra.ci.jenkins.io on arm64 (controller and agents) #3823

=> Let's scope the initial implementation to only infra.ci.jenkins.io agents, and only 1 "non system" nodepool of type linux/arm64 so we can start switching workloads out of privatek8s.

add a new AKS cluster only for infra.ci.jenkins.io and release.ci.jenkins.io Kubernetes agents

• Create a new AKS cluster privatek8s-sponsored
  • Managed in jenkins-infra/azure with Terraform see Add a new private kubernetes cluster in the new sponsored azure subscription #3923 (comment)
  • Same settings as the current [privatek8s] (https://github.com/jenkins-infra/azure/blob/0cf6c87851af116dc6b7ec955683753db8b47ce5/privatek8s.tf#L24) but with only nodepools for agents
  • set network to manage bridge between the current privatek8s cluster and this one
• add this cluster to kubernetes-management
  • Creates a service-account with token + insert it in infra.ci secrets
  • Add initial "foundation" services (private + public ingress, cert-manager, datadog, falco)

use the new cluster from infra.ci.jenkins.io

• Add the generated secrets from kubernetes in the infra.ci controller
  • Add to sops
  • Add to jcasc
• Create a new kubernetes cloud in infra.ci
  • Manually first
  • Port as code in jcasc

migration/cleanup during this time:

• Switch the workload to the new kubernetes cloud kubernetes-infrasicjio-agents1
• check all jobs and for each using storage account, check if they need an extension of their network rules to add the new cluster
• remove the old kubernetes cloud from infra.ci to avoid consuming

Definition of done

when the cost will be moved from the CDF payed account to the sponsored one

[smerle33]

Update: proposal for the new AKS cluster to be soon created:

• Subscription: the Azure "sponsored" subscription (same as cijenkinsio-agents-1 [ci.jenkins.io] Migrate ci.jenkins.io EKS clusters out from CloudBees AWS account #3954)

• Name: infracijenkinsio-agents-1. This name is valid as per learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/create-upgrade-delete/aks-common-issues-faq#what-naming-restrictions-are-enforced-for-aks-resources-and-parameters-

  • No dots, only hyphens, letters and number. Less than 63 characters.
  • Full name of the service (infracijenkinsioagents1) to make identification easier
  • agents wording to make explicit this is the only acceptable usage for this cluster
  • Suffix -1 as we'll most probably need to create more clusters in the future (AWS and eventually DOKS): migration will be easier if we can increment while keeping the the same naming convention

• Connectivity:

  • API access restricted to Jenkins Infra admins, privatek8s (to allow infra.ci to operate the cluster with terraform AND spawning agents)
    • Should use its own subnet in the ["infra_ci_jenkins_io_sponsorship"] virtual network.
    • ⚠️ Need to refine the network sizing: This vnet already is a /23 and already has 2 x /24 subnets. Need to carefully plan the sizing of the new subnet with the AKS network rules + sizing of Nodes and pods.
    • No ingress controller to be added:
      • None was present on cik8s
      • The ACP is not needed on infra.
      • No network segregation between node pools: this cluster is not multi tenant

• Node pools: (same convention as ci [ci.jenkins.io] Migrate ci.jenkins.io EKS clusters out from CloudBees AWS account #3954 (comment))

  • All Linux node pools should be Azure Linux as base OS (already done for infra.ci and release.ci controllers)

  • Naming convention: given the constraint when naming node pool (ref. ):

    • Linux Node pools (12 char. max.) will have:
      • the OS on 1-char (l for Azure Linux, w for Windows, u for Ubuntu Linux)
      • the CPU arch on 3 chars (x86 for Intel/AMD x86_64 or a64 for arm64)
      • The estimation of the number of pods agents expected to be run on a single node of this pool (integer) preceded by a n (n3, n24, etc.) on 3 chars max
      • An eventual suffix on 3 letters to specify a custom usage. May replace the sizing if needed.

  • Linux node pools naming examples:

    • lx86n3 => Azure Linux x86_64 nodes used which can run 3 "normal" pod agents at the same time
    • la64n2side => Linux arm64 nodes used to run 2 "side" pod (e.g. custom applications such as ACP).

  • no Windows Node pool planned

  • Expecting the following mappings (comparing to the existing privatek8s AKS cluster):

    • syspool => exact matching with privatek8s
      • Auto-scaled with minimum of 1 and maximum of 3 as per recommandations
    • infracipool lx86n41 (Azure Linux node pool for agents number "1" supporting 4x pod agents)
      • No spot as we don't have access in the Azure subscription
      • Auto-scaled with minimum of 0 and maximum of 20 (minimum 0 as no spot instances)
    • infraciarm64 => la64n21 (Azure Linux node for agents ARM64 pool number "1" supporting 2x pod agents)
      • No spot as we don't have access in the Azure subscription
      • Auto-scaled with minimum of 0 and maximum of 20 (minimum 0 as no spot instances)

EDIT:
pool names : lx86n14agt1 and la64n14agt1

[smerle33]

the only expected "application" is the Datadog's cluster-agent (2 pods). We should reuse the system pool to run it as it's not an heavy consumer: confirmed by checking the load on the ci.jenkins.io-agents-1 cluster with the same kind of node pool.

TODO: add taint toleration to allow datadog cluster-agent to spawn on the system pool "CriticalAddonsOnly=true:NoSchedule" taint to the default node pool

[smerle33]

datadog namespace created manually:

>kubectl get ns                  
NAME              STATUS   AGE
datadog           Active   5s

[dduportal]

Reopening: The cluster is not used by infra.ci yet

[smerle33]

We had to :

• change the certificate for the cluster to be non encoded as base64 (vs the documentation of the kubernetes plugin) - fix(privatek8s/infra.ci): decode base64 certificate for kubernetes cloud kubernetes-management#5332
• allow access to the azure vault through the azure UI and feat(vnets) enable Keyvault and Storage endpoints for infracijio-agents-1 subnet azure-net#258
• allow peering between this new cluster infracijenkinsioagent1 and privatek8s and publick8s - feat(vnets) allow infracijio-agents1 to reach private AKS clusters azure-net#259 and hotfix(infraci-sponsorship): fix provider for peering to public sponsorship azure-net#260

[dduportal]

Update: infra.ci.jenkins.io is now using the new cluster.

• Config updated by feat(privatek8s/infra.ci.jenkins.io) migrate container agents workload to infracijioagents1 kubernetes-management#5342 (removed the jnlp-windows and renamed the jnlp-linux)
• As the outbound IP changed, some fixes had to be done. The Terraform project Azure was "locked" due to these blocker: I had to run some hotfixes from local machine to unblock: lesson learned for later
  • Missing PR between new cluster and DB vnet: hotfix(vnets) add peering between infracijio-agents-1 and public-db azure-net#262
  • Missing cluster subnet in allow-list regarding IP restrictions: jenkins-infra/azure@4ba0615 and (private) https://github.com/jenkins-infra/terraform-states/commit/9f21cca289743ce465ee610d36c8be26662ed463

=> All Kubernetes, Terraform and Website jobs are green

[dduportal]

Keeping the issue opened for the week end. Some cleanup might be needed, but CDF is not paying anymore for these agents!

[dduportal]

Update: a bit of cleanup with jenkins-infra/azure#747 and jenkins-infra/kubernetes-management#5343

[dduportal]

Closing as it works as expected