[get.jenkins.io] Track storage usage and provisionning

[dduportal]

Service(s)

Azure, get.jenkins.io, Other

Summary

Since #3917, the file storage used for get.jenkins.io is provisioned as code by https://github.com/jenkins-infra/azure/blob/65cac226ef66b9a7f15af1a8c24a92ea4a6f0c35/get.jenkins.io.tf#L40-L44

We have 2 main tasks from this:

• Keep the Kubernetes PV/PVC definition in sync with the provisioned value (at least the size in Gb). An updatecli manifest on jenkins-infra/kubernetes-management should be good enough for this sync

  • e.g.. if we change terraform in jenkins-infra/azure then we expect a PR on jenkins-infra/kubernetes-management so we control "when" and "what" we change. Because changing values on a PVC can fail (immutable values requiring re-creation of the PV/PVC) or have big impact
  • Attributes to be synced:
    • Size (mandatory): https://github.com/jenkins-infra/kubernetes-management/blob/ad94ce33fe435be7282e90758f415ccb4055920e/config/get-jenkins-io.yaml#L46
  • Resource Group name: https://github.com/jenkins-infra/kubernetes-management/blob/ad94ce33fe435be7282e90758f415ccb4055920e/config/get-jenkins-io.yaml#L51
  • File share name: https://github.com/jenkins-infra/kubernetes-management/blob/ad94ce33fe435be7282e90758f415ccb4055920e/config/get-jenkins-io.yaml#L52

• We also have to check datadog + pagerduty to alert us when we reach the 80% usage in the volume, so we can increase the provisioned size

  • Reminder: before [get.jenkins.io/mirrors/mirrorbit - Azure] High costs due to usage of Azure File Storage #3917 we had 1024Gb provisioned but we went for a 700 Gb. This volume will always grow but no need to pay for provisionned storage we don't use.

Reproduction steps

No response

[lemeurherve]

Opened jenkins-infra/kubernetes-management#4918 for the first task of the issue.

Note, I've included the File Share name and resource group name but I'm wondering if it's not a bit overkill.

[lemeurherve]

Opened jenkins-infra/datadog#241 for the second task, monitoring all Azure File Share volume usages.

[lemeurherve]

As noted in jenkins-infra/kubernetes-management#4918 (comment),

you will add the websites FileShare check here too ?

Not here:

• the website File Share is for the release core process, not directly related to get.jenkins.io service
• the values for the website File Share are stored as a secret with sops(1) for release.ci.jenkins.io config

[dduportal]

Closing as we've waited 2 weeks and the monitoring looks good 👍