Install Rook/Ceph which supports CSI volume snapshot in preview environment

[jenting]

Description

Install Rook/Ceph v1.9.5 in the preview environment which supports CSI volume snapshot.
We overwrite some default values:

• Disable the Ceph FS

  gitpod/.werft/vm/manifests/rook-ceph/operator.yaml

  Line 30 in d19c9b5

  ROOK_CSI_ENABLE_CEPHFS: "false"

• StorageClass
  • Disable the volume expansion

    gitpod/.werft/vm/manifests/rook-ceph/storageclass-test.yaml

    Line 55 in d19c9b5

    allowVolumeExpansion: false

  • Change volumeBindingMode from Immediate to WaitForFirstConsumer

    gitpod/.werft/vm/manifests/rook-ceph/storageclass-test.yaml

    Line 57 in d19c9b5

    volumeBindingMode: WaitForFirstConsumer

Related Issue(s)

Fixes #10201

How to test

1. Open the URL https://jenting-in5edb98e0f2.preview.gitpod-dev.com/workspaces
2. Enable the feature flag persistent_volume_claim in the admin panel.
3. Launch a workspace, for example, launch github.com/gitpod-io/gitpod.
4. Write some data into it.
5. Stop the workspace.
6. Launch the workspace again, and make sure the data is persistent.

Release Notes

None

Documentation

None

[werft-gitpod-dev-com]

started the job as gitpod-build-jenting-install-a-storage-vendor-10201.22 because the annotations in the pull request description changed
(with .werft/ from main)

[sagor999]

code LGTM
but adding hold until I can test it (currently not an admin in this preview env, so cannot enable feature flag for PVC use)
/hold

[vulkoingim]

I think this breaks VMs - I've reverted the one pushed to master here: #10735

[vulkoingim]

Looks good, besides the change to virtio I commented on 👍

I trust that you've tested it :)

[meysholdt]

hi @jenting, thx for this PR!

I see you're adding a 2nd volume to all preview VMs... can you share how this is helpful?
For the sake of simplicity, would it work as well if Rook/Ceph would use folders on the root volume?

[sagor999]

Tested, works like a charm. Removing my hold.
/unhold

[sagor999]

Why did it merge??? There was still a pending review required from @meysholdt
@gitpod-io/platform

[ArthurSens]

I believe Aleks review already counted as platform's approval 🙂

[jenting]

hi @jenting, thx for this PR!

I see you're adding a 2nd volume to all preview VMs... can you share how this is helpful? For the sake of simplicity, would it work as well if Rook/Ceph would use folders on the root volume?

The Rook/Ceph deprecated directory-based storage, ref to issue.

According to the Roo/Ceph requirements, at least one of these local storage options is required:

• Raw devices (no partitions or formatted filesystems)
• Raw partitions (no formatted filesystem)

From the current VM image, the block device information is

$ lsblk -f
NAME FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0
     squash 4.0                                                    0   100% /snap/core20/1405
loop1
     squash 4.0                                                    0   100% /snap/core18/2409
loop2
     squash 4.0                                                    0   100% /snap/google-cloud-sdk/243
loop3
     squash 4.0                                                    0   100% /snap/lxd/22923
loop4
     squash 4.0                                                    0   100% /snap/snapd/15534
loop5
     squash 4.0                                                    0   100% /snap/snapd/15904
sda                                                                         
├─sda1
│    xfs          root  4aa52ca2-508b-425c-b595-bc1667b2e800  164.4G    18% /var/lib/kubelet/pods/c700ec62-dcec-4434-90c5-64b9b29b5aa5/volume-subpaths/config/mysql/2
│                                                                           /var/lib/kubelet/pods/471099cc-7e33-4bd9-b41a-cb771f6ac5fe/volume-subpaths/configuration/image-builder-mk3/0
│                                                                           /var/lib/kubelet/pods/791bb338-c72b-4178-87ae-76c61991ca37/volume-subpaths/config/public-api-server/0
│                                                                           /var/lib/kubelet/pods/471099cc-7e33-4bd9-b41a-cb771f6ac5fe/volume-subpaths/gitpod-ca-certificate/update-ca-certificates/1
│                                                                           /var/lib/kubelet/pods/51cccdf7-1791-4593-901a-bc354bbf48c0/volume-subpaths/gitpod-ca-certificate/update-ca-certificates/1
│                                                                           /var/lib/kubelet/pods/78f92750-bf76-4999-9a46-26223fe90835/volume-subpaths/config/fluent-bit/1
│                                                                           /var/lib/kubelet/pods/78f92750-bf76-4999-9a46-26223fe90835/volume-subpaths/config/fluent-bit/0
│                                                                           /
├─sda14
│                                                                           
└─sda15
     vfat   FAT32 EFI   71CA-7080                              99.1M     5% /boot/efi
vda  iso966 Jolie cidata
                        2022-06-08-02-01-51-00                              

We could consider using sda14 or sda15, however, after deploying Rook/Ceph, the sda14 and sda15 aren't fulfilled the Rook/Ceph basic requirements, here is the log and ceph status output.

2022-06-07 15:39:08.572196 I | cephosd: skipping device "sda14": ["Insufficient space (<5GB)"].
2022-06-07 15:39:08.572246 I | cephosd: skipping device "sda15" with mountpoint "efi"
2022-06-07 15:39:08.572252 I | cephosd: skipping device "vda" because it contains a filesystem "iso9660"

Therefore, without changing the existing partitions, the feasible way is the mount the 2nd block device as storage.
We choose 30Gi because the Gitpod regular workspace disk size is 30Gi now.