Pod Security Policy for filesystem resize

[Bryji]

If you attempt to change the size of the storage volume (in AWS), the operator configuration setting spilo_privileged must be set to true otherwise you get this error in the logs (placing this for future googlers encountering this):

could not sync cluster: could not sync persistent volumes: could not sync volumes: could not resize the filesystem on pod \"<namespace>/<pod name>\": could not execute: command terminated with exit code 1

Potential improvements:

• Use a PodSecurityPolicy with finer grained permissions so the running pods in privileged mode is not required for this feature
• documentation: mention this config in the resizing section: https://postgres-operator.readthedocs.io/en/latest/reference/cluster_manifest/#ebs-volume-resizing
  (I could potentially look at these if there is interest)

[jonhatallasf]

I second this feature request. Giving a pod full root privileges to the kubernetes node may be a security dealbreaker for my company to use this operator.

[Bryji]

@jonhatallasf - spilo_privileged does default to false, it can stay that way if you're able to forgo the dynamic PV resizing feature.

[jonhatallasf]

PV resizing feature is probably the biggest benefit we've identified of moving from stable helm postgres to operator

[frittentheke]

@jonhatallasf @Bryji may I kindly point you to my feature request / issue about letting CSI do the volume work and avoid all this: #460

To be honest it was necessary and quite cool to have the operator do this via direct calls to the AWS API and resize2fs. But with the availability of CSI including resizing and snapshotting the operator can be freed from this burden and be less "AWS-flavoured" in the process.

[Bryji]

Thanks @frittentheke . We deploy on multiple clouds, so a cloud-neutral implementation would be most welcome. So this feature is in alpha as of 1.15 and beta in 1.16 it seems? We're just moving our clusters over to 1.15 as of now. I don't know what the k8s version support policy is for the operator, maybe it could be implemented in a way that falls back to previous behaviour for earlier versions of k8s (I'm guessing many adopters could still be <1.15).

[frittentheke]

Certainly this switch to CSI would have to be backwards compatible for a few releases if it happens soon. But CSI is GA and moving forward very quickly and will replace all of the storage handling Kubernetes kubelet still does natively soon. And it takes some time to implement and merge the changes into the operator (provided they decide to re-implement volume resizing via CSI even).
But most importantly running privileged containers is not something that should be promoted / required any further without any (real) need for it.

My 2 cents: Older Kubernetes versions (as more than 3 releases behind the current stable) are not really something to worry about. It's just not the right kind of software stack if you are looking for LTS 😛

[jorgex1]

Hi all, are there any tips for this kind of error if setting spilo_privileged still doesn't get it to work?

[frittentheke]

You might want to checkout the just released version 1.6.0 (https://github.com/zalando/postgres-operator/releases/tag/v1.6.0) of the Operator which now can use CSI to resize volumes: #958

[FxKu]

Closing this one now, as with the new v1.6.0 release you can now use the K8s-native resize mode via PVCs.