Add custom packs volumes

[cognifloyd]

New feature: Shared packs volumes st2.packs.volumes. Allow using cluster-specific persistent volumes to store packs, virtualenvs, and (optionally) configs. This enables using st2 pack install. It even works with st2packs images in st2.packs.images.

For a good description of the thinking behind this PR see: #18 (comment) (and the rest of the comments in that issue)

• add st2.packs.volumes section to values.yaml
• consolidate packs-volume-mounts into templates
• add st2.packs.volumes definitions to packs-volumes templates
• consolidate pack-configs-volume into templates
• add pack-configs-volume to more pods when st2.packs.volumes.enabled
• add error messages if st2.packs.volumes config is incomplete
• When st2.packs.volumes.enabled, make register-content packs-initContainer include system packs
• disable packs.configs ConfigMap if it is not needed/used
• add instructions for st2.packs.volumes values
• add st2.packs.volumes caveats similar to those documented in Allow mounting of NFS volumes instead of using st2packs for pack management #118
• allow using st2.packs.volumes with st2.packs.images
• Allow st2.packs.configs to work with st2.packs.volumes.configs
• add changelog entry

Related #160

Resolves #18
Closes #118

[ericreeves]

This looks pretty slick and flexible! I may start testing it pretty quickly, and will drop a note if I do!

[cognifloyd]

K. I think this is ready for review :) I look forward to feedback.

[cognifloyd]

I just reviewed #160, so I started puzzling over if we could also have the chart add the actual persistentVolumeClaims. But, once I read all the discussions in there, I think including the PVCs in the stackstorm-ha chart has some gotchas.

Quoting #160 (comment):

Is it possible for you to use StatefulSet with persistentVolumeTemplate instead?
Using what you've suggested deletes pvc when chart is deleted.
I tried using

annotations:
  helm.sh/resource-policy: keep

& it works fine until we reinstall the chart which fails with:

release stackstorm failed: persistentvolumeclaims "st2-packs-pvol-stackstorm" already exists

Or is there any way we can resuse PV with dynamic provisioning.

Maybe this is another reason to keep the PVC or other storage definition and management separate from the StackStorm chart. I wouldn't want any kind of helm maintenance (install, upgrade, delete, reinstall) to touch that storage, at least in the cluster I'm working on.

Right now I'm preparing to upgrade from a modified 1ppc installation, switching over to the stackstorm-ha chart. I will be reusing mongo, rabbitmq, redis, and my rook-ceph (via flexVolume) installations. So, I'll only be replacing the StackStorm deployments, services (plus migrating some configMaps and secrets to the stackstorm-ha format). I'm trying very hard to make this upgrade as seamless as possible.

That said, I'd be okay if others wanted the chart to optionally also create persistentVolumeClaims, even though I don't need or want them. Any thoughts?

[ericreeves]

I agree with you 100%. Storage for me will be managed outside of the stackstorm-ha chart. I'm using NFS (EFS) right now, but I'm considering switching to the aws-efs-csi driver presenting the EFS volume as a PVC which I will simply tell stackstorm-ha to use. This way, my storage and application tiers have a nice bit of separation, which feels like a clean demarcation point for management and provides greater levels of safety and control.

In the interest of simplicity, my vote is that stackstorm-ha should not create the PVC's, it should simply allow you to reference them as a volume source for the pods that require it. Creating the PVC's opens a giant can of worms and potential complexity that doesn't feel like it should be stackstorm-ha's job to deal with.

[cognifloyd]

rebased on master

[cognifloyd]

Thanks @ericreeves for the fixes in cognifloyd#1 and cognifloyd#2 . I've cherry-picked them in.

[ericreeves]

I have tested this PR in my cluster and it is working great. I think it's the best proposed solution for the shared storage issue so far.

[cognifloyd]

rebased on master + included nindent reformat changes

[ericreeves]

Still working great for me with all of the latest master changes merged in!

[cognifloyd]

I just came across an article that touches in how diverse the k8s storage landscape is.
https://blocksandfiles.com/2021/07/08/29444/

Something generic like this PR will be better for stackstorm-ha in the long run.

[ericreeves]

There have been a couple of attempts at resolving the need for shared storage here, and I believe this is by far the least prescriptive and most elegant.

BYOV. (Bring Your Own Volumes!)

[ericreeves]

Hopefully this gets merged, and then I am working on a way to allow this to peacefully coexist with st2.packs.images by only mounting the packs.images on the st2-register-content Job. This way you can still bake pack images and have them dumped on to the shared storage and registered automatically.

I have a POC of this already functioning in my fork, but want to wait until this gets merged to send a PR as not to muddy the goals of this PR.

[arm4b]

@cognifloyd Thanks for the Pull Request! 👍

I'm just gathering feedback from a few folks who were using custom volumes already if this can fit and work for everyone.

It looks optimistic so far.
Also thanks @ericreeves for trying this out already and help!

[cognifloyd]

@armab Sweet! I'm glad this is getting more eyes on it! Thanks for socializing this one!

[cognifloyd]

One issue I just ran into, is rebuilding all the virtualenvs after upgrading to 3.4 is more involved.
I ended up using the st2client pod and running this:

cd /opt/stackstorm/packs
/opt/stackstorm/st2/bin/st2-packs-setup-virtualenv *

I couldn't use st2ctl reload --register-recreate-virtualenvs as described in the docs because that doesn't work with the HA install.

I'm not sure if there's an elegant way to add a conditional upgrade job that runs when going from one st2 version to another.
But, that can be addressed in a separate issue/PR.

[cognifloyd]

Also, in our environment, we use the same pack repos across multiple environments (each env has its own set of volumes).
So, the packs do not have any of the rules enabled by default. We always enable the env-specific rules after updating packs with st2 pack install.
However, every helm upgrade runs st2-register-pack-content which disables all the rules again.

Forcing the installation of packs via st2packs images sidesteps this issue. Now that we're enabling packs volumes, more people will run into this.

The postStart hook works well for the system packs which shouldn't be replaced with st2 pack install, it won't however help in this case since packs are installed after the pods are already up and running.

This is the core issue about this: StackStorm/st2#3973

[chris3081]

Tested on AKS, we are one day in with light testing, using BYO PVCs and its happy so far. The testing continues.

[cognifloyd]

Thanks @chris3081! I'm glad it's working so far.

[arm4b]

Got more positive testing feedback about this feature 👍

@cognifloyd Could you please synchronize the https://docs.stackstorm.com/install/k8s_ha.html with the new stackstorm-ha Readme?

[arm4b]

Thanks everyone! 🎉

[cognifloyd]

Cool. I'll work on a docs PR this weekend. Thanks!

[cognifloyd]

Doc changes for this PR were just merged: StackStorm/st2docs#1089

[arm4b]

Thanks a lot @cognifloyd !