FIX: Prevent repeated creation of unused Docker volumes

[petiepooo]

Running 'docker volume ls' on a node that has been running for some time shows many abandoned volumes, such as created by the so-zeek container. I believe this is due to the dockerfile Volume directive at https://github.com/Security-Onion-Solutions/securityonion-image/blob/70e29245f8b923fd03164ae3b0b80efa4eaff8fd/so-zeek/Dockerfile#L84. Run 'docker inspect so-zeek |grep -C2 _data' to see so-zeek's active volumes.

Of the local volumes listed there, "/nsm/zeek/logs" and "/nsm/zeek/spool" are bind-mounted from the host so their entry here has no effect, "/opt/zeek/share/bro" is not used (files are bound into /opt/zeek/share/zeek instead), and "/opt/zeek/etc/" contains some bound files but they can be bound into the ephemeral container instead of a local volume. If I understand it, you wouldn't need a volume directive in that dockerfile at all.

The so-strelka-gatekeeper, so-strelka-coordinator, and so-sensoroni containers also appear to create unused docker volumes on each run. so-redis stores its dumpfile in a local volume, but doesn't appear to reuse it on its next run. so-kratos (/kratos-conf volume) binds some host files into the volume, but doesn't use it otherwise.

These abandoned docker volumes are not removed and accumulate over time, cluttering the volume list and taking up hundreds of MB on the root volume after a few months. Run 'sudo du -hs /var/lib/docker/volumes' to see how much space is used.

[petiepooo]

It looks like so-zeek has been updated in 2.3.230, but the problem still exists in other docker images.

The problem:
I believe something in kratos's init.sls is triggering a "volume create/container create/container destroy" event sequence each time highstate is applied. This seems to be because the /opt/so/conf/kratos directory to which files are written is bound to /kratos-conf within the running so-kratos container, so salt creates a temporary container to mount the volume and check something. But, because the /kratos-data volume is not bound anywhere, it creates a new docker volume for it each time a container based on the so-kratos image is created.

Between kratos, soc, and (all 3) redis containers, there are five unused docker volumes created at every call to highstate. That's 20/hour, 480/day, or 14,400/month!

The solution:
Remove the VOLUME directive from all of your images' Dockerfiles, including so-kratos and so-soc. None of those directories actually require a docker data volume. If a directory is to be bind-mounted to the host filesystem, it need not also be a volume, but if you do have a volume specified in the Dockerfile, it should be bound to something when run.

If you're using the generic redis:6-alpine docker image as I believe you are, you'll need to bind-mount a directory from the host to override the volume-mount for each of the images that use it as a base, including so-redis, so-strelka-gatekeeper and so-strelka-coordinator. That or build a custom redis image without it or actually use a persistent named docker volume in each.

How to debug:
To watch the docker events, run the command stdbuf -oL docker events --since $(date +%Y-%M-%dT%H:%M) | grep -A3 -i "volume create" and apply highstate. You'll see a "create volume" event followed by a "create container" event showing which container required the volume.

How to verify:
Ultimately, when all images are fixed and stale volumes are purged with docker volume purge -f, no new volumes should appear on boot or on highstate.

Dev Tip:
Please consider building a VM to run a stable, long-running system with limited disk space that runs your apps and processes traffic continuously. Upgrade it on each release, but only after a release is actually published. Periodically clone it so you have examples of systems at various releases and ages. Do not reinstall the system once it is in production so you can see how things accumulate over time. Such a setup would help you catch this issue as well as see that old zeek logs, processed strelka files, and completed pcap jobs are also never cleaned up...

[petiepooo]

mentioned in #11072

[petiepooo]

Thank you, @jertel, for fixing the soc and kratos images. I think a couple more file edits are needed.

The stock redis:6-alpine image used by so-redis also has a VOLUME specified at https://github.com/docker-library/redis/blob/f2da8752a05b783eb805b67ad7a56a997a0fe91f/6.2/alpine/Dockerfile#L95

so this container:

securityonion/salt/strelka/init.sls

Line 199 in 3e5f354

- image: {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-redis:{{ VERSION }}

this container:

securityonion/salt/strelka/init.sls

Line 212 in 3e5f354

- image: {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-redis:{{ VERSION }}

and this container:

securityonion/salt/redis/init.sls

Line 57 in 3e5f354

- image: {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-redis:{{ VERSION }}

will also need a bind entry to mount /data to some directory on the host in order to keep these containers from creating a docker volume on each instantiation.

[petiepooo]

I see those are fixed now in f9cbde1 and 66bb127. Thank you.