Longhorn Storage - right sizing and monitoring

[johnml1135]

Disk usage can explode when there are many SMT engines being made and not deleted: Disk usage: min(kubelet_volume_stats_used_bytes{namespace=~"nlp|serval"}) by (persistentvolumeclaim, namespace)/ on (persistentvolumeclaim, namespace) min(kube_persistentvolumeclaim_resource_requests_storage_bytes{namespace=~"nlp|serval"}) by (persistentvolumeclaim, namespace) * 100

The following was from some load testing - that did have an out-of-disk space error. We need to to both right-size the longhorn storage based upon expected usage and monitor it's growth to not crash the system.

[johnml1135]

If there are issues, we can make them bigger.