PVC and PV are not re-used when redeploying/updating MM

[pat-s]

Chart version: 5.4.0
K8s version: 1.21.4

When redeploying or updating, the helm chart creates a new PV and a new PVC claim instead of re-using the old ones.

pvc-00563e91-ddde-40f7-9f14-d394324ec8b3   10Gi       RWO            Retain           Released   mattermost/mattermost-mattermost-team-edition           ebs-sc                  9d
pvc-1580f883-c152-410e-9b1c-dbe5faae1082   10Gi       RWO            Retain           Bound      mattermost/mattermost-mattermost-team-edition           ebs-sc                  15m

kubectl get pv

pvc-1580f883-c152-410e-9b1c-dbe5faae1082   10Gi       RWO            Retain           Released   mattermost/mattermost-mattermost-team-edition           ebs-sc                  112m

A workaround is to set persistence.data.existingClaim referencing a PVC name which claims an existing PV which contains the data to fix that.
However this requires to have existing PV and PVC and I assume this won't work on the first deployment in which both are not yet existent.

Also this looks like a bug to me as I'd expect the chart to re-use the previous PVC.

More specifically, I am using the following TF setup now which works across updates

  set {
    name  = "persistence.data.existingClaim"
    value = "pvc-mattermost-data"
  }

resource "kubernetes_persistent_volume_claim" "pvc-mattermost-data" {
  metadata {
    name      = "pvc-mattermost-data"
    namespace = "mattermost"
  }
  spec {
    access_modes = ["ReadWriteOnce"]
    resources {
      requests = {
        storage = "10Gi"
      }
    }
    volume_name        = "pvc-00563e91-ddde-40f7-9f14-d394324ec8b3"
    storage_class_name = "ebs-sc"
  }
}

The same applies also to the PVC for plugins.

This might also relate to #200 and #251.

[lfvjimisola]

Just to confirm, if you use the helm chart to upgrade your Mattermost instance then all data is lost (configuration, messages, uploaded files etc)?

[pat-s]

Messages are stored in the DB and the config is living in a values.yml file.

Only files are/were lost for me.

[cpanato]

do you have a step to reproduce the issue?

[pat-s]

do you have a step to reproduce the issue?

Simply changing the image tag and redeploying should suffice. At least this is what caused this twice in our instance.
We have a terraform-backed deployment but happy to share the code if it helps.

But in general everything that forces a pod recreation should be able to trigger this as the PVC were not re-used for new pods.

[cpanato]

i will try to reproduce the issue, just one final question, which chart are you using? the team edition or the enterprise one? @pat-s

[pat-s]

Thanks. Team edition.

[cpanato]

@pat-s
I've set up a local and a remote cluster (EKS) and deployed the chart with the persistence and then made an upgrade and the volumes were not deleted.

my data and the custom installed plugin were in place.
Maybe is something together with terraform which I'm not sure.

[cpanato]

pressed the wrong button to comment.

[pat-s]

then made an upgrade and the volumes were not deleted.

They were not deleted on my side, just not re-used. I.e. the chart created new PV and PVCs, which had no content then.

I can try to replicate the behaviour again if needed on our dev cluster.
I don't think TF should have an influence here, in the end the helm_release module also just applies the helm chart with the values.yml specifications.

[danielsand]

I don't think TF should have an influence here, in the end the helm_release module also just applies the helm chart with the values.yml specifications.

i ran into a different issue - just some experience from that.

helm upgrade reused every time the PVC volumes from before - so all is working as expected from the helm chart and helm

terraforms helm provider seems to be the issue on your case @pat-s

I bumped into the error after cancelling a previous run (the cloud provider stack on fulfilling a PVC request). That left some orphan resources (svc,cm,sa,...), after deleting them I could re-run the apply just fine.

Originally posted by @marpada in hashicorp/terraform-provider-helm#425 (comment)

hashicorp/terraform-provider-helm#592

[pat-s]

I am still hitting this issue and can still reproduce it (using TF + helm). It does not necessarily happen when updating the chart but when (for some reason) removing the deployment and creating a new deployment (with the same settings).
In this case, the chart will create a new PV and PVC and not reuse the existing ones.

This bugs me quite a bit and this helm chart is the only one within ~ 20 I am maintaining across different clusters which has this issue.
Anyhow, I don't have enought time to inspect this in more detail, I am more interested in a working and robust solution.
Over the last years I've lost the plugins dir at least 4 times due to this behavior :(

The only workable solution seems to be to not let the chart creat the PVC but use terraform and reference to an existingClaim in the helm chart.

In the MM helm chart I reference an existing PVC like so:

persistence:
  # location: /mattermost/data
  data:
    enabled: true
    size: 10Gi
    storageClass: ebs-sc
    accessMode: ReadWriteOnce
    existingClaim: "mattermost-team-edition"
  # location: /mattermost/client/plugins
  plugins:
    enabled: true
    size: 1Gi
    ##
    storageClass: ebs-sc
    accessMode: ReadWriteOnce
    existingClaim: "mattermost-team-edition-plugins"

The PVC and corresponding PV are created (or in my case imported) by terraform.
If you have trouble importing, have a look at this hashicorp guide.

resource "kubernetes_persistent_volume_claim" "mattermost-team-edition-plugins" {
  metadata {
    name      = "mattermost-team-edition-plugins"
    namespace = "mattermost"
  }
  spec {
    access_modes       = ["ReadWriteOnce"]
    storage_class_name = "ebs-sc"
    resources {
      requests = {
        storage = "1Gi"
      }
    }
    volume_name = "pvc-42ffb542-9abf-4a30-80e1-af8ecd98ca05"
  }
}

resource "kubernetes_persistent_volume_claim" "mattermost-team-edition" {
  metadata {
    name      = "mattermost-team-edition"
    namespace = "mattermost"
  }
  spec {
    access_modes       = ["ReadWriteOnce"]
    storage_class_name = "ebs-sc"
    resources {
      requests = {
        storage = "10Gi"
      }
    }
    volume_name = "pvc-4fe296f3-cef0-4e7a-b366-9217f19afe6b"
  }
}

resource "kubernetes_persistent_volume" "pvc-42ffb542-9abf-4a30-80e1-af8ecd98ca05" {

  metadata {
    name = "pvc-42ffb542-9abf-4a30-80e1-af8ecd98ca05"
  }

  spec {
    capacity = {
      storage = "1Gi"
    }
    access_modes                     = ["ReadWriteOnce"]
    persistent_volume_reclaim_policy = "Retain"
    storage_class_name               = "ebs-sc"
    persistent_volume_source {
      csi {
        driver        = "ebs.csi.aws.com"
        fs_type       = "ext4"
        read_only     = false
        volume_handle = "vol-0a2bcb6a8c26cee79"
        volume_attributes = {
          "storage.kubernetes.io/csiProvisionerIdentity" : "1657554492068-8081-ebs.csi.aws.com"
        }
      }
    }
  }
}

resource "kubernetes_persistent_volume" "pvc-4fe296f3-cef0-4e7a-b366-9217f19afe6b" {

  metadata {
    name = "pvc-4fe296f3-cef0-4e7a-b366-9217f19afe6b"
  }

  spec {
    capacity = {
      storage = "10Gi"
    }
    access_modes                     = ["ReadWriteOnce"]
    persistent_volume_reclaim_policy = "Retain"
    storage_class_name               = "ebs-sc"
    persistent_volume_source {
      csi {
        driver        = "ebs.csi.aws.com"
        fs_type       = "ext4"
        read_only     = false
        volume_handle = "vol-05e73ef89055b933c"
        volume_attributes = {
          "storage.kubernetes.io/csiProvisionerIdentity" : "1657554492068-8081-ebs.csi.aws.com"
        }
      }
    }
  }
}