Increase MongoDB Datastore volume size

[christian-kreuzberger-dtx]

Right now our MongoDB deployment used by the mongodb-datastore uses a volume with 1 GB space. This is sufficient for demos and workshops, but for long-term usage we should increase this volume.

The deployment is configured in the installer manifests here: https://github.com/keptn/keptn/tree/master/installer/manifests/logging/mongodb

---

During testing with 0.6.2 I found that we can add a lot of events before we reach the 1 GB limit.
I was spamming a start-evaluation event per second (which results in four more events) for over 3 hours, and this resulted in 24 Megabytes of additional storage used for MongoDB.

IMHO we should increase the limit of the volume to 5 GB to be on the safe side for a long time.

[jetzlstorfer]

I agree with you that it should be production ready. If that means increasing the storage already in the beginning, I would go for it.

[checkelmann]

Perhaps it would be a good idea to make this configurable by the installer command.
Like keptn install --datastore-size=5Gi and have de suitable default value set if not defined. Within the documentation should be a sample of how many space an event will utilize as @christian-kreuzberger-dtx said. X Events ~ 24MB, so the user could project how much space he needs to configure.

What could be documented is how to expand a PVC if space is used completely (which is IMHO a K8s Administrative Task and should not be part of a keptn command).

Like, adjust the storage size in the PVC, if a block device like CEPH, AWS EBS is used, restart the pods afterward.

kubectl -n keptn-datastore edit pvc mongodata

spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard
  volumeMode: Filesystem

[christian-kreuzberger-dtx]

Problem with editing the PVC is that we might overwrite the PVC when upgrading Keptn. Any thoughts on this @bacherfl ?

[christian-kreuzberger-dtx]

We just discussed this:

Configuring the size via the installer is something that might be possible in the future.
for now we will increase the size to 5 GB to stay on the safe side.

[jetzlstorfer]

@zohaib85 volunteered to take a look at it. Couldn't assign him to the issue, thus I'm mentioning this here.