Added emptyDirs for provisioning directories

[Bobbins228]

Description

Added emptyDirs to provisioning directories.

Relevant issues/tickets

#501

Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist

• This change requires a documentation update
• I have added tests that prove my fix is effective or that my feature works
• I have added a test case that will be used to verify my changes
• Verified independently on a cluster by reviewer

Verification steps

[NissesSenap]

Is there a risk that these folders grow in size? My guess is no but I don't know.
Recently I have hit some issues with emptyDir:s growing by allot, like 100Gb ^^
This created issues with evictions on other pods due to missing disk for them when they wanted to use like 5Mb or something.

Internally we have started to apply the following to our emptyDir configs.

      emptyDir:
        sizeLimit: 10Mi

Another option could of course be to tell the users that thinks it's a problem they can always use:
https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#setting-requests-and-limits-for-local-ephemeral-storage

spec.containers[].resources.limits.ephemeral-storage
spec.containers[].resources.requests.ephemeral-storage

Yeah that is probably better because then we don't have to think about how much grafana might use these emptyDir.

[Bobbins228]

@NissesSenap Are suggesting that we go with telling the users that they can use those ephemeral storage limits or somehow add these limits to the emptyDirs through the operator?

[NissesSenap]

Sorry for the confusion @Bobbins228, I did a few to many things at the same time yesterday so my entry wasn't that great.
I thought a bit more about it and I do think spec.containers[].resources.limits.ephemeral-storage would be the way to go but it shouldn't be included in this PR.
Sadly we don't support limits on ephemeral-storage currently but I will create a seperate ticket for it.

Overall I think this PR looks good, I will take a second check later today.

[NissesSenap]

Ran the code locally and can't see any of the errors at startup any more.
LGTM
Thanks @Bobbins228

[mcwienczek]

I think this functionality is missing in version 5 of the operator. Should I create a regression bug for this?

[NissesSenap]

Sounds reasonable. We will check it out