parameterize PVC storage class (ingester, querier, compactor)

[halcyondude]

What this PR does / why we need it:

Improve the initial UX for folks deploying Loki via ksonnet to clusters where VMWare Cloud is not being used.

• surface the StorageClass used for the Persistent Volume Claim(s) in loki/config.libsonnet when running the Ingester and/or Querier as a StatefulSet (instead of a Deployment)
• It leaves the default to 'fast' (see below) to not break any existing users/systems
• While this could be patched from a consumer (of loki's ksonnet), this is brittle. By parking it here it's immediately clear to newcomers that OOTB it's using a specific PVC StorageClass.
• The default/current behavior fails unless the VMware Cloud Provisioner is configured in a cluster, and will require folks to figure out why it does not work. Those folks will also have to figure out why changing the StorageClass to what they need then running tk apply don't work either, as the PVC needs to be deleted then re-added.

Background / Context

The current StorageClass is of type 'fast' and config like below will be generated.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ingester
  namespace: loki

...

spec:
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: ingester-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: fast

• https://kubernetes.io/docs/concepts/storage/storage-classes
• fast corresponds to the VMware Cloud Provider (vCP) StorageClass provisioner
• While k8s has a notion of a Default StorageClass, it relies on a cluster admin to configure the DefaultStorageClass admission plugin. This permits creating PVC's that don't specify a storage class (or use ""). More info/overview: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class-1

While making the default "" would also make sense, it would have the potential to break existing deployments (that use VMW Cloud) if the cluster admin's have not configured the DefaultStorageClass.

[CLAassistant]

All committers have signed the CLA.

[codecov-commenter]

Codecov Report

Merging #2718 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2718      +/-   ##
==========================================
- Coverage   61.46%   61.44%   -0.02%     
==========================================
  Files         173      173              
  Lines       13421    13421              
==========================================
- Hits         8249     8247       -2     
- Misses       4421     4422       +1     
- Partials      751      752       +1     

Impacted Files	Coverage Δ
pkg/logql/evaluator.go	92.38% <0.00%> (-0.43%)	⬇️

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[slim-bean]

LGTM, thanks!

[slim-bean]

@halcyondude not sure what happened to cause the conflict, if you have a chance could you see if you could rebase?

[cyriltovena]

indentation changed, I tried to push to the branch but I'm not allowed.

[halcyondude]

@halcyondude not sure what happened to cause the conflict, if you have a chance could you see if you could rebase?

done!