The storage documentation is inaccurate

[pts-kevinqiu]

What did you do?
Creating a Prometheus object using kube-prometheus with the storage spec defined according to the documentation here.

Here's my final prometheus-k8s.yaml:

apiVersion: monitoring.coreos.com/v1alpha1
kind: Prometheus
metadata:
  name: k8s
  labels:
    prometheus: k8s
spec:
  replicas: 2
  version: v1.7.0
  serviceAccountName: prometheus-k8s
  serviceMonitorSelector:
    matchExpressions:
    - {key: k8s-app, operator: Exists}
  ruleSelector:
    matchLabels:
      role: prometheus-rulefiles
      prometheus: k8s
  resources:
    requests:
      # 2Gi is default, but won't schedule if you don't have a node with >2Gi
      # memory. Modify based on your target and time-series count for
      # production use. This value is mainly meant for demonstration/testing
      # purposes.
      memory: 400Mi
  storage:
    volumeClaimTemplate:
      metadata:
        annotations:
          name: prometheus
      spec:
        resources:
          requests:
            storage: 10Gi
  alerting:
    alertmanagers:
    - namespace: monitoring
      name: alertmanager-main
      port: web

Then I did kubectl apply -f prometheus-k8s.yaml -n monitoring.

What did you expect to see?

The prometheus TPR being created along with the prometheus pods.

What did you see instead? Under which circumstances?

The pods are not created. Investigating the prometheus-operator logs:

ts=2017-07-06T20:40:55Z caller=operator.go:972 component=prometheusoperator msg="updating config skipped, no configuration change"
E0706 20:40:55.293250       1 operator.go:526] Sync "monitoring/k8s" failed: syncing version failed: scaling in progress, 2 expected replicas, 0 found replicas

Didn't tell me much except that something happened that prevented the pods from being created.

Looking at the kube controller-manager logs:

I0706 20:44:53.055411       1 event.go:217] Event(v1.ObjectReference{Kind:"StatefulSet", Namespace:"monitoring", Name:"prometheus-k8s", UID:"47782512-628b-11e7-b968-02f07ab6cc05", APIVersion:"apps", ResourceVersion:"1126009", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' create Claim prometheus-k8s-db-prometheus-k8s-0 for Pod prometheus-k8s-0 in StatefulSet prometheus-k8s failed error: PersistentVolumeClaim "prometheus-k8s-db-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value

Apparently spec validation failed. resource[storage] is a required value even though the documentation didn't put there. The API spec indicate they're "DEPRECATED".

How can I use volume claim template without filling in the other irrelevant fields, such as class, selector, and resource?

[mindw]

This affects the alertmanager as well.

[fabxc]

For Prometheus this was fixed with #451 and was introduced with changes to how storage is declared.

The change was not done for Alertmanager yet and thus the problem shouldn't exist there. The code indicates that we are correctly setting the resources there too. @mindw can you elaborate what you observed?

[brancz]

The state of the master branch is only what will be released with the next release, which will deprecate the current storage declaration. Documentation on the current usage can be found on coreos.com.

[brancz]

Storage docs in master are now accurate for the latest release (v0.11.0), but have not landed on coreos.com yet.