Add support for user-events monitoring

docs/configurationChoices.md

@@ -254,7 +254,9 @@ probes:
 
 ### Metrics and prometheus support
 
-OpenWhisk distinguishes between system and user metrics. System metrics typically contain information about system performance and use Kamon to collect. User metrics encompass information about action performance which is sent to Kafka in a form of events.
+OpenWhisk distinguishes between `system` and `user` metrics. System metrics typically contain information about system performance and use Kamon to collect. User metrics encompass information about action performance which is sent to Kafka in a form of events.
+
+#### System metrics
 
 If you want to collect system metrics, store and display them with prometheus, use below configuration in `mycluster.yaml`:
 
@@ -263,24 +265,34 @@ metrics:
   prometheusEnabled: true
 ```
 
-You also need to enable your prometheus to scrape those metrics, add below `scrape_configs` to your prometheus configuration file:
+This will automatically spin up a Prometheus server inside your cluster that will start scraping `controller` and `invoker` metrics.
+
+You can access Prometheus by using port forwarding:
 ```
-global:
-  scrape_interval: 1s
-scrape_configs:
-  - job_name: 'kamon-metrics'
-    static_configs:
-      - targets:['<controller_host>:8080','<invoker_host>:8080']
+kubectl port-forward svc/owdev-prometheus-server 9090:9090 --namespace openwhisk
 ```
-**Note:** replace `<controller_host>` and `<invoker_host>` with the real host name of controller and invoker.
 
-If you want to enable user metrics, use below configuration in `mycluster.yaml`:
+#### User metrics
+
+If you want to enable user metrics, use the below configuration in `mycluster.yaml`:
 
 ```
 metrics:
   userMetricsEnabled: true
 ```
 
+This will install [User-events](https://github.com/apache/openwhisk/tree/master/core/monitoring/user-events), [Prometheus](https://github.com/prometheus/prometheus) and [Grafana](https://github.com/grafana/grafana) on your cluster with already preconfigured Grafana dashboards for visualizing user generated metrics.
+
+The dashboards can be accessed here:
+```
+https://<whisk.ingress.apiHostName>:<whisk.ingress.apiHostPort>/monitoring/dashboards
+```
+All dashboards can be viewed anonymously and by default admin Grafana credentials are `admin/admin`. Use the bellow configuration in `mycluster.yaml` to change Grafana's admin password:
+```
+grafana:
+  adminPassword: admin
+```
+
 # Configure pod disruptions budget
 
 To avoid openwhisk components from [voluntary and nonvoluntary disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) which are managed by Kubernetes built-in controllers, you can configure PDB in `mycluster.yaml`.

helm/openwhisk/configMapFiles/prometheus/prometheus.yaml

@@ -0,0 +1,47 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# See https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml
+# For details on various options
+global:
+  scrape_interval: 10s
+scrape_configs:
+
+  - job_name: 'kubernetes-pods'
+    kubernetes_sd_configs:
+      - role: pod
+    relabel_configs:
+      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
+        action: keep
+        regex: true
+      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
+        action: replace
+        target_label: __metrics_path__
+        regex: (.+)
+      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
+        action: replace
+        regex: ([^:]+)(?::\d+)?;(\d+)
+        replacement: $1:$2
+        target_label: __address__
+      - action: labelmap
+        regex: __meta_kubernetes_pod_label_(.+)
+      - source_labels: [__meta_kubernetes_namespace]
+        action: replace
+        target_label: kubernetes_namespace
+      - source_labels: [__meta_kubernetes_pod_name]
+        action: replace
+        target_label: kubernetes_pod_name

helm/openwhisk/templates/_helpers.tpl

@@ -311,3 +311,13 @@ imagePullSecrets:
 -Dkubernetes.master=https://$KUBERNETES_SERVICE_HOST -Dwhisk.spi.ContainerFactoryProvider=org.apache.openwhisk.core.containerpool.kubernetes.KubernetesContainerFactoryProvider
 {{- end -}}
 {{- end -}}
+
+{{/* hostname for prometheus server */}}
+{{- define "openwhisk.prometheus_server_host" -}}
+{{ .Release.Name }}-prometheus-server.{{ .Release.Namespace }}.svc.{{ .Values.k8s.domain }}
+{{- end -}}
+
+{{/* hostname for grafana */}}
+{{- define "openwhisk.grafana_host" -}}
+{{ .Release.Name }}-grafana.{{ .Release.Namespace }}.svc.{{ .Values.k8s.domain }}
+{{- end -}}

helm/openwhisk/templates/controller-pod.yaml

@@ -34,6 +34,13 @@ spec:
       labels:
         name: {{ .Release.Name }}-controller
 {{ include "openwhisk.label_boilerplate" . | indent 8 }}
+
+      {{- if .Values.metrics.prometheusEnabled }}
+      annotations:
+        prometheus.io/scrape: 'true'
+        prometheus.io/port:   '{{ .Values.controller.port }}'
+      {{- end }}
+
     spec:
       serviceAccountName: {{ .Release.Name }}-core
       restartPolicy: {{ .Values.controller.restartPolicy }}

helm/openwhisk/templates/grafana-cm-dashboard.yaml

@@ -0,0 +1,37 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{ if or .Values.metrics.prometheusEnabled .Values.metrics.userMetricsEnabled }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ .Release.Name }}-grafana-dsb-cfg
+  labels:
+    name: {{ .Release.Name }}-grafana-dsb-cfg
+data:
+  dashboard.yml: |
+    apiVersion: 1
+    providers:
+    - name: 'Prometheus'
+      orgId: 1
+      folder: ''
+      type: file
+      disableDeletion: false
+      editable: true
+      options:
+        path: /var/lib/grafana/dashboards
+{{- end }}

helm/openwhisk/templates/grafana-cm-datasource.yaml

@@ -0,0 +1,33 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{ if or .Values.metrics.prometheusEnabled .Values.metrics.userMetricsEnabled }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ .Release.Name }}-grafana-ds-cfg
+  labels:
+    name: {{ .Release.Name }}-grafana-ds-cfg
+data:
+  datasource.yml: |
+    datasources:
+    - name: Prometheus
+      type: prometheus
+      access: proxy
+      isDefault: true
+      url: http://{{ include "openwhisk.prometheus_server_host" . }}:{{ .Values.prometheus.port }}
+{{- end }}

helm/openwhisk/templates/grafana-cm-download.yaml

@@ -0,0 +1,36 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{ if or .Values.metrics.prometheusEnabled .Values.metrics.userMetricsEnabled }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ .Release.Name }}-grafana-download
+  labels:
+    name: {{ .Release.Name }}-grafana-download
+data:
+  download_dashboards.sh: |
+    #!/usr/bin/env sh
+    {{- range $index, $url := .Values.grafana.dashboards }}
+      curl -sk \
+      --connect-timeout 60 \
+      --max-time 60 \
+      -H "Accept: application/json" \
+      -H "Content-Type: application/json;charset=UTF-8" \
+      {{ $url }}  > /var/lib/grafana/dashboards/{{ $index }}.json
+    {{- end }}
+{{- end }}

helm/openwhisk/templates/grafana-pod.yaml

@@ -0,0 +1,112 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{ if or .Values.metrics.prometheusEnabled .Values.metrics.userMetricsEnabled }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ .Release.Name }}-grafana
+  labels:
+    name: {{ .Release.Name }}-grafana
+{{ include "openwhisk.label_boilerplate" . | indent 4 }}
+spec:
+  serviceName: {{ .Release.Name }}-grafana
+  replicas: {{ .Values.grafana.replicaCount }}
+  selector:
+    matchLabels:
+      name: {{ .Release.Name }}-grafana
+  template:
+    metadata:
+      labels:
+        name: {{ .Release.Name }}-grafana
+{{ include "openwhisk.label_boilerplate" . | indent 8 }}
+    spec:
+      restartPolicy: {{ .Values.user_events.restartPolicy }}
+      {{- if .Values.affinity.enabled }}
+      affinity:
+{{ include "openwhisk.affinity.core" . | indent 8 }}
+{{ include "openwhisk.affinity.selfAntiAffinity" ( printf "%s-grafana" .Release.Name ) | indent 8 }}
+      {{- end }}
+
+      {{- if .Values.toleration.enabled }}
+      tolerations:
+{{ include "openwhisk.toleration.core" . | indent 8 }}
+      {{- end }}
+
+      initContainers:
+      - name: download-dashboards
+        image: "appropriate/curl:latest"
+        imagePullPolicy: Always
+        command: ["sh", "/etc/grafana/download_dashboards.sh"]
+        resources:
+          limits:
+            cpu: 100m
+            memory: 128Mi
+        volumeMounts:
+        - name: grafana-dashboard-volume
+          mountPath: /var/lib/grafana/dashboards
+        - name: grafana-download-volume
+          mountPath: "/etc/grafana"
+      containers:
+        - name: grafana
+          image: "{{- .Values.docker.registry.name -}}{{- .Values.grafana.imageName -}}:{{- .Values.grafana.imageTag -}}"
+          env:
+          - name: "GF_AUTH_ANONYMOUS_ENABLED"
+            value: "true"
+          - name: "GF_SECURITY_ADMIN_PASSWORD"
+            value: {{ .Values.grafana.adminPassword }}
+          - name: "GF_PATHS_PROVISIONING"
+            value: "/var/lib/grafana/config"
+          - name: "GF_USERS_ALLOW_SIGN_UP"
+            value: "false"
+          - name: "GF_SERVER_ROOT_URL"
+            value: "%(protocol)s://%(domain)s/monitoring"
+          - name: "GF_SERVER_SERVE_FROM_SUB_PATH"
+            value: "true"
+
+          ports:
+            - containerPort: {{ .Values.grafana.port }}
+          livenessProbe:
+            httpGet:
+              path: /api/health
+              port: {{ .Values.grafana.port }}
+            initialDelaySeconds: 30
+            timeoutSeconds: 5
+          readinessProbe:
+            httpGet:
+              path: /api/health
+              port: {{ .Values.grafana.port }}
+          volumeMounts:
+          - name: grafana-datasource-config-volume
+            mountPath: /var/lib/grafana/config/datasources
+          - name: grafana-dashboard-config-volume
+            mountPath: /var/lib/grafana/config/dashboards
+          - name: grafana-dashboard-volume
+            mountPath: /var/lib/grafana/dashboards
+      volumes:
+      - name: grafana-datasource-config-volume
+        configMap:
+          name: "{{ .Release.Name }}-grafana-ds-cfg"
+      - name: grafana-dashboard-config-volume
+        configMap:
+          name: "{{ .Release.Name }}-grafana-dsb-cfg"
+      - name: grafana-download-volume
+        configMap:
+          name: "{{ .Release.Name }}-grafana-download"
+      - name: grafana-dashboard-volume
+        emptyDir: {}
+{{- end }}

helm/openwhisk/templates/grafana-svc.yaml

@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+{{ if or .Values.metrics.prometheusEnabled .Values.metrics.userMetricsEnabled }}
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ .Release.Name }}-grafana
+  labels:
+    name: {{ .Release.Name }}-grafana
+spec:
+  selector:
+    name: {{ .Release.Name }}-grafana
+  ports:
+  - port: {{ .Values.grafana.port }}
+    name: http
+{{- end }}

helm/openwhisk/templates/invoker-pod.yaml

@@ -43,6 +43,13 @@ spec:
       labels:
         name: {{ .Release.Name }}-invoker
 {{ include "openwhisk.label_boilerplate" . | indent 8 }}
+
+      {{- if .Values.metrics.prometheusEnabled }}
+      annotations:
+        prometheus.io/scrape: 'true'
+        prometheus.io/port:   '{{ .Values.invoker.port }}'
+      {{- end }}
+
     spec:
 {{- if eq .Values.invoker.containerFactory.impl "kubernetes" }}
       serviceAccountName: {{ .Release.Name }}-invoker