operator-framework · varshaprasad96 · Jun 1, 2020 · May 20, 2020 · ecordell · May 29, 2020
diff --git a/enhancements/sdk-metrics.md b/enhancements/sdk-metrics.md
@@ -0,0 +1,179 @@
+---
+title: sdk-metrics
+  - "@varshaprasad96"
+reviewers:
+  - "@dmesser"
+  - "@joelanford"
+approvers:
+  - "@dmesser" 
+  - "@joelanford"
+creation-date: 2020-05-19
+last-updated: 2020-05-19
+status: implementable
+---
+
+# Operator SDK metrics
+
+## Release Signoff Checklist
+
+- [X] Enhancement is `implementable`
+- [ ] Design details are appropriately documented from clear requirements
+- [ ] Test plan is defined
+- [ ] Graduation criteria for dev preview, tech preview, GA
+
+## Summary
+
+The enhancement is intended to define the bundle resources in which SDK labels can be added, so that they can be read by OLM and included in the current OLM metrics. These metrics will help us in providing a measure to evaluate the reachability of Operator SDK among community operators.
+
+## Motivation
+
+Collecting metrics regarding the usage of Operator SDK among operator authors is important to measure the reachability of the tool. Since OLM already has a pipeline built to obtain metrics on operators and report it to the RedHat telemetry, we intend to utilize the same to obtain data on operators using SDK to build bundle images.
+
+### Goals
+
+The goal of this enhancement is to :
+1. List the bundle resources on which SDK stamp will be added.
+2. The format of the stamp which will be added.
+
+### Design details
+
+The current bundle image has the following project structure:
+
+```
+ ...
+  bundle.Dockerfile
+  ...
+  $ tree ./deploy/olm-catalog/test-operator/
+  ./deploy/olm-catalog/test-operator/
+  ├── manifests
+  │   ├── example.com_tests_crd.yaml
+  │   └── test-operator.clusterserviceversion.yaml
+  └── metadata
+      └── annotations.yaml
+```
+The proposed SDK stamps will contain data related to:
+* Version of SDK used
+* SDK project layout
+
+The stamps would be present in the following locations:
+
+1. bundle.Dockerfile
+
+The dockerfile will include stamps in the form of `LABEL` which will follow the following format:
+
+```
+LABEL operators.operatorframework.io.metrics.mediatype.v1 = “metrics+v1”
+LABEL operators.operatorframework.io.metrics.builder = “operator-sdk-v0.17.0”
+LABEL operators.operatorframework.io.metrics.project_layout = “sdk_project_layout/layout_version”
+```
+
+2. annotations.yaml
+
+Similar to the format followed in dockerfile, `annotations.yaml` will have the following labels:
+
+```
+operators.operatorframework.io.metrics.mediatype.v1 = “metrics+v1”
+operators.operatorframework.io.metrics.builder = “operator-sdk-v0.17.0”
+operators.operatorframework.io.metrics.project_layout = “sdk_project_layout/layout_version”
+```
+
+For example, in case of memcached-operator having Kubebuilder layout, the labels can be written as:
+
+```
+LABEL operators.operatorframework.io.metrics.mediatype.v1 = “metrics+v1”
+LABEL operators.operatorframework.io.metrics.builder = “operator-sdk-v0.17.0”
+LABEL operators.operatorframework.io.metrics.project_layout = “go.kubebuilder.io/v2.0.0”
+```
+
+Having labels on `bundle.dockerfile` and `annotations.yaml` will help in collecting metrics regarding the use of SDK from indexed operator bundle images.
+
+**Note**: The labels are designed to be generic, and are intended to follow the prometheus label format, after the prefix - `operators.operatorframework.io.metrics`, so that mapping them with metric labels would be clearer and easier. This format can also be followed by operator owners who do not use SDK, but would like to include any other custom stamp values.
+
+3. Cluster service version
+
+The cluster service version generated by SDK using `operator-sdk generate csv` command can have a similar stamp included in the metadata. It will be present as an [annotation][kubernetes_annotation]. For example, while generating a new CSV, the following additional information will be present in Object metadata:
+
+```Go
+func newCSV(name, version string) (*olmapiv1alpha1.ClusterServiceVersion, error) {
+    csv := &olmapiv1alpha1.ClusterServiceVersion{
+      ObjectMeta: metav1.ObjectMeta{
+        Name:      getCSVName(name, version),
+        Namespace: "placeholder",
+        Annotations: map[string]string{
+          "capabilities": "Basic Install",
+          "operators.operatorframework.io/builder": "operator-sdk-v0.17.0",
+          "operators.operatorframework.io/project_layout": "go.kubebuilder.io/v2.0.0",
+		}
+    }
+    ...
+  }
+}
+```
+
+The corresponding CSV will appear to be:
+
+```yaml
+apiVersion: operators.coreos.com/v1alpha1
+kind: ClusterServiceVersion
+metadata:
+  annotations:
+    capabilities: Basic Install
+    operators.operatorframework.io/builder: "operator-sdk-v0.17.0"
+    operators.operatorframework.io/project_layout: "go.kubebuilder.io/v2.0.0"
+...
+```
+
+In case of legacy projects, which do not use Kubebuilder layout, the `operators.operatorframework.io/project_layout` annotation will only represent the kind of operator (go/ansible/helm). In those cases, the annotations would look like:
+
+```Go
+Annotations: map[string]string{
+  "capabilities": "Basic Install",
+  "operators.operatorframework.io/builder": "operator-sdk-v0.15.0",
+  "operators.operatorframework.io/project_layout": "go",
+},
+```
+
+4. Custom Resource Definition
+
+As OLM will be moving towards having CSV-less bundles, generation of CSV would not be performed by SDK. Injecting stamps on CRDs would be helpful to propagate metadata regarding SDK through bundles.
+
+For operator SDK projects after Kubebuilder integration, CRDs will have the annotations implemented using Kustomize patches and for legacy project layouts it will be injected through the existing built-in CRD generation. In both the cases, the CRDs would appear to look like:
+
+```yaml
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  name: memcacheds.cache.example.com
+    annotations:
+      operators.operatorframework.io/builder: "operator-sdk-v0.17.0"
+      operators.operatorframework.io/project_layout: "go.kubebuilder.io/v2.0.0"
+```
+
+#### Integration of SDK stamps with OLM metrics
+
+The main aim of adding SDK stamps to bundle images and CSV is to embed SDK related data with OLM metrics. Currently, OLM tracks the lifecycle of CSV and reports metrics related to its success or failure. The first step, would be to include SDK metadata in those metrics by adding labels. The two relevant metrics where we can add labels currently are - `csvSucceeded` and `csvAbnormal`.
+
+For example, `csvSucceeded` metric would be modified to be:
+
+```Go
+	csvSucceeded = prometheus.NewGaugeVec(
+		prometheus.GaugeOpts{
+			Name: "csv_succeeded",
+			Help: "Successful CSV install",
+		},
+		[]string{NAMESPACE_LABEL, NAME_LABEL, VERSION_LABEL, BUILDER, PROJECT_LAYOUT},
+	)
+```
+
+Here, the value of `BUILDER` would be taken from `operators.operatorframework.io/builder` and `PROJECT_LAYOUT` would represent the value of `operators.operatorframework.io/project_layout` from CSV annotations.
+
+In future, additional labels or metrics can be added based on requirements.
+
+### Limitations
+
+1. The metrics which would be collected through this process will not cover the audience which uses Operator SDK but not OLM to deploy and run their operators.
+2. Currently, `operator-sdk generate csv` officially supports only Go Operators. Ansible and Helm Operators will be fully supported in future. Hence, the collected metrics would majorly cover only Go operators.
+3. Implementation limitation - As operator registry does not have an api to modify the bundle dockerfiles or manifests, the implementation for addition of labels will involve parsing of the relevant files and rewriting them to include the mentioned labels. 
+
+
+[kubernetes_annotation]: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/