Proposal: Introduce Custom Metrics API

This proposal details the custom metrics API as proposed in the new
monitoring vision. It is designed for use with the HPA, but should be
generally useful for controllers that wish to consumer custom metrics.

Author: DirectXMan12

contributors/design-proposals/custom-metrics-api.md

@@ -0,0 +1,325 @@
+Custom Metrics API
+==================
+
+The new [metrics monitoring vision](monitoring_architecture.md) proposes
+an API that the Horizontal Pod Autoscaler can use to access arbitrary
+metrics.
+
+Similarly to the [master metrics API](resource-metrics-api.md), the new
+API should be structured around accessing metrics by referring to
+Kubernetes objects (or groups thereof) and a metric name.  For this
+reason, the API could be useful for other consumers (most likely
+controllers) that want to consume custom metrics (similarly to how the
+master metrics API is generally useful to multiple cluster components).
+
+The HPA can refer to metrics describing all pods matching a label
+selector, as well as an arbitrary named object.
+
+API Paths
+---------
+
+The root API path will look like `/apis/custom-metrics/v1alpha1`.  For
+brevity, this will be left off below.
+
+- `/{object-type}/{object-name}/{metric-name...}`:  retrieve the given
+  metric for the given non-namespaced object (e.g. Node, PersistentVolume)
+
+- `/{object-type}/*/{metric-name...}`: retrieve the given metric for all
+  non-namespaced objects of the given type
+
+- `/{object-type}/*/{metric-name...}?labelSelector=foo`: retrieve the
+  given metric for all non-namespaced objects of the given type matching
+  the given label selector
+
+- `/namespaces/{namespace-name}/{object-type}/{object-name}/{metric-name...}`:
+  retrieve the given metric for the given namespaced object
+
+- `/namespaces/{namespace-name}/{object-type}/*/{metric-name...}`: retrieve the given metric for all
+  namespaced objects of the given type
+
+- `/namespaces/{namespace-name}/{object-type}/*/{metric-name...}?labelSelector=foo`: retrieve the given
+  metric for all namespaced objects of the given type matching the
+  given label selector
+
+- `/namespaces/{namespace-name}/metrics/{metric-name}`: retrieve the given
+  metric which describes the given namespace.
+
+For example, to retrieve the custom metric "hits-per-second" for all
+ingress objects matching "app=frontend` in the namespaces "webapp", the
+request might look like:
+
+```
+GET /apis/custom-metrics/v1alpha1/namespaces/webapp/ingress.extensions/*/hits-per-second?labelSelector=app%3Dfrontend`
+
+---
+
+Verb: GET
+Namespace: webapp
+APIGroup: custom-metrics
+APIVersion: v1alpha1
+Resource: ingress.extensions
+Subresource: hits-per-second
+Name: ResourceAll(*)
+```
+
+Notice that getting metrics which describe a namespace follows a slightly
+different pattern from other resources; Since namespaces cannot feasibly
+have unbounded subresource names (due to collision with resource names,
+etc), we introduce a pseudo-resource named "metrics", which represents
+metrics describing namespaces, where the resource name is the metric name:
+
+```
+GET /apis/custom-metrics/v1alpha1/namespaces/webapp/metrics/queue-length
+
+---
+
+Verb: GET
+Namespace: webapp
+APIGroup: custom-metrics
+APIVersion: v1alpha1
+Resource: metrics
+Name: queue-length
+```
+
+API Path Design, Discovery, and Authorization
+---------------------------------------------
+
+The API paths in this proposal are designed to a) resemble normal
+Kubernetes APIs, b) facilitate writing authorization rules, and c)
+allow for discovery.
+
+Since the API structure follows the same structure as other Kubernetes
+APIs, it allows for fine grained control over access to metrics.  Access
+can be controlled on a per-metric basic (each metric is a subresource, so
+metrics may be whitelisted by allowing access to a particular
+resource-subresource pair), or granted in general for a namespace (by
+allowing access to any resource in the `custom-metrics` API group).
+
+Similarly, since metrics are simply subresources, a normal Kubernetes API
+discovery document can be published by the adapter's API server, allowing
+clients to discover the available metrics.
+
+Note that we introduce the syntax of having a name of ` * ` here since
+there is no current syntax for getting the output of a subresource on
+multiple objects.
+
+API Objects
+-----------
+
+The request URLs listed above will return the `MetricValueList` type described
+below (when a name is given that is not ` * `, the API should simply return a
+list with a single element):
+
+```go
+
+// a list of values for a given metric for some set of objects
+type MetricValueList struct {
+    metav1.TypeMeta`json:",inline"`
+    metav1.ListMeta`json:"metadata,omitempty"`
+
+    // the value of the metric across the described objects
+    Items []MetricValue `json:"items"`
+}
+
+// a metric value for some object
+type MetricValue struct {
+    metav1.TypeMeta`json:",inline"`
+
+    // a reference to the described object
+    DescribedObject ObjectReference `json:"describedObject"`
+
+    // the name of the metric
+    MetricName string `json:"metricName"`
+
+    // indicates the time at which the metrics were produced
+    Timestamp unversioned.Time `json:"timestamp"`
+
+    // indicates the window ([Timestamp-Window, Timestamp]) from
+    // which these metrics were calculated, when returning rate
+    // metrics calculated from cumulative metrics (or zero for
+    // non-calculated instantaneous metrics).
+    WindowSeconds int64 `json:"window"`
+
+    // the value of the metric for this
+    Value resource.Quantity
+}
+```
+
+For instance, the example request above would yield the following object:
+
+```json
+{
+    "kind": "MetricValueList",
+    "apiVersion": "custom-metrics/v1alpha1",
+    "items": [
+        {
+            "metricName": "hits-per-second",
+            "describedObject": {
+                "kind": "Ingress",
+                "apiVersion": "extensions",
+                "name": "server1",
+                "namespace": "webapp"
+            },
+            "timestamp": SOME_TIMESTAMP_HERE,
+            "windowSeconds": "10",
+            "value": "10"
+        },
+        {
+            "metricName": "hits-per-second",
+            "describedObject": {
+                "kind": "Ingress",
+                "apiVersion": "extensions",
+                "name": "server2",
+                "namespace": "webapp"
+            },
+            "timestamp": ANOTHER_TIMESTAMP_HERE,
+            "windowSeconds": "10",
+            "value": "15"
+        }
+    ]
+}
+```
+
+Semantics
+---------
+
+### Object Types ###
+
+In order to properly identify resources, we must use resource names
+qualified with group names (since the group for the requests will always
+be `custom-metrics`).
+
+The `object-type` parameter should be the string form of
+`unversioned.GroupResource`.  Note that we do not include version in this;
+we simply wish to uniquely identify all the different types of objects in
+Kubernetes.  For example, the pods resource (which exists in the un-named
+legacy API group) would be represented simply as `pods`, while the jobs
+resource (which exists in the `batch` API group) would be represented as
+`jobs.batch`.
+
+In the case of cross-group object renames, the adapter should maintain
+a list of "equivalent versions" that the monitoring system uses. This is
+monitoring-system dependent (for instance, the monitoring system might
+record all HorizontalPodAutoscalers as in `autoscaling`, but should be
+aware that HorizontalPodAutoscaler also exist in `extensions`).
+
+Note that for namespace metrics, we use a pseudo-resource called
+`metrics`.  Since there is no resource in the legacy API group, this will
+not clash with any existing resources.
+
+### Metric Names ###
+
+Metric names must be able to appear as a single subresource.  In particular,
+metric names, *as passed to the API*, may not contain the characters '%', '/',
+or '?', and may not be named '.' or '..' (but may contain these sequences).
+Note, specifically, that URL encoding is not acceptable to escape the forbidden
+characters, due to issues in the Go URL handling libraries. Otherwise, metric
+names are open-ended.
+
+### Metric Values and Timing ###
+
+There should be only one metric value per object requested.  The returned
+metrics should be the most recently available metrics, as with the resource
+metrics API.  Implementers *should* attempt to return all metrics with roughly
+identical timestamps and windows (when appropriate), but consumers should also
+verify that any differences in timestamps are within tolerances for
+a particular application (e.g. a dashboard might simply display the older
+metric with a note, while the horizontal pod autoscaler controller might choose
+to pretend it did not receive that metric value).
+
+### Labeled Metrics (or lack thereof) ###
+
+For metrics systems that support differentiating metrics beyond the
+Kubernetes object hierarchy (such as using additional labels), the metrics
+systems should have a metric which represents all such series aggregated
+together. Additionally, implementors may choose to identify the individual
+"sub-metrics" via the metric name, but this is expected to be fairly rare,
+since it most likely requires specific knowledge of individual metrics.
+For instance, suppose we record filesystem usage by filesystem inside the
+container. There should then be a metric `filesystem/usage`, and the
+implementors of the API may choose to expose more detailed metrics like
+`filesystem/usage/my-first-filesystem`.
+
+### Resource Versions ###
+
+API implementors should set the `resourceVersion` field based on the
+scrape time of the metric.  The resource version is expected to increment
+when the scrape/collection time of the returned metric changes.  While the
+API does not support writes, and does not currently support watches,
+populating resource version preserves the normal expected Kubernetes API
+semantics.
+
+Relationship to HPA v2
+----------------------
+
+The URL paths in this API are designed to correspond to different source
+types in the [HPA v2](hpa-v2.md).  Specifially, the `pods` source type
+corresponds to a URL of the form
+`/namespaces/$NS/pods/*/$METRIC_NAME?labelSelector=foo`, while the
+`object` source type corresponds to a URL of the form
+`/namespaces/$NS/$RESOURCE.$GROUP/$OBJECT_NAME/$METRIC_NAME`.
+
+The HPA then takes the results, aggregates them together (in the case of
+the former source type), and uses the resulting value to produce a usage
+ratio.
+
+The resource source type is taken from the API provided by the
+"metrics" API group (the master/resource metrics API).
+
+The HPA will consume the API as a federated API server.
+
+Relationship to Resource Metrics API
+------------------------------------
+
+The metrics presented by this API may be a superset of those present in the
+resource metrics API, but this is not guaranteed.  Clients that need the
+information in the resource metrics API should use that to retrieve those
+metrics, and supplement those metrics with this API.
+
+Mechanical Concerns
+-------------------
+
+This API is intended to be implemented by monitoring pipelines (e.g.
+inside Heapster, or as an adapter on top of a solution like Prometheus).
+It shares many mechanical requirements with normal Kubernetes APIs, such
+as the need to support encoding different versions of objects in both JSON
+and protobuf, as well as acting as a discoverable API server.  For these
+reasons, it is expected that implemenators will make use of the Kubernetes
+genericapiserver code.  If implementors choose not to use this, they must
+still follow all of the Kubernetes API server conventions in order to work
+properly with consumers of the API.
+
+Specifically, they must support the semantics of the GET verb in
+Kubernetes, including outputting in different API versions and formats as
+requested by the client.  They must support integrating with API discovery
+(including publishing a discovery document, etc).
+
+Location
+--------
+
+The types and clients for this API will live in a separate repository
+under the Kubernetes organization (e.g. `kubernetes/metrics`).  This
+repository will most likely also house other metrics-related APIs for
+Kubernetes (e.g. historical metrics API definitions, the resource metrics
+API definitions, etc).
+
+Note that there will not be a canonical implemenation of the custom
+metrics API under Kubernetes, just the types and clients.  Implementations
+will be left up to the monitoring pipelines.
+
+Alternative Considerations
+--------------------------
+
+### Quantity vs Float ###
+
+In the past, custom metrics were represented as floats.  In general,
+however, Kubernetes APIs are not supposed to use floats. The API proposed
+above thus uses `resource.Quantity`.  This adds a bit of encoding
+overhead, but makes the API line up nicely with other Kubernetes APIs.
+
+### Labeled Metrics ###
+
+Many metric systems support labeled metrics, allowing for dimenisionality
+beyond the Kubernetes object hierarchy.  Since the HPA currently doesn't
+support specifying metric labels, this is not supported via this API.  We
+may wish to explore this in the future.