Mirror integrations kubernetes package to test/packages kubernetes

test/packages/kubernetes/_dev/build/build.yml

@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: git@1.10

test/packages/kubernetes/_dev/build/docs/README.md

@@ -0,0 +1,84 @@
+# Kubernetes integration
+
+This integration is used to collect metrics from 
+[Kubernetes clusters](https://kubernetes.io/).
+
+As one of the main pieces provided for Kubernetes monitoring, this integration is capable of fetching metrics from several components:
+
+- [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
+- [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)
+- [apiserver](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/)
+- [controller-manager](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/)
+- [scheduler](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/)
+- [proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/)
+
+Some of the previous components are running on each of the Kubernetes nodes (like `kubelet` or `proxy`) while others provide
+a single cluster-wide endpoint. This is important to determine the optimal configuration and running strategy
+for the different datasets included in the integration.
+
+For a complete reference on how to configure and run this package on Kubernetes as part of a `DaemonSet` and a `Deployment`,
+there's a complete [example manifest](https://github.com/elastic/beats/blob/master/deploy/kubernetes/elastic-agent-kubernetes.yaml)
+available.
+
+#### Kubernetes endpoints and metricsets
+
+Kubernetes module is a bit complex as its internal datasets require access to a wide variety of endpoints.
+
+This section highlights and introduces some groups of datasets with similar endpoint access needs. 
+For more details on the datasets see `configuration example` and the `datasets` sections below.
+
+
+#### node / system / pod / container / module / volume
+
+The datasets `container`, `node`, `pod`, `system` and `volume` require access to the `kubelet endpoint` in each of
+the Kubernetes nodes, hence it's recommended to include them as part
+of an `Agent DaemonSet` or standalone Agents running on the hosts.
+
+Depending on the version and configuration of Kubernetes nodes, `kubelet` might provide a read only http port (typically 10255),
+which is used in some configuration examples. But in general, and lately, this endpoint requires SSL (`https`) access
+(to port 10250 by default) and token based authentication.
+
+
+##### state_* and event
+
+All datasets with the `state_` prefix require `hosts` field pointing to `kube-state-metrics`
+service within the cluster. As the service provides cluster-wide metrics, there's no need to fetch them per node,
+hence the recommendation is to run these datasets as part of an `Agent Deployment` with one only replica.
+
+Note: Kube-state-metrics is not deployed by default in Kubernetes. For these cases the instructions for its
+deployment are available [here](https://github.com/kubernetes/kube-state-metrics#kubernetes-deployment). 
+Generally `kube-state-metrics` runs a `Deployment` and is accessible via a service called `kube-state-metrics` on
+`kube-system` namespace, which will be the service to use in our configuration.
+
+state_* datasets are not enabled by default.
+
+#### apiserver
+
+The apiserver dataset requires access to the Kubernetes API, which should be easily available in all Kubernetes
+environments. Depending on the Kubernetes configuration, the API access might require SSL (`https`) and token
+based authentication.
+
+#### proxy
+
+The proxy dataset requires access to the proxy endpoint in each of Kubernetes nodes, hence it's recommended
+to configure it as a part of an `Agent DaemonSet`.
+
+#### scheduler and controllermanager
+
+These datasets require access to the Kubernetes `controller-manager` and `scheduler` endpoints. By default, these pods
+run only on master nodes, and they are not exposed via a Service, but there are different strategies
+available for its configuration:
+
+- Create `Kubernetes Services` to make `kube-controller-manager` and `kube-scheduler` available and configure
+ the datasets to point to these services as part of an `Agent Deployment`.
+- Run these datasets as part an `Agent Daemonset` (with HostNetwork setting) with a `nodeSelector` to only run on Master nodes.
+
+These datasets are not enabled by default.
+
+Note: In some "As a Service" Kubernetes implementations, like `GKE`, the master nodes or even the pods running on
+the masters won't be visible. In these cases it won't be possible to use `scheduler` and `controllermanager` metricsets.
+
+## Compatibility
+
+The Kubernetes package is tested with Kubernetes 1.13.x, 1.14.x, 1.15.x, 1.16.x, 1.17.x, and 1.18.x
+

test/packages/kubernetes/_dev/build/docs/events.md

@@ -0,0 +1,16 @@
+# events
+
+## Metrics
+
+### event
+
+This is the `event` dataset of the Kubernetes package. It collects Kubernetes events
+related metrics.
+
+If Leader Election is activated (default behaviour) only the `elastic agent` which holds the leadership lock
+will retrieve events related metrics.
+This is relevant in multi-node kubernetes cluster and prevents duplicate data.
+
+{{event "event"}}
+
+{{fields "event"}}

test/packages/kubernetes/_dev/build/docs/kube-apiserver.md

@@ -0,0 +1,37 @@
+# kube-apiserver
+
+## Metrics
+
+### apiserver
+
+This is the `apiserver` dataset of the Kubernetes package, in charge of retrieving metrics
+from the Kubernetes API (available at `/metrics`).
+
+This metricset needs access to the `apiserver` component of Kubernetes, accessible typically
+by any POD via the `kubernetes.default` service or via environment
+variables (`KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT`).
+
+If Leader Election is activated (default behaviour) only the `elastic agent` which holds the leadership lock
+will retrieve metrics from the `apiserver`.
+This is relevant in multi-node kubernetes cluster and prevents duplicate data.
+
+When the API uses https, the pod will need to authenticate using its default token and trust
+the server using the appropriate CA file.
+
+Configuration example using https and token based authentication:
+
+
+In order to access the `/metrics` path of the API service, some Kubernetes environments might
+require the following permission to be added to a ClusterRole.
+
+```yaml
+rules:
+- nonResourceURLs:
+  - /metrics
+  verbs:
+  - get
+```
+
+{{event "apiserver"}}
+
+{{fields "apiserver"}}

test/packages/kubernetes/_dev/build/docs/kube-controller-manager.md

@@ -0,0 +1,12 @@
+# kube-controller-manager
+
+## Metrics
+
+### controllermanager
+
+This is the `controllermanager` dataset for the Kubernetes package. It collects from
+Kubernetes controller component `metrics` endpoint.
+
+{{event "controllermanager"}}
+
+{{fields "controllermanager"}}

test/packages/kubernetes/_dev/build/docs/kube-proxy.md

@@ -0,0 +1,12 @@
+# kube-proxy
+
+## Metrics
+
+### proxy
+
+This is the `proxy` dataset of the Kubernetes package. It collects metrics
+from Kubernetes Proxy component.
+
+{{event "proxy"}}
+
+{{fields "proxy"}}

test/packages/kubernetes/_dev/build/docs/kube-scheduler.md

@@ -0,0 +1,12 @@
+# kube-scheduler
+
+## Metrics
+
+### scheduler
+
+This is the `scheduler` dataset of the Kubernetes package. It collects metrics
+from Kubernetes Scheduler component.
+
+{{event "scheduler"}}
+
+{{fields "scheduler"}}

test/packages/kubernetes/_dev/build/docs/kube-state-metrics.md

@@ -0,0 +1,132 @@
+# kube-state-metrics
+
+## Metrics
+
+If Leader Election is activated (default behaviour) only the `elastic agent` which holds the leadership lock
+will retrieve metrics from the `kube_state_metrics`.
+This is relevant in multi-node kubernetes cluster and prevents duplicate data.
+
+### state_container
+
+This is the `state_container` dataset of the Kubernetes package. It collects container related
+metrics from `kube_state_metrics`.
+
+{{event "state_container"}}
+
+{{fields "state_container"}}
+
+### state_cronjob
+
+This is the `state_cronjob` dataset of the Kubernetes package. It collects cronjob related
+metrics from `kube_state_metrics`.
+
+{{event "state_cronjob"}}
+
+{{fields "state_cronjob"}}
+
+### state_daemonset
+
+This is the `state_daemonset` dataset of the Kubernetes package. It collects daemonset related
+metrics from `kube_state_metrics`.
+
+{{event "state_daemonset"}}
+
+{{fields "state_daemonset"}}
+
+### state_deployment
+
+This is the `state_deployment` dataset of the Kubernetes package. It collects deployment related
+metrics from `kube_state_metrics`.
+
+{{event "state_deployment"}}
+
+{{fields "state_deployment"}}
+
+### state_job
+
+This is the `state_job` dataset of the Kubernetes package. It collects job related
+metrics from `kube_state_metrics`.
+
+{{event "state_job"}}
+
+{{fields "state_job"}}
+
+### state_node
+
+This is the `state_node` dataset of the Kubernetes package. It collects node related
+metrics from `kube_state_metrics`.
+
+{{event "state_node"}}
+
+{{fields "state_node"}}
+
+### state_persistentvolume
+
+This is the `state_persistentvolume` dataset of the Kubernetes package. It collects 
+PersistentVolume related metrics from `kube_state_metrics`.
+
+{{event "state_persistentvolume"}}
+
+{{fields "state_persistentvolume"}}
+
+### state_persistentvolumeclaim
+
+This is the `state_persistentvolumeclaim` dataset of the Kubernetes package. It collects 
+PersistentVolumeClaim related metrics from `kube_state_metrics`.
+
+{{event "state_persistentvolumeclaim"}}
+
+{{fields "state_persistentvolumeclaim"}}
+
+### state_pod
+
+This is the `state_pod` dataset of the Kubernetes package. It collects 
+Pod related metrics from `kube_state_metrics`.
+
+{{event "state_pod"}}
+
+{{fields "state_pod"}}
+
+### state_replicaset
+
+This is the `state_replicaset` dataset of the Kubernetes package. It collects 
+Replicaset related metrics from `kube_state_metrics`.
+
+{{event "state_replicaset"}}
+
+{{fields "state_replicaset"}}
+
+### state_resourcequota
+
+This is the `state_resourcequota` dataset of the Kubernetes package. It collects ResourceQuota related metrics
+from `kube_state_metrics`.
+
+{{event "state_resourcequota"}}
+
+{{fields "state_resourcequota"}}
+
+### state_service
+
+This is the `state_service` dataset of the Kubernetes package. It collects 
+Service related metrics from `kube_state_metrics`.
+
+{{event "state_service"}}
+
+{{fields "state_service"}}
+
+### state_statefulset
+
+This is the `state_statefulset` dataset of the Kubernetes package.
+
+{{event "state_statefulset"}}
+
+{{fields "state_statefulset"}}
+
+### state_storageclass
+
+This is the `state_storageclass` dataset of the Kubernetes package. It collects 
+StorageClass related metrics from `kube_state_metrics`.
+
+{{event "state_storageclass"}}
+
+{{fields "state_storageclass"}}

test/packages/kubernetes/_dev/build/docs/kubelet.md

@@ -0,0 +1,48 @@
+# kubelet
+
+## Metrics
+
+### container
+
+This is the `container` dataset of the Kubernetes package. It collects container related metrics
+from Kubelet's monitoring APIs.
+
+{{event "container"}}
+
+{{fields "container"}}
+
+### node
+
+This is the `node` dataset of the Kubernetes package. It collects Node related metrics
+from Kubelet's monitoring APIs.
+
+{{event "node"}}
+
+{{fields "node"}}
+
+### pod
+
+This is the `pod` dataset of the Kubernetes package. It collects Pod related metrics
+from Kubelet's monitoring APIs.
+
+{{event "pod"}}
+
+{{fields "pod"}}
+
+### system
+
+This is the `system` dataset of the Kubernetes package. It collects System related metrics
+from Kubelet's monitoring APIs.
+
+{{event "system"}}
+
+{{fields "system"}}
+
+### volume
+
+This is the `volume` dataset of the Kubernetes package. It collects Volume related metrics
+from Kubelet's monitoring APIs.
+
+{{event "volume"}}
+
+{{fields "volume"}}

test/packages/kubernetes/_dev/deploy/k8s/cluster-role-binding.yaml

@@ -0,0 +1,15 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  labels:
+    app.kubernetes.io/name: kube-state-metrics
+    app.kubernetes.io/version: 2.0.0-rc.1
+  name: kube-state-metrics
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: kube-state-metrics
+subjects:
+  - kind: ServiceAccount
+    name: kube-state-metrics
+    namespace: kube-system