diff --git a/test/packages/kubernetes/_dev/build/build.yml b/test/packages/kubernetes/_dev/build/build.yml new file mode 100644 index 0000000000..002aa15659 --- /dev/null +++ b/test/packages/kubernetes/_dev/build/build.yml @@ -0,0 +1,3 @@ +dependencies: + ecs: + reference: git@1.10 diff --git a/test/packages/kubernetes/_dev/build/docs/README.md b/test/packages/kubernetes/_dev/build/docs/README.md new file mode 100644 index 0000000000..f7dc53d855 --- /dev/null +++ b/test/packages/kubernetes/_dev/build/docs/README.md @@ -0,0 +1,84 @@ +# Kubernetes integration + +This integration is used to collect metrics from +[Kubernetes clusters](https://kubernetes.io/). + +As one of the main pieces provided for Kubernetes monitoring, this integration is capable of fetching metrics from several components: + +- [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) +- [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) +- [apiserver](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) +- [controller-manager](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/) +- [scheduler](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/) +- [proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) + +Some of the previous components are running on each of the Kubernetes nodes (like `kubelet` or `proxy`) while others provide +a single cluster-wide endpoint. This is important to determine the optimal configuration and running strategy +for the different datasets included in the integration. + +For a complete reference on how to configure and run this package on Kubernetes as part of a `DaemonSet` and a `Deployment`, +there's a complete [example manifest](https://github.com/elastic/beats/blob/master/deploy/kubernetes/elastic-agent-kubernetes.yaml) +available. + +#### Kubernetes endpoints and metricsets + +Kubernetes module is a bit complex as its internal datasets require access to a wide variety of endpoints. + +This section highlights and introduces some groups of datasets with similar endpoint access needs. +For more details on the datasets see `configuration example` and the `datasets` sections below. + + +#### node / system / pod / container / module / volume + +The datasets `container`, `node`, `pod`, `system` and `volume` require access to the `kubelet endpoint` in each of +the Kubernetes nodes, hence it's recommended to include them as part +of an `Agent DaemonSet` or standalone Agents running on the hosts. + +Depending on the version and configuration of Kubernetes nodes, `kubelet` might provide a read only http port (typically 10255), +which is used in some configuration examples. But in general, and lately, this endpoint requires SSL (`https`) access +(to port 10250 by default) and token based authentication. + + +##### state_* and event + +All datasets with the `state_` prefix require `hosts` field pointing to `kube-state-metrics` +service within the cluster. As the service provides cluster-wide metrics, there's no need to fetch them per node, +hence the recommendation is to run these datasets as part of an `Agent Deployment` with one only replica. + +Note: Kube-state-metrics is not deployed by default in Kubernetes. For these cases the instructions for its +deployment are available [here](https://github.com/kubernetes/kube-state-metrics#kubernetes-deployment). +Generally `kube-state-metrics` runs a `Deployment` and is accessible via a service called `kube-state-metrics` on +`kube-system` namespace, which will be the service to use in our configuration. + +state_* datasets are not enabled by default. + +#### apiserver + +The apiserver dataset requires access to the Kubernetes API, which should be easily available in all Kubernetes +environments. Depending on the Kubernetes configuration, the API access might require SSL (`https`) and token +based authentication. + +#### proxy + +The proxy dataset requires access to the proxy endpoint in each of Kubernetes nodes, hence it's recommended +to configure it as a part of an `Agent DaemonSet`. + +#### scheduler and controllermanager + +These datasets require access to the Kubernetes `controller-manager` and `scheduler` endpoints. By default, these pods +run only on master nodes, and they are not exposed via a Service, but there are different strategies +available for its configuration: + +- Create `Kubernetes Services` to make `kube-controller-manager` and `kube-scheduler` available and configure + the datasets to point to these services as part of an `Agent Deployment`. +- Run these datasets as part an `Agent Daemonset` (with HostNetwork setting) with a `nodeSelector` to only run on Master nodes. + +These datasets are not enabled by default. + +Note: In some "As a Service" Kubernetes implementations, like `GKE`, the master nodes or even the pods running on +the masters won't be visible. In these cases it won't be possible to use `scheduler` and `controllermanager` metricsets. + +## Compatibility + +The Kubernetes package is tested with Kubernetes 1.13.x, 1.14.x, 1.15.x, 1.16.x, 1.17.x, and 1.18.x + diff --git a/test/packages/kubernetes/_dev/build/docs/events.md b/test/packages/kubernetes/_dev/build/docs/events.md new file mode 100644 index 0000000000..bd2ce5a228 --- /dev/null +++ b/test/packages/kubernetes/_dev/build/docs/events.md @@ -0,0 +1,16 @@ +# events + +## Metrics + +### event + +This is the `event` dataset of the Kubernetes package. It collects Kubernetes events +related metrics. + +If Leader Election is activated (default behaviour) only the `elastic agent` which holds the leadership lock +will retrieve events related metrics. +This is relevant in multi-node kubernetes cluster and prevents duplicate data. + +{{event "event"}} + +{{fields "event"}} \ No newline at end of file diff --git a/test/packages/kubernetes/_dev/build/docs/kube-apiserver.md b/test/packages/kubernetes/_dev/build/docs/kube-apiserver.md new file mode 100644 index 0000000000..a28c98cc00 --- /dev/null +++ b/test/packages/kubernetes/_dev/build/docs/kube-apiserver.md @@ -0,0 +1,37 @@ +# kube-apiserver + +## Metrics + +### apiserver + +This is the `apiserver` dataset of the Kubernetes package, in charge of retrieving metrics +from the Kubernetes API (available at `/metrics`). + +This metricset needs access to the `apiserver` component of Kubernetes, accessible typically +by any POD via the `kubernetes.default` service or via environment +variables (`KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT`). + +If Leader Election is activated (default behaviour) only the `elastic agent` which holds the leadership lock +will retrieve metrics from the `apiserver`. +This is relevant in multi-node kubernetes cluster and prevents duplicate data. + +When the API uses https, the pod will need to authenticate using its default token and trust +the server using the appropriate CA file. + +Configuration example using https and token based authentication: + + +In order to access the `/metrics` path of the API service, some Kubernetes environments might +require the following permission to be added to a ClusterRole. + +```yaml +rules: +- nonResourceURLs: + - /metrics + verbs: + - get +``` + +{{event "apiserver"}} + +{{fields "apiserver"}} \ No newline at end of file diff --git a/test/packages/kubernetes/_dev/build/docs/kube-controller-manager.md b/test/packages/kubernetes/_dev/build/docs/kube-controller-manager.md new file mode 100644 index 0000000000..01faeab816 --- /dev/null +++ b/test/packages/kubernetes/_dev/build/docs/kube-controller-manager.md @@ -0,0 +1,12 @@ +# kube-controller-manager + +## Metrics + +### controllermanager + +This is the `controllermanager` dataset for the Kubernetes package. It collects from +Kubernetes controller component `metrics` endpoint. + +{{event "controllermanager"}} + +{{fields "controllermanager"}} \ No newline at end of file diff --git a/test/packages/kubernetes/_dev/build/docs/kube-proxy.md b/test/packages/kubernetes/_dev/build/docs/kube-proxy.md new file mode 100644 index 0000000000..6df5baf0c8 --- /dev/null +++ b/test/packages/kubernetes/_dev/build/docs/kube-proxy.md @@ -0,0 +1,12 @@ +# kube-proxy + +## Metrics + +### proxy + +This is the `proxy` dataset of the Kubernetes package. It collects metrics +from Kubernetes Proxy component. + +{{event "proxy"}} + +{{fields "proxy"}} \ No newline at end of file diff --git a/test/packages/kubernetes/_dev/build/docs/kube-scheduler.md b/test/packages/kubernetes/_dev/build/docs/kube-scheduler.md new file mode 100644 index 0000000000..bfd8b65c25 --- /dev/null +++ b/test/packages/kubernetes/_dev/build/docs/kube-scheduler.md @@ -0,0 +1,12 @@ +# kube-scheduler + +## Metrics + +### scheduler + +This is the `scheduler` dataset of the Kubernetes package. It collects metrics +from Kubernetes Scheduler component. + +{{event "scheduler"}} + +{{fields "scheduler"}} \ No newline at end of file diff --git a/test/packages/kubernetes/_dev/build/docs/kube-state-metrics.md b/test/packages/kubernetes/_dev/build/docs/kube-state-metrics.md new file mode 100644 index 0000000000..4358201cd3 --- /dev/null +++ b/test/packages/kubernetes/_dev/build/docs/kube-state-metrics.md @@ -0,0 +1,132 @@ +# kube-state-metrics + +## Metrics + +If Leader Election is activated (default behaviour) only the `elastic agent` which holds the leadership lock +will retrieve metrics from the `kube_state_metrics`. +This is relevant in multi-node kubernetes cluster and prevents duplicate data. + +### state_container + +This is the `state_container` dataset of the Kubernetes package. It collects container related +metrics from `kube_state_metrics`. + +{{event "state_container"}} + +{{fields "state_container"}} + +### state_cronjob + +This is the `state_cronjob` dataset of the Kubernetes package. It collects cronjob related +metrics from `kube_state_metrics`. + +{{event "state_cronjob"}} + +{{fields "state_cronjob"}} + +### state_daemonset + +This is the `state_daemonset` dataset of the Kubernetes package. It collects daemonset related +metrics from `kube_state_metrics`. + +{{event "state_daemonset"}} + +{{fields "state_daemonset"}} + +### state_deployment + +This is the `state_deployment` dataset of the Kubernetes package. It collects deployment related +metrics from `kube_state_metrics`. + +{{event "state_deployment"}} + +{{fields "state_deployment"}} + +### state_job + +This is the `state_job` dataset of the Kubernetes package. It collects job related +metrics from `kube_state_metrics`. + +{{event "state_job"}} + +{{fields "state_job"}} + +### state_node + +This is the `state_node` dataset of the Kubernetes package. It collects node related +metrics from `kube_state_metrics`. + +{{event "state_node"}} + +{{fields "state_node"}} + +### state_persistentvolume + +This is the `state_persistentvolume` dataset of the Kubernetes package. It collects +PersistentVolume related metrics from `kube_state_metrics`. + +{{event "state_persistentvolume"}} + +{{fields "state_persistentvolume"}} + +### state_persistentvolumeclaim + +This is the `state_persistentvolumeclaim` dataset of the Kubernetes package. It collects +PersistentVolumeClaim related metrics from `kube_state_metrics`. + +{{event "state_persistentvolumeclaim"}} + +{{fields "state_persistentvolumeclaim"}} + +### state_pod + +This is the `state_pod` dataset of the Kubernetes package. It collects +Pod related metrics from `kube_state_metrics`. + +{{event "state_pod"}} + +{{fields "state_pod"}} + +### state_replicaset + +This is the `state_replicaset` dataset of the Kubernetes package. It collects +Replicaset related metrics from `kube_state_metrics`. + +{{event "state_replicaset"}} + +{{fields "state_replicaset"}} + +### state_resourcequota + +This is the `state_resourcequota` dataset of the Kubernetes package. It collects ResourceQuota related metrics +from `kube_state_metrics`. + +{{event "state_resourcequota"}} + +{{fields "state_resourcequota"}} + +### state_service + +This is the `state_service` dataset of the Kubernetes package. It collects +Service related metrics from `kube_state_metrics`. + +{{event "state_service"}} + +{{fields "state_service"}} + +### state_statefulset + +This is the `state_statefulset` dataset of the Kubernetes package. + +{{event "state_statefulset"}} + +{{fields "state_statefulset"}} + +### state_storageclass + +This is the `state_storageclass` dataset of the Kubernetes package. It collects +StorageClass related metrics from `kube_state_metrics`. + +{{event "state_storageclass"}} + +{{fields "state_storageclass"}} \ No newline at end of file diff --git a/test/packages/kubernetes/_dev/build/docs/kubelet.md b/test/packages/kubernetes/_dev/build/docs/kubelet.md new file mode 100644 index 0000000000..54e684a860 --- /dev/null +++ b/test/packages/kubernetes/_dev/build/docs/kubelet.md @@ -0,0 +1,48 @@ +# kubelet + +## Metrics + +### container + +This is the `container` dataset of the Kubernetes package. It collects container related metrics +from Kubelet's monitoring APIs. + +{{event "container"}} + +{{fields "container"}} + +### node + +This is the `node` dataset of the Kubernetes package. It collects Node related metrics +from Kubelet's monitoring APIs. + +{{event "node"}} + +{{fields "node"}} + +### pod + +This is the `pod` dataset of the Kubernetes package. It collects Pod related metrics +from Kubelet's monitoring APIs. + +{{event "pod"}} + +{{fields "pod"}} + +### system + +This is the `system` dataset of the Kubernetes package. It collects System related metrics +from Kubelet's monitoring APIs. + +{{event "system"}} + +{{fields "system"}} + +### volume + +This is the `volume` dataset of the Kubernetes package. It collects Volume related metrics +from Kubelet's monitoring APIs. + +{{event "volume"}} + +{{fields "volume"}} \ No newline at end of file diff --git a/test/packages/kubernetes/_dev/deploy/k8s/cluster-role-binding.yaml b/test/packages/kubernetes/_dev/deploy/k8s/cluster-role-binding.yaml new file mode 100644 index 0000000000..ff5c2be138 --- /dev/null +++ b/test/packages/kubernetes/_dev/deploy/k8s/cluster-role-binding.yaml @@ -0,0 +1,15 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + labels: + app.kubernetes.io/name: kube-state-metrics + app.kubernetes.io/version: 2.0.0-rc.1 + name: kube-state-metrics +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: kube-state-metrics +subjects: + - kind: ServiceAccount + name: kube-state-metrics + namespace: kube-system diff --git a/test/packages/kubernetes/_dev/deploy/k8s/cluster-role.yaml b/test/packages/kubernetes/_dev/deploy/k8s/cluster-role.yaml new file mode 100644 index 0000000000..1b567f9257 --- /dev/null +++ b/test/packages/kubernetes/_dev/deploy/k8s/cluster-role.yaml @@ -0,0 +1,108 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + app.kubernetes.io/name: kube-state-metrics + app.kubernetes.io/version: 2.0.0-rc.1 + name: kube-state-metrics +rules: + - apiGroups: + - "" + resources: + - configmaps + - secrets + - nodes + - pods + - services + - resourcequotas + - replicationcontrollers + - limitranges + - persistentvolumeclaims + - persistentvolumes + - namespaces + - endpoints + verbs: + - list + - watch + - apiGroups: + - apps + resources: + - statefulsets + - daemonsets + - deployments + - replicasets + verbs: + - list + - watch + - apiGroups: + - batch + resources: + - cronjobs + - jobs + verbs: + - list + - watch + - apiGroups: + - autoscaling + resources: + - horizontalpodautoscalers + verbs: + - list + - watch + - apiGroups: + - authentication.k8s.io + resources: + - tokenreviews + verbs: + - create + - apiGroups: + - authorization.k8s.io + resources: + - subjectaccessreviews + verbs: + - create + - apiGroups: + - policy + resources: + - poddisruptionbudgets + verbs: + - list + - watch + - apiGroups: + - certificates.k8s.io + resources: + - certificatesigningrequests + verbs: + - list + - watch + - apiGroups: + - storage.k8s.io + resources: + - storageclasses + - volumeattachments + verbs: + - list + - watch + - apiGroups: + - admissionregistration.k8s.io + resources: + - mutatingwebhookconfigurations + - validatingwebhookconfigurations + verbs: + - list + - watch + - apiGroups: + - networking.k8s.io + resources: + - networkpolicies + - ingresses + verbs: + - list + - watch + - apiGroups: + - coordination.k8s.io + resources: + - leases + verbs: + - list + - watch diff --git a/test/packages/kubernetes/_dev/deploy/k8s/conrjob.yaml b/test/packages/kubernetes/_dev/deploy/k8s/conrjob.yaml new file mode 100644 index 0000000000..472da4598b --- /dev/null +++ b/test/packages/kubernetes/_dev/deploy/k8s/conrjob.yaml @@ -0,0 +1,19 @@ +apiVersion: batch/v1beta1 +kind: CronJob +metadata: + name: hello +spec: + schedule: "*/1 * * * *" + jobTemplate: + spec: + template: + spec: + containers: + - name: hello + image: busybox + imagePullPolicy: IfNotPresent + command: + - /bin/sh + - -c + - date; echo Hello from the Kubernetes cluster + restartPolicy: OnFailure diff --git a/test/packages/kubernetes/_dev/deploy/k8s/deployment.yaml b/test/packages/kubernetes/_dev/deploy/k8s/deployment.yaml index 72bb670262..420eece825 100644 --- a/test/packages/kubernetes/_dev/deploy/k8s/deployment.yaml +++ b/test/packages/kubernetes/_dev/deploy/k8s/deployment.yaml @@ -1,24 +1,44 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: nginx-deployment + labels: + app.kubernetes.io/name: kube-state-metrics + app.kubernetes.io/version: 2.0.0-rc.1 + name: kube-state-metrics + namespace: kube-system spec: + replicas: 1 selector: matchLabels: - app: nginx - replicas: 16 - strategy: - type: RollingUpdate - rollingUpdate: - maxSurge: 1 - maxUnavailable: 0 + app.kubernetes.io/name: kube-state-metrics template: metadata: labels: - app: nginx + app.kubernetes.io/name: kube-state-metrics + app.kubernetes.io/version: 2.0.0-rc.1 spec: containers: - - name: nginx - image: nginx:1.14.2 + - image: quay.io/coreos/kube-state-metrics:v1.9.0 + livenessProbe: + httpGet: + path: /healthz + port: 8080 + initialDelaySeconds: 5 + timeoutSeconds: 5 + name: kube-state-metrics ports: - - containerPort: 80 + - containerPort: 8080 + name: http-metrics + - containerPort: 8081 + name: telemetry + readinessProbe: + httpGet: + path: / + port: 8081 + initialDelaySeconds: 5 + timeoutSeconds: 5 + securityContext: + runAsUser: 65534 + nodeSelector: + kubernetes.io/os: linux + serviceAccountName: kube-state-metrics diff --git a/test/packages/kubernetes/_dev/deploy/k8s/job.yaml b/test/packages/kubernetes/_dev/deploy/k8s/job.yaml new file mode 100644 index 0000000000..c1087e28c4 --- /dev/null +++ b/test/packages/kubernetes/_dev/deploy/k8s/job.yaml @@ -0,0 +1,17 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: hello +spec: + template: + spec: + containers: + - name: hello + image: busybox + imagePullPolicy: IfNotPresent + command: + - /bin/sh + - -c + - date; echo Hello from the Kubernetes cluster + restartPolicy: OnFailure + backoffLimit: 4 diff --git a/test/packages/kubernetes/_dev/deploy/k8s/pv.yaml b/test/packages/kubernetes/_dev/deploy/k8s/pv.yaml new file mode 100644 index 0000000000..bca6cd1eb5 --- /dev/null +++ b/test/packages/kubernetes/_dev/deploy/k8s/pv.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: PersistentVolume +metadata: + name: task-pv-volume + labels: + name: task-pv-volume +spec: + storageClassName: generic + capacity: + storage: 2Ki + accessModes: + - ReadWriteOnce + hostPath: + path: "/tmp/data" diff --git a/test/packages/kubernetes/_dev/deploy/k8s/pvc.yaml b/test/packages/kubernetes/_dev/deploy/k8s/pvc.yaml new file mode 100644 index 0000000000..7b24222960 --- /dev/null +++ b/test/packages/kubernetes/_dev/deploy/k8s/pvc.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: task-pv-claim +spec: + storageClassName: generic + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Ki + selector: + matchLabels: + name: task-pv-volume diff --git a/test/packages/kubernetes/_dev/deploy/k8s/rq.yaml b/test/packages/kubernetes/_dev/deploy/k8s/rq.yaml new file mode 100644 index 0000000000..07c6d882a9 --- /dev/null +++ b/test/packages/kubernetes/_dev/deploy/k8s/rq.yaml @@ -0,0 +1,17 @@ +apiVersion: v1 +kind: List +items: + - apiVersion: v1 + kind: ResourceQuota + metadata: + name: pods-high + spec: + hard: + cpu: "1000" + memory: 200Ki + pods: "10" + scopeSelector: + matchExpressions: + - operator: In + scopeName: PriorityClass + values: ["high"] diff --git a/test/packages/kubernetes/_dev/deploy/k8s/service-account.yaml b/test/packages/kubernetes/_dev/deploy/k8s/service-account.yaml new file mode 100644 index 0000000000..1708b4beac --- /dev/null +++ b/test/packages/kubernetes/_dev/deploy/k8s/service-account.yaml @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + labels: + app.kubernetes.io/name: kube-state-metrics + app.kubernetes.io/version: 2.0.0-rc.1 + name: kube-state-metrics + namespace: kube-system diff --git a/test/packages/kubernetes/_dev/deploy/k8s/service.yaml b/test/packages/kubernetes/_dev/deploy/k8s/service.yaml new file mode 100644 index 0000000000..ef80dc44bc --- /dev/null +++ b/test/packages/kubernetes/_dev/deploy/k8s/service.yaml @@ -0,0 +1,19 @@ +apiVersion: v1 +kind: Service +metadata: + labels: + app.kubernetes.io/name: kube-state-metrics + app.kubernetes.io/version: 2.0.0-rc.1 + name: kube-state-metrics + namespace: kube-system +spec: + clusterIP: None + ports: + - name: http-metrics + port: 8080 + targetPort: http-metrics + - name: telemetry + port: 8081 + targetPort: telemetry + selector: + app.kubernetes.io/name: kube-state-metrics diff --git a/test/packages/kubernetes/_dev/deploy/k8s/ss.yaml b/test/packages/kubernetes/_dev/deploy/k8s/ss.yaml new file mode 100644 index 0000000000..037d913837 --- /dev/null +++ b/test/packages/kubernetes/_dev/deploy/k8s/ss.yaml @@ -0,0 +1,22 @@ +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: web +spec: + selector: + matchLabels: + app: nginx # has to match .spec.template.metadata.labels + serviceName: "nginx" + replicas: 1 # by default is 1 + template: + metadata: + labels: + app: nginx # has to match .spec.selector.matchLabels + spec: + terminationGracePeriodSeconds: 10 + containers: + - name: nginx + image: k8s.gcr.io/nginx-slim:0.8 + ports: + - containerPort: 80 + name: web diff --git a/test/packages/kubernetes/_dev/deploy/variants.yml b/test/packages/kubernetes/_dev/deploy/variants.yml deleted file mode 100644 index 02f0fffaca..0000000000 --- a/test/packages/kubernetes/_dev/deploy/variants.yml +++ /dev/null @@ -1,5 +0,0 @@ -# this is not valid right now and is not used at all, it exists to make linter happy -variants: - v1: - SERVICE_VERSION: 1.3.0 -default: v1 diff --git a/test/packages/kubernetes/changelog.yml b/test/packages/kubernetes/changelog.yml index 11d6363d75..74b56f0fd5 100644 --- a/test/packages/kubernetes/changelog.yml +++ b/test/packages/kubernetes/changelog.yml @@ -1,6 +1,101 @@ # newer versions go on top - version: "999.999.999" + changes: + - description: Give the package highest priority + type: enhancement # can be one of: enhancement, bugfix, breaking-change + link: https://github.com/elastic/elastic-package/pull/460 +- version: "0.12.0" + changes: + - description: Update kubernetes package ecs fields with orchestrator.cluster.url and orchestrator.cluster.name + type: enhancement + link: https://github.com/elastic/integrations/pull/1407 +- version: "0.11.1" + changes: + - description: Escape special characters in docs + type: enhancement + link: https://github.com/elastic/integrations/pull/1405 +- version: "0.11.0" + changes: + - description: Update documentation to fit mdx spec + type: enhancement + link: https://github.com/elastic/integrations/pull/1401 +- version: "0.10.0" + changes: + - description: Update integration description + type: enhancement + link: https://github.com/elastic/integrations/pull/1364 +- version: "0.9.1" + changes: + - description: Add missing field "kubernetes.daemonset.name" field for state_pod and state_container + type: bugfix + link: https://github.com/elastic/integrations/pull/1335 +- version: "0.9.0" + changes: + - description: Enhance kubernetes package with state_job data stream + type: enhancement + link: https://github.com/elastic/integrations/pull/1018 +- version: "0.8.0" + changes: + - description: Leverage leader election in kubernetes integration + type: enhancement + link: https://github.com/elastic/integrations/pull/1321 +- version: "0.7.0" + changes: + - description: Add _meta information to Kubernetes fields + type: enhancement + link: https://github.com/elastic/integrations/pull/1289 +- version: "0.6.0" + changes: + - description: Introduce kubernetes package granularity using input_groups + type: enhancement + link: https://github.com/elastic/integrations/pull/1018 +- version: "0.5.3" + changes: + - description: Add missing field "kubernetes.statefulset.replicas.ready" + type: enhancement + link: https://github.com/elastic/integrations/pull/1160 +- version: "0.5.2" + changes: + - description: Fix stack compatability + type: bugfix + link: https://github.com/elastic/integrations/pull/998 +- version: "0.5.1" + changes: + - description: Fix references to env variables + type: bugfix + link: https://github.com/elastic/integrations/pull/970 +- version: "0.5.0" + changes: + - description: Add missing field "kubernetes.selectors.*" and extra https settings for controllermanager and scheduler datastreams + type: enhancement + link: https://github.com/elastic/integrations/pull/886 +- version: "0.4.5" + changes: + - description: Add missing field "kubernetes.pod.ip" + type: enhancement + link: https://github.com/elastic/integrations/pull/919 +- version: "0.4.4" + changes: + - description: Updating package owner + type: enhancement + link: https://github.com/elastic/integrations/pull/766 +- version: "0.4.3" + changes: + - description: Correct sample event file. + type: bugfix # can be one of: enhancement, bugfix, breaking-change + link: https://github.com/elastic/integrations/pull/754 +- version: "0.4.2" + changes: + - description: Change kibana.version constraint to be more conservative. + type: bugfix + link: https://github.com/elastic/integrations/pull/749 +- version: "0.4.1" + changes: + - description: Add missing fields + type: enhancement # can be one of: enhancement, bugfix, breaking-change + link: https://github.com/elastic/integrations/pull/745 +- version: "0.1.0" changes: - description: initial release type: enhancement # can be one of: enhancement, bugfix, breaking-change - link: https://github.com/elastic/elastic-package/pull/245 + link: https://github.com/elastic/integrations/pull/70 diff --git a/test/packages/kubernetes/data_stream/apiserver/_dev/deploy/k8s/.empty b/test/packages/kubernetes/data_stream/apiserver/_dev/deploy/k8s/.empty new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/packages/kubernetes/data_stream/apiserver/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/apiserver/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..cf6f20d0c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/apiserver/_dev/test/system/test-default-config.yml @@ -0,0 +1,7 @@ +service: kubernetes +data_stream: + vars: + bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token + period: 5s + hosts: + - https://kubernetes.default:443 diff --git a/test/packages/kubernetes/data_stream/apiserver/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/apiserver/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..46426492a2 --- /dev/null +++ b/test/packages/kubernetes/data_stream/apiserver/agent/stream/stream.yml.hbs @@ -0,0 +1,18 @@ +metricsets: ["apiserver"] +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} + +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} + +{{#if bearer_token_file}} +bearer_token_file: {{bearer_token_file}} +ssl.certificate_authorities: +{{#each ssl.certificate_authorities}} + - {{this}} +{{/each}} +{{/if}} diff --git a/test/packages/kubernetes/data_stream/apiserver/fields/agent.yml b/test/packages/kubernetes/data_stream/apiserver/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/apiserver/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/apiserver/fields/base-fields.yml b/test/packages/kubernetes/data_stream/apiserver/fields/base-fields.yml new file mode 100644 index 0000000000..7c798f4534 --- /dev/null +++ b/test/packages/kubernetes/data_stream/apiserver/fields/base-fields.yml @@ -0,0 +1,12 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. diff --git a/test/packages/kubernetes/data_stream/apiserver/fields/ecs.yml b/test/packages/kubernetes/data_stream/apiserver/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/apiserver/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/apiserver/fields/fields.yml b/test/packages/kubernetes/data_stream/apiserver/fields/fields.yml new file mode 100644 index 0000000000..d53efadb1a --- /dev/null +++ b/test/packages/kubernetes/data_stream/apiserver/fields/fields.yml @@ -0,0 +1,190 @@ +- name: kubernetes.apiserver + type: group + fields: + - name: request.client + type: keyword + description: | + Client executing requests + - name: request.resource + type: keyword + description: | + Requested resource + - name: request.subresource + type: keyword + description: | + Requested subresource + - name: request.scope + type: keyword + description: | + Request scope (cluster, namespace, resource) + - name: request.verb + type: keyword + description: | + HTTP verb + - name: request.code + type: keyword + description: | + HTTP code + - name: request.content_type + type: keyword + description: | + Request HTTP content type + - name: request.dry_run + type: keyword + description: | + Wether the request uses dry run + - name: request.kind + type: keyword + description: | + Kind of request + - name: request.component + type: keyword + description: | + Component handling the request + - name: request.group + type: keyword + description: | + API group for the resource + - name: request.version + type: keyword + description: | + version for the group + - name: request.handler + type: keyword + description: | + Request handler + - name: request.method + type: keyword + description: | + HTTP method + - name: request.host + type: keyword + description: | + Request host + - name: process + type: group + fields: + - name: cpu.sec + type: double + metric_type: counter + description: CPU seconds + - name: memory.resident.bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: Bytes in resident memory + - name: memory.virtual.bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: Bytes in virtual memory + - name: fds.open.count + type: long + metric_type: gauge + description: Number of open file descriptors + - name: started.sec + type: double + metric_type: gauge + description: Seconds since the process started + - name: http + type: group + fields: + - name: request.duration.us.percentile.* + type: object + description: Request duration microseconds percentiles + - name: request.duration.us.sum + type: double + metric_type: counter + unit: micros + description: Request duration microseconds cumulative sum + - name: request.duration.us.count + type: long + metric_type: counter + unit: micros + description: Request count for duration + - name: request.size.bytes.percentile.* + type: object + description: Request size percentiles + - name: request.size.bytes.sum + type: long + format: bytes + unit: byte + metric_type: counter + description: Request size cumulative sum + - name: request.size.bytes.count + type: long + unit: byte + metric_type: counter + description: Request count for size + - name: response.size.bytes.percentile.* + type: object + description: Response size percentiles + - name: response.size.bytes.sum + type: long + format: bytes + unit: byte + metric_type: counter + description: Response size cumulative sum + - name: response.size.bytes.count + type: long + metric_type: counter + description: Response count + - name: request.count + type: long + metric_type: counter + description: Request count for response + - name: client.request.count + type: long + metric_type: counter + description: Number of requests as client + - name: request + type: group + fields: + - name: count + type: long + metric_type: counter + description: Number of requests + - name: latency.sum + type: long + metric_type: counter + description: Requests latency, sum of latencies in microseconds + - name: latency.count + type: long + metric_type: counter + description: Request latency, number of requests + - name: latency.bucket.* + type: object + description: Request latency histogram buckets + - name: duration.us.sum + type: long + metric_type: counter + description: Request duration, sum in microseconds + - name: duration.us.count + type: long + metric_type: counter + description: Request duration, number of operations + - name: duration.us.bucket.* + type: object + description: Request duration, histogram buckets + - name: current.count + type: long + metric_type: counter + description: Inflight requests + - name: longrunning.count + type: long + metric_type: counter + description: Number of requests active long running requests + - name: etcd.object.count + type: long + metric_type: gauge + description: Number of kubernetes objects at etcd + - name: audit.event.count + type: long + metric_type: counter + description: Number of audit events + - name: audit.rejected.count + type: long + metric_type: counter + description: Number of audit rejected events diff --git a/test/packages/kubernetes/data_stream/apiserver/manifest.yml b/test/packages/kubernetes/data_stream/apiserver/manifest.yml new file mode 100644 index 0000000000..86a58b1eb3 --- /dev/null +++ b/test/packages/kubernetes/data_stream/apiserver/manifest.yml @@ -0,0 +1,45 @@ +title: Kubernetes API Server metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + vars: + - name: bearer_token_file + type: text + title: Bearer Token File + multi: false + required: true + show_user: true + default: /var/run/secrets/kubernetes.io/serviceaccount/token + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - https://${env.KUBERNETES_SERVICE_HOST}:${env.KUBERNETES_SERVICE_PORT} + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 30s + - name: ssl.certificate_authorities + type: text + title: SSL Certificate Authorities + multi: true + required: true + show_user: true + default: + - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + title: Kubernetes API Server metrics + description: Collect Kubernetes API Server metrics diff --git a/test/packages/kubernetes/data_stream/apiserver/sample_event.json b/test/packages/kubernetes/data_stream/apiserver/sample_event.json new file mode 100644 index 0000000000..fda7d3222a --- /dev/null +++ b/test/packages/kubernetes/data_stream/apiserver/sample_event.json @@ -0,0 +1,62 @@ +{ + "@timestamp": "2020-06-25T12:30:34.616Z", + "metricset": { + "name": "apiserver", + "period": 30000 + }, + "service": { + "address": "10.96.0.1:443", + "type": "kubernetes" + }, + "event": { + "dataset": "kubernetes.apiserver", + "module": "kubernetes", + "duration": 114780772 + }, + "kubernetes": { + "apiserver": { + "request": { + "client": "metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format", + "version": "v1", + "count": 3, + "scope": "cluster", + "content_type": "application/vnd.kubernetes.protobuf", + "code": "200", + "verb": "LIST", + "component": "apiserver", + "resource": "nodes" + } + } + }, + "ecs": { + "version": "1.5.0" + }, + "agent": { + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat" + }, + "host": { + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "os": { + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)" + } + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/container/_dev/deploy/k8s/.empty b/test/packages/kubernetes/data_stream/container/_dev/deploy/k8s/.empty new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/packages/kubernetes/data_stream/container/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/container/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..6c69562d49 --- /dev/null +++ b/test/packages/kubernetes/data_stream/container/_dev/test/system/test-default-config.yml @@ -0,0 +1,7 @@ +service: kubernetes +data_stream: + vars: + bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token + period: 5s + hosts: + - https://{{Hostname}}:10250 diff --git a/test/packages/kubernetes/data_stream/container/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/container/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..bdfe1999ea --- /dev/null +++ b/test/packages/kubernetes/data_stream/container/agent/stream/stream.yml.hbs @@ -0,0 +1,11 @@ +metricsets: ["container"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if bearer_token_file}} +bearer_token_file: {{bearer_token_file}} +ssl.verification_mode: {{ssl.verification_mode}} +{{/if}} diff --git a/test/packages/kubernetes/data_stream/container/fields/agent.yml b/test/packages/kubernetes/data_stream/container/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/container/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/container/fields/base-fields.yml b/test/packages/kubernetes/data_stream/container/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/container/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/container/fields/ecs.yml b/test/packages/kubernetes/data_stream/container/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/container/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/container/fields/fields.yml b/test/packages/kubernetes/data_stream/container/fields/fields.yml new file mode 100644 index 0000000000..f09af30a34 --- /dev/null +++ b/test/packages/kubernetes/data_stream/container/fields/fields.yml @@ -0,0 +1,199 @@ +- name: kubernetes.container + type: group + fields: + - name: start_time + type: date + description: | + Start time + - name: cpu + type: group + fields: + - name: usage + type: group + fields: + - name: core + type: group + fields: + - name: ns + type: long + metric_type: gauge + description: | + Container CPU Core usage nanoseconds + - name: nanocores + type: long + metric_type: gauge + description: | + CPU used nanocores + - name: node.pct + type: scaled_float + format: percent + unit: percent + metric_type: gauge + description: | + CPU usage as a percentage of the total node allocatable CPU + - name: limit.pct + type: scaled_float + format: percent + unit: percent + metric_type: gauge + description: | + CPU usage as a percentage of the defined limit for the container (or total node allocatable CPU if unlimited) + - name: logs + type: group + fields: + - name: available + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Logs available capacity in bytes + - name: capacity + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Logs total capacity in bytes + - name: used + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Logs used capacity in bytes + - name: inodes + type: group + fields: + - name: count + type: long + metric_type: gauge + description: | + Total available inodes + - name: free + type: long + metric_type: gauge + description: | + Total free inodes + - name: used + type: long + metric_type: gauge + description: | + Total used inodes + - name: memory + type: group + fields: + - name: available + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Total available memory + - name: usage + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Total memory usage + - name: node.pct + type: scaled_float + format: percent + unit: percent + metric_type: gauge + description: | + Memory usage as a percentage of the total node allocatable memory + - name: limit.pct + type: scaled_float + format: percent + unit: percent + metric_type: gauge + description: | + Memory usage as a percentage of the defined limit for the container (or total node allocatable memory if unlimited) + - name: rss + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + RSS memory usage + - name: workingset + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Working set memory usage + - name: pagefaults + type: long + metric_type: counter + description: | + Number of page faults + - name: majorpagefaults + type: long + metric_type: counter + description: | + Number of major page faults + - name: rootfs + type: group + fields: + - name: capacity + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Root filesystem total capacity in bytes + - name: available + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Root filesystem total available in bytes + - name: used + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Root filesystem total used in bytes + - name: inodes + type: group + fields: + - name: used + type: long + metric_type: gauge + description: | + Used inodes diff --git a/test/packages/kubernetes/data_stream/container/manifest.yml b/test/packages/kubernetes/data_stream/container/manifest.yml new file mode 100644 index 0000000000..9aa96fe85a --- /dev/null +++ b/test/packages/kubernetes/data_stream/container/manifest.yml @@ -0,0 +1,44 @@ +title: Kubernetes Container metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: bearer_token_file + type: text + title: Bearer Token File + multi: false + required: true + show_user: true + default: /var/run/secrets/kubernetes.io/serviceaccount/token + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - https://${env.NODE_NAME}:10250 + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + - name: ssl.verification_mode + type: text + title: SSL Verification Mode + multi: false + required: true + show_user: true + default: none + title: Kubernetes Container metrics + description: Collect Kubernetes Container metrics diff --git a/test/packages/kubernetes/data_stream/container/sample_event.json b/test/packages/kubernetes/data_stream/container/sample_event.json new file mode 100644 index 0000000000..2bbe7c0a93 --- /dev/null +++ b/test/packages/kubernetes/data_stream/container/sample_event.json @@ -0,0 +1,150 @@ +{ + "@timestamp": "2020-06-25T12:32:29.748Z", + "kubernetes": { + "namespace": "kube-system", + "node": { + "name": "minikube" + }, + "pod": { + "name": "metricbeat-g9fc6" + }, + "container": { + "rootfs": { + "used": { + "bytes": 61440 + }, + "inodes": { + "used": 17 + }, + "available": { + "bytes": 6724222976 + }, + "capacity": { + "bytes": 17361141760 + } + }, + "logs": { + "used": { + "bytes": 1617920 + }, + "inodes": { + "count": 9768928, + "used": 223910, + "free": 9545018 + }, + "available": { + "bytes": 6724222976 + }, + "capacity": { + "bytes": 17361141760 + } + }, + "start_time": "2020-06-25T07:19:37Z", + "name": "metricbeat", + "cpu": { + "usage": { + "node": { + "pct": 0.00015289625 + }, + "limit": { + "pct": 0.00015289625 + }, + "nanocores": 611585, + "core": { + "ns": 12206519774 + } + } + }, + "memory": { + "pagefaults": 10164, + "majorpagefaults": 528, + "available": { + "bytes": 188600320 + }, + "usage": { + "limit": { + "pct": 0.005608354460473573 + }, + "bytes": 94306304, + "node": { + "pct": 0.005608354460473573 + } + }, + "workingset": { + "bytes": 21114880 + }, + "rss": { + "bytes": 18386944 + } + } + } + }, + "host": { + "containerized": false, + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ], + "hostname": "minikube", + "architecture": "x86_64", + "os": { + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)" + }, + "name": "minikube", + "id": "b0e83d397c054b8a99a431072fe4617b" + }, + "agent": { + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a", + "name": "minikube" + }, + "metricset": { + "period": 10000, + "name": "container" + }, + "service": { + "address": "minikube:10250", + "type": "kubernetes" + }, + "event": { + "dataset": "kubernetes.container", + "module": "kubernetes", + "duration": 11091346 + }, + "ecs": { + "version": "1.5.0" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/controllermanager/_dev/deploy/k8s/.empty b/test/packages/kubernetes/data_stream/controllermanager/_dev/deploy/k8s/.empty new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/packages/kubernetes/data_stream/controllermanager/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/controllermanager/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..60403494f7 --- /dev/null +++ b/test/packages/kubernetes/data_stream/controllermanager/_dev/test/system/test-default-config.yml @@ -0,0 +1,8 @@ +service: kubernetes +data_stream: + vars: + period: 5s + hosts: + - https://0.0.0.0:10257 + bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token + ssl.verification_mode: "none" diff --git a/test/packages/kubernetes/data_stream/controllermanager/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/controllermanager/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..37b11e6e20 --- /dev/null +++ b/test/packages/kubernetes/data_stream/controllermanager/agent/stream/stream.yml.hbs @@ -0,0 +1,11 @@ +metricsets: ["controllermanager"] +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} + +{{#if bearer_token_file}} +bearer_token_file: {{bearer_token_file}} +ssl.verification_mode: {{ssl.verification_mode}} +{{/if}} diff --git a/test/packages/kubernetes/data_stream/controllermanager/fields/agent.yml b/test/packages/kubernetes/data_stream/controllermanager/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/controllermanager/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/controllermanager/fields/base-fields.yml b/test/packages/kubernetes/data_stream/controllermanager/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/controllermanager/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/controllermanager/fields/ecs.yml b/test/packages/kubernetes/data_stream/controllermanager/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/controllermanager/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/controllermanager/fields/fields.yml b/test/packages/kubernetes/data_stream/controllermanager/fields/fields.yml new file mode 100644 index 0000000000..d3a4e449b3 --- /dev/null +++ b/test/packages/kubernetes/data_stream/controllermanager/fields/fields.yml @@ -0,0 +1,153 @@ +- name: kubernetes.controllermanager + type: group + fields: + - name: handler + type: keyword + description: | + Request handler + - name: code + type: keyword + description: | + HTTP code + - name: method + type: keyword + description: | + HTTP method + - name: host + type: keyword + description: | + Request host + - name: name + type: keyword + description: | + Name for the resource + - name: zone + type: keyword + description: | + Infrastructure zone + - name: process + type: group + fields: + - name: cpu.sec + type: double + metric_type: counter + description: CPU seconds + - name: memory.resident.bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: Bytes in resident memory + - name: memory.virtual.bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: Bytes in virtual memory + - name: fds.open.count + type: long + metric_type: gauge + description: Number of open file descriptors + - name: started.sec + type: double + metric_type: gauge + description: Seconds since the process started + - name: http + type: group + fields: + - name: request.duration.us.percentile.* + type: object + description: Request duration microseconds percentiles + - name: request.duration.us.sum + type: double + unit: micros + metric_type: counter + description: Request duration microseconds cumulative sum + - name: request.duration.us.count + type: long + unit: micros + metric_type: counter + description: Request count for duration + - name: request.size.bytes.percentile.* + type: object + description: Request size percentiles + - name: request.size.bytes.sum + type: long + format: bytes + unit: byte + metric_type: counter + description: Request size cumulative sum + - name: request.size.bytes.count + type: long + unit: byte + metric_type: counter + description: Request count for size + - name: response.size.bytes.percentile.* + type: object + description: Response size percentiles + - name: response.size.bytes.sum + type: long + format: bytes + unit: byte + metric_type: counter + description: Response size cumulative sum + - name: response.size.bytes.count + type: long + unit: byte + metric_type: counter + description: Response count + - name: request.count + type: long + metric_type: counter + description: Request count for response + - name: client.request.count + type: long + metric_type: counter + description: | + Number of requests as client + - name: workqueue + type: group + fields: + - name: longestrunning.sec + type: double + metric_type: gauge + description: Longest running processors + - name: unfinished.sec + type: double + metric_type: gauge + description: Unfinished processors + - name: adds.count + type: long + metric_type: counter + description: Workqueue add count + - name: depth.count + type: long + metric_type: gauge + description: Workqueue depth count + - name: retries.count + type: long + metric_type: counter + description: Workqueue number of retries + - name: node.collector + type: group + fields: + - name: eviction.count + type: long + metric_type: counter + description: Number of node evictions + - name: unhealthy.count + type: long + metric_type: gauge + description: Number of unhealthy nodes + - name: count + type: long + metric_type: gauge + description: Number of nodes + - name: health.pct + type: long + metric_type: gauge + description: Percentage of healthy nodes + - name: leader.is_master + type: boolean + description: | + Whether the node is master diff --git a/test/packages/kubernetes/data_stream/controllermanager/manifest.yml b/test/packages/kubernetes/data_stream/controllermanager/manifest.yml new file mode 100644 index 0000000000..2857e48690 --- /dev/null +++ b/test/packages/kubernetes/data_stream/controllermanager/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes Controller Manager metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: bearer_token_file + type: text + title: Bearer Token File + multi: false + required: true + show_user: true + default: /var/run/secrets/kubernetes.io/serviceaccount/token + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - https://0.0.0.0:10257 + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + - name: ssl.verification_mode + type: text + title: SSL Verification Mode + multi: false + required: true + show_user: true + default: none + title: Kubernetes Controller Manager metrics + description: Collect Kubernetes Controller Manager metrics diff --git a/test/packages/kubernetes/data_stream/controllermanager/sample_event.json b/test/packages/kubernetes/data_stream/controllermanager/sample_event.json new file mode 100644 index 0000000000..7aa39a4dd4 --- /dev/null +++ b/test/packages/kubernetes/data_stream/controllermanager/sample_event.json @@ -0,0 +1,93 @@ +{ + "@timestamp": "2020-06-25T12:33:29.643Z", + "kubernetes": { + "controllermanager": { + "workqueue": { + "unfinished": { + "sec": 0 + }, + "adds": { + "count": 0 + }, + "depth": { + "count": 0 + }, + "longestrunning": { + "sec": 0 + }, + "retries": { + "count": 0 + } + }, + "name": "certificate" + } + }, + "event": { + "dataset": "kubernetes.controllermanager", + "module": "kubernetes", + "duration": 8893806 + }, + "ecs": { + "version": "1.5.0" + }, + "host": { + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ], + "hostname": "minikube", + "architecture": "x86_64", + "os": { + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "name": "minikube" + }, + "agent": { + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a", + "name": "minikube", + "type": "metricbeat" + }, + "metricset": { + "period": 10000, + "name": "controllermanager" + }, + "service": { + "address": "localhost:10252", + "type": "kubernetes" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/event/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/event/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..fe1ecc78ce --- /dev/null +++ b/test/packages/kubernetes/data_stream/event/agent/stream/stream.yml.hbs @@ -0,0 +1,6 @@ +metricsets: ["event"] +period: {{period}} +add_metadata: {{add_metadata}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/event/fields/agent.yml b/test/packages/kubernetes/data_stream/event/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/event/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/event/fields/base-fields.yml b/test/packages/kubernetes/data_stream/event/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/event/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/event/fields/ecs.yml b/test/packages/kubernetes/data_stream/event/fields/ecs.yml new file mode 100644 index 0000000000..5c8f7e557a --- /dev/null +++ b/test/packages/kubernetes/data_stream/event/fields/ecs.yml @@ -0,0 +1,10 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/event/fields/fields.yml b/test/packages/kubernetes/data_stream/event/fields/fields.yml new file mode 100644 index 0000000000..c0cdac1f96 --- /dev/null +++ b/test/packages/kubernetes/data_stream/event/fields/fields.yml @@ -0,0 +1,99 @@ +- name: kubernetes.event + type: group + fields: + - name: count + type: long + metric_type: counter + description: | + Count field records the number of times the particular event has occurred + - name: timestamp + type: group + fields: + - name: first_occurrence + type: date + description: | + Timestamp of first occurrence of event + - name: last_occurrence + type: date + description: | + Timestamp of last occurrence of event + - name: message + type: text + description: | + Message recorded for the given event + - name: reason + type: keyword + description: | + Reason recorded for the given event + - name: type + type: keyword + description: | + Type of the given event + - name: source + type: group + fields: + - name: component + type: keyword + description: | + Component from which the event is generated + - name: host + type: keyword + description: | + Node name on which the event is generated + - name: metadata + type: group + fields: + - name: timestamp + type: group + fields: + - name: created + type: date + description: | + Timestamp of creation of the given event + - name: generate_name + type: keyword + description: | + Generate name of the event + - name: name + type: keyword + description: | + Name of the event + - name: namespace + type: keyword + description: | + Namespace in which event was generated + - name: resource_version + type: keyword + description: | + Version of the event resource + - name: uid + type: keyword + description: | + Unique identifier to the event object + - name: self_link + type: keyword + description: | + URL representing the event + - name: involved_object + type: group + fields: + - name: api_version + type: keyword + description: | + API version of the object + - name: kind + type: keyword + description: | + API kind of the object + - name: name + type: keyword + description: | + name of the object + - name: resource_version + type: keyword + description: | + resource version of the object + - name: uid + type: keyword + description: | + UUID version of the object diff --git a/test/packages/kubernetes/data_stream/event/manifest.yml b/test/packages/kubernetes/data_stream/event/manifest.yml new file mode 100644 index 0000000000..bab134bb4f --- /dev/null +++ b/test/packages/kubernetes/data_stream/event/manifest.yml @@ -0,0 +1,29 @@ +title: Kubernetes Event metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + vars: + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + title: Kubernetes Event metrics + description: Collect Kubernetes Event metrics diff --git a/test/packages/kubernetes/data_stream/event/sample_event.json b/test/packages/kubernetes/data_stream/event/sample_event.json new file mode 100644 index 0000000000..01c0c93fcf --- /dev/null +++ b/test/packages/kubernetes/data_stream/event/sample_event.json @@ -0,0 +1,78 @@ +{ + "@timestamp": "2020-06-25T12:30:27.575Z", + "metricset": { + "name": "event" + }, + "agent": { + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc" + }, + "ecs": { + "version": "1.5.0" + }, + "event": { + "dataset": "kubernetes.event", + "module": "kubernetes" + }, + "service": { + "type": "kubernetes" + }, + "kubernetes": { + "event": { + "metadata": { + "uid": "604e39e0-862f-4615-9cec-8cb62299dea3", + "resource_version": "485630", + "timestamp": { + "created": "2020-06-25T07:20:25.000Z" + }, + "name": "monitor.161bb862545e3099", + "namespace": "beats", + "self_link": "/api/v1/namespaces/beats/events/monitor.161bb862545e3099", + "generate_name": "" + }, + "timestamp": { + "first_occurrence": "2020-06-25T07:20:25.000Z", + "last_occurrence": "2020-06-25T12:30:27.000Z" + }, + "message": "Failed to find referenced backend beats/monitor: Elasticsearch.elasticsearch.k8s.elastic.co \"monitor\" not found", + "reason": "AssociationError", + "type": "Warning", + "count": 1861, + "source": { + "host": "", + "component": "kibana-association-controller" + }, + "involved_object": { + "api_version": "kibana.k8s.elastic.co/v1", + "resource_version": "101842", + "name": "monitor", + "kind": "Kibana", + "uid": "45a19de5-5eef-4090-a2d3-dbceb0a28af8" + } + } + }, + "host": { + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + } + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/node/_dev/deploy/k8s/.empty b/test/packages/kubernetes/data_stream/node/_dev/deploy/k8s/.empty new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/packages/kubernetes/data_stream/node/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/node/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..6c69562d49 --- /dev/null +++ b/test/packages/kubernetes/data_stream/node/_dev/test/system/test-default-config.yml @@ -0,0 +1,7 @@ +service: kubernetes +data_stream: + vars: + bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token + period: 5s + hosts: + - https://{{Hostname}}:10250 diff --git a/test/packages/kubernetes/data_stream/node/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/node/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..c4a3cbda5e --- /dev/null +++ b/test/packages/kubernetes/data_stream/node/agent/stream/stream.yml.hbs @@ -0,0 +1,11 @@ +metricsets: ["node"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if bearer_token_file}} +bearer_token_file: {{bearer_token_file}} +ssl.verification_mode: {{ssl.verification_mode}} +{{/if}} diff --git a/test/packages/kubernetes/data_stream/node/fields/agent.yml b/test/packages/kubernetes/data_stream/node/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/node/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/node/fields/base-fields.yml b/test/packages/kubernetes/data_stream/node/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/node/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/node/fields/ecs.yml b/test/packages/kubernetes/data_stream/node/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/node/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/node/fields/fields.yml b/test/packages/kubernetes/data_stream/node/fields/fields.yml new file mode 100644 index 0000000000..dc46f35f2a --- /dev/null +++ b/test/packages/kubernetes/data_stream/node/fields/fields.yml @@ -0,0 +1,198 @@ +- name: kubernetes.node + type: group + fields: + - name: start_time + type: date + description: | + Start time + - name: cpu + type: group + fields: + - name: usage + type: group + fields: + - name: core + type: group + fields: + - name: ns + type: long + metric_type: gauge + description: | + Node CPU Core usage nanoseconds + - name: nanocores + type: long + metric_type: gauge + description: | + CPU used nanocores + - name: memory + type: group + fields: + - name: available + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Total available memory + - name: usage + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Total memory usage + - name: rss + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + RSS memory usage + - name: workingset + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Working set memory usage + - name: pagefaults + type: long + metric_type: counter + description: | + Number of page faults + - name: majorpagefaults + type: long + metric_type: counter + description: | + Number of major page faults + - name: network + type: group + fields: + - name: rx + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: counter + description: | + Received bytes + - name: errors + type: long + description: | + Rx errors + - name: tx + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: counter + description: | + Transmitted bytes + - name: errors + type: long + metric_type: counter + description: | + Tx errors + - name: fs + type: group + fields: + - name: capacity + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Filesystem total capacity in bytes + - name: available + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Filesystem total available in bytes + - name: used + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Filesystem total used in bytes + - name: inodes + type: group + fields: + - name: used + type: long + metric_type: gauge + description: | + Number of used inodes + - name: count + type: long + metric_type: gauge + description: | + Number of inodes + - name: free + type: long + metric_type: gauge + description: | + Number of free inodes + - name: runtime + type: group + fields: + - name: imagefs + type: group + fields: + - name: capacity + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Image filesystem total capacity in bytes + - name: available + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Image filesystem total available in bytes + - name: used + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Image filesystem total used in bytes diff --git a/test/packages/kubernetes/data_stream/node/manifest.yml b/test/packages/kubernetes/data_stream/node/manifest.yml new file mode 100644 index 0000000000..bbf7ebb45d --- /dev/null +++ b/test/packages/kubernetes/data_stream/node/manifest.yml @@ -0,0 +1,44 @@ +title: Kubernetes Node metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: bearer_token_file + type: text + title: Bearer Token File + multi: false + required: true + show_user: true + default: /var/run/secrets/kubernetes.io/serviceaccount/token + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - https://${env.NODE_NAME}:10250 + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + - name: ssl.verification_mode + type: text + title: SSL Verification Mode + multi: false + required: true + show_user: true + default: none + title: Kubernetes Node metrics + description: Collect Kubernetes Node metrics diff --git a/test/packages/kubernetes/data_stream/node/sample_event.json b/test/packages/kubernetes/data_stream/node/sample_event.json new file mode 100644 index 0000000000..3128c86a50 --- /dev/null +++ b/test/packages/kubernetes/data_stream/node/sample_event.json @@ -0,0 +1,148 @@ +{ + "@timestamp": "2020-06-25T12:34:39.723Z", + "event": { + "dataset": "kubernetes.node", + "module": "kubernetes", + "duration": 13042307 + }, + "service": { + "type": "kubernetes", + "address": "minikube:10250" + }, + "host": { + "containerized": false, + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ], + "name": "minikube", + "hostname": "minikube", + "architecture": "x86_64", + "os": { + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81" + }, + "id": "b0e83d397c054b8a99a431072fe4617b" + }, + "metricset": { + "name": "node", + "period": 10000 + }, + "kubernetes": { + "labels": { + "beta_kubernetes_io/os": "linux", + "kubernetes_io/arch": "amd64", + "kubernetes_io/hostname": "minikube", + "kubernetes_io/os": "linux", + "node-role_kubernetes_io/master": "", + "beta_kubernetes_io/arch": "amd64" + }, + "node": { + "memory": { + "available": { + "bytes": 12746428416 + }, + "usage": { + "bytes": 5670916096 + }, + "workingset": { + "bytes": 4068896768 + }, + "rss": { + "bytes": 3252125696 + }, + "pagefaults": 31680, + "majorpagefaults": 0 + }, + "network": { + "rx": { + "bytes": 107077476, + "errors": 0 + }, + "tx": { + "bytes": 67457933, + "errors": 0 + } + }, + "fs": { + "available": { + "bytes": 6655090688 + }, + "capacity": { + "bytes": 17361141760 + }, + "used": { + "bytes": 9689358336 + }, + "inodes": { + "count": 9768928, + "used": 224151, + "free": 9544777 + } + }, + "runtime": { + "imagefs": { + "capacity": { + "bytes": 17361141760 + }, + "used": { + "bytes": 8719928568 + }, + "available": { + "bytes": 6655090688 + } + } + }, + "start_time": "2020-06-25T07:18:38Z", + "name": "minikube", + "cpu": { + "usage": { + "core": { + "ns": 6136184971873 + }, + "nanocores": 455263291 + } + } + } + }, + "agent": { + "name": "minikube", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a" + }, + "ecs": { + "version": "1.5.0" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/pod/_dev/deploy/k8s/.empty b/test/packages/kubernetes/data_stream/pod/_dev/deploy/k8s/.empty new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/packages/kubernetes/data_stream/pod/fields/base-fields.yml b/test/packages/kubernetes/data_stream/pod/fields/base-fields.yml index e851bc3347..d43ffd7964 100644 --- a/test/packages/kubernetes/data_stream/pod/fields/base-fields.yml +++ b/test/packages/kubernetes/data_stream/pod/fields/base-fields.yml @@ -21,7 +21,12 @@ - name: pod.uid type: keyword description: > - Kubernetes Pod UID + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP - name: namespace type: keyword @@ -52,6 +57,13 @@ description: > Kubernetes annotations map + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + - name: replicaset.name type: keyword description: > diff --git a/test/packages/kubernetes/data_stream/pod/fields/ecs.yml b/test/packages/kubernetes/data_stream/pod/fields/ecs.yml index d7ce11b7c1..adbab0f791 100644 --- a/test/packages/kubernetes/data_stream/pod/fields/ecs.yml +++ b/test/packages/kubernetes/data_stream/pod/fields/ecs.yml @@ -7,3 +7,7 @@ - name: service.type type: keyword description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/pod/fields/fields.yml b/test/packages/kubernetes/data_stream/pod/fields/fields.yml index 74a954832b..3073996d3b 100644 --- a/test/packages/kubernetes/data_stream/pod/fields/fields.yml +++ b/test/packages/kubernetes/data_stream/pod/fields/fields.yml @@ -14,10 +14,13 @@ - name: bytes type: long format: bytes + unit: byte + metric_type: counter description: | Received bytes - name: errors type: long + metric_type: counter description: | Rx errors - name: tx @@ -26,10 +29,13 @@ - name: bytes type: long format: bytes + unit: byte + metric_type: counter description: | Transmitted bytes - name: errors type: long + metric_type: counter description: | Tx errors - name: cpu @@ -40,16 +46,22 @@ fields: - name: nanocores type: long + unit: byte + metric_type: gauge description: | CPU used nanocores - name: node.pct type: scaled_float format: percent + unit: percent + metric_type: gauge description: | CPU usage as a percentage of the total node CPU - name: limit.pct type: scaled_float format: percent + unit: percent + metric_type: gauge description: | CPU usage as a percentage of the defined limit for the pod containers (or total node CPU if one or more containers of the pod are unlimited) - name: memory @@ -61,16 +73,22 @@ - name: bytes type: long format: bytes + unit: byte + metric_type: gauge description: | Total memory usage - name: node.pct type: scaled_float format: percent + unit: percent + metric_type: gauge description: | Memory usage as a percentage of the total node allocatable memory - name: limit.pct type: scaled_float format: percent + unit: percent + metric_type: gauge description: | Memory usage as a percentage of the defined limit for the pod containers (or total node allocatable memory if unlimited) - name: available @@ -79,6 +97,8 @@ - name: bytes type: long format: bytes + unit: percent + metric_type: gauge description: | Total memory available - name: working_set @@ -87,6 +107,8 @@ - name: bytes type: long format: bytes + unit: percent + metric_type: gauge description: | Total working set memory - name: rss @@ -95,14 +117,18 @@ - name: bytes type: long format: bytes + unit: percent + metric_type: gauge description: | Total resident set size memory - name: page_faults type: long + metric_type: counter description: | Total page faults - name: major_page_faults type: long + metric_type: counter description: | Total major page faults - name: ip diff --git a/test/packages/kubernetes/data_stream/pod/manifest.yml b/test/packages/kubernetes/data_stream/pod/manifest.yml index 8445751eed..624d996a13 100644 --- a/test/packages/kubernetes/data_stream/pod/manifest.yml +++ b/test/packages/kubernetes/data_stream/pod/manifest.yml @@ -25,7 +25,7 @@ streams: required: true show_user: true default: - - https://${NODE_NAME}:10250 + - https://${env.NODE_NAME}:10250 - name: period type: text title: Period diff --git a/test/packages/kubernetes/data_stream/proxy/_dev/deploy/k8s/.empty b/test/packages/kubernetes/data_stream/proxy/_dev/deploy/k8s/.empty new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/packages/kubernetes/data_stream/proxy/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/proxy/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0807bd3e70 --- /dev/null +++ b/test/packages/kubernetes/data_stream/proxy/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + period: 5s + hosts: + - http://{{Hostname}}:10249 diff --git a/test/packages/kubernetes/data_stream/proxy/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/proxy/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..1723efeec0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/proxy/agent/stream/stream.yml.hbs @@ -0,0 +1,6 @@ +metricsets: ["proxy"] +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} diff --git a/test/packages/kubernetes/data_stream/proxy/fields/agent.yml b/test/packages/kubernetes/data_stream/proxy/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/proxy/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/proxy/fields/base-fields.yml b/test/packages/kubernetes/data_stream/proxy/fields/base-fields.yml new file mode 100644 index 0000000000..7c798f4534 --- /dev/null +++ b/test/packages/kubernetes/data_stream/proxy/fields/base-fields.yml @@ -0,0 +1,12 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. diff --git a/test/packages/kubernetes/data_stream/proxy/fields/ecs.yml b/test/packages/kubernetes/data_stream/proxy/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/proxy/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/proxy/fields/fields.yml b/test/packages/kubernetes/data_stream/proxy/fields/fields.yml new file mode 100644 index 0000000000..5b50c2299f --- /dev/null +++ b/test/packages/kubernetes/data_stream/proxy/fields/fields.yml @@ -0,0 +1,123 @@ +- name: kubernetes.proxy + type: group + fields: + - name: handler + type: keyword + description: | + Request handler + - name: code + type: keyword + description: | + HTTP code + - name: method + type: keyword + description: | + HTTP method + - name: host + type: keyword + description: | + Request host + - name: process + type: group + fields: + - name: cpu.sec + type: double + metric_type: counter + description: CPU seconds + - name: memory.resident.bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: Bytes in resident memory + - name: memory.virtual.bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: Bytes in virtual memory + - name: fds.open.count + type: long + metric_type: gauge + description: Number of open file descriptors + - name: started.sec + type: double + metric_type: gauge + description: Seconds since the process started + - name: http + type: group + fields: + - name: request.duration.us.percentile.* + type: object + description: Request duration microseconds percentiles + - name: request.duration.us.sum + type: double + metric_type: counter + unit: micros + description: Request duration microseconds cumulative sum + - name: request.duration.us.count + type: long + metric_type: counter + unit: micros + description: Request count for duration + - name: request.size.bytes.percentile.* + type: object + description: Request size percentiles + - name: request.size.bytes.sum + type: long + format: bytes + unit: byte + metric_type: counter + description: Request size cumulative sum + - name: request.size.bytes.count + type: long + unit: byte + metric_type: counter + description: Request count for size + - name: response.size.bytes.percentile.* + type: object + description: Response size percentiles + - name: response.size.bytes.sum + type: long + format: bytes + unit: byte + metric_type: counter + description: Response size cumulative sum + - name: response.size.bytes.count + type: long + metric_type: counter + description: Response count + - name: request.count + type: long + metric_type: counter + description: Request count + - name: client.request.count + type: long + metric_type: counter + description: | + Number of requests as client + - name: sync + type: group + fields: + - name: rules.duration.us.sum + type: long + metric_type: counter + description: SyncProxyRules duration, sum of durations in microseconds + - name: rules.duration.us.count + type: long + metric_type: counter + description: SyncProxyRules duration, number of operations + - name: rules.duration.us.bucket.* + type: object + description: SyncProxyRules duration, histogram buckets + - name: networkprogramming.duration.us.sum + type: long + metric_type: counter + description: Network programming duration, sum in microseconds + - name: networkprogramming.duration.us.count + type: long + metric_type: counter + description: Network programming duration, number of operations + - name: networkprogramming.duration.us.bucket.* + type: object + description: Network programming duration, histogram buckets diff --git a/test/packages/kubernetes/data_stream/proxy/manifest.yml b/test/packages/kubernetes/data_stream/proxy/manifest.yml new file mode 100644 index 0000000000..9c7a9e3688 --- /dev/null +++ b/test/packages/kubernetes/data_stream/proxy/manifest.yml @@ -0,0 +1,23 @@ +title: Kubernetes Proxy metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + vars: + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - localhost:10249 + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes Proxy metrics + description: Collect Kubernetes Proxy metrics diff --git a/test/packages/kubernetes/data_stream/proxy/sample_event.json b/test/packages/kubernetes/data_stream/proxy/sample_event.json new file mode 100644 index 0000000000..50ba492ea8 --- /dev/null +++ b/test/packages/kubernetes/data_stream/proxy/sample_event.json @@ -0,0 +1,216 @@ +{ + "@timestamp": "2020-06-25T12:35:29.639Z", + "agent": { + "name": "minikube", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a" + }, + "host": { + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "name": "minikube", + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ], + "hostname": "minikube", + "architecture": "x86_64", + "os": { + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false + }, + "kubernetes": { + "proxy": { + "sync": { + "rules": { + "duration": { + "us": { + "sum": 763620.9329999998, + "count": 18, + "bucket": { + "1000": 0, + "2000": 0, + "4000": 0, + "8000": 0, + "16000": 0, + "32000": 10, + "64000": 16, + "128000": 17, + "256000": 18, + "512000": 18, + "1024000": 18, + "2048000": 18, + "4096000": 18, + "8192000": 18, + "16384000": 18, + "+Inf": 18 + } + } + } + }, + "networkprogramming": { + "duration": { + "us": { + "count": 19, + "bucket": { + "0": 0, + "250000": 4, + "500000": 8, + "1000000": 11, + "2000000": 11, + "3000000": 11, + "4000000": 11, + "5000000": 11, + "6000000": 11, + "7000000": 11, + "8000000": 11, + "9000000": 11, + "10000000": 11, + "11000000": 11, + "12000000": 11, + "13000000": 11, + "14000000": 11, + "15000000": 11, + "16000000": 11, + "17000000": 11, + "18000000": 11, + "19000000": 11, + "20000000": 11, + "21000000": 11, + "22000000": 11, + "23000000": 11, + "24000000": 11, + "25000000": 11, + "26000000": 11, + "27000000": 11, + "28000000": 11, + "29000000": 11, + "30000000": 11, + "31000000": 11, + "32000000": 11, + "33000000": 11, + "34000000": 11, + "35000000": 11, + "36000000": 11, + "37000000": 11, + "38000000": 11, + "39000000": 11, + "40000000": 11, + "41000000": 11, + "42000000": 11, + "43000000": 11, + "44000000": 11, + "45000000": 11, + "46000000": 11, + "47000000": 11, + "48000000": 11, + "49000000": 11, + "50000000": 11, + "51000000": 11, + "52000000": 11, + "53000000": 11, + "54000000": 11, + "55000000": 11, + "56000000": 11, + "57000000": 11, + "58000000": 11, + "59000000": 11, + "60000000": 11, + "65000000": 11, + "70000000": 11, + "75000000": 11, + "80000000": 11, + "85000000": 11, + "90000000": 11, + "95000000": 11, + "100000000": 11, + "105000000": 11, + "110000000": 11, + "115000000": 11, + "120000000": 11, + "150000000": 11, + "180000000": 11, + "210000000": 11, + "240000000": 11, + "270000000": 11, + "300000000": 11, + "+Inf": 19 + }, + "sum": 5571080914163.27 + } + } + } + }, + "process": { + "cpu": { + "sec": 8 + }, + "memory": { + "resident": { + "bytes": 37609472 + }, + "virtual": { + "bytes": 143990784 + } + }, + "started": { + "sec": 1593069580.69 + }, + "fds": { + "open": { + "count": 17 + } + } + } + } + }, + "ecs": { + "version": "1.5.0" + }, + "event": { + "module": "kubernetes", + "duration": 2031254, + "dataset": "kubernetes.proxy" + }, + "metricset": { + "name": "proxy", + "period": 10000 + }, + "service": { + "address": "localhost:10249", + "type": "kubernetes" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/scheduler/_dev/deploy/k8s/.empty b/test/packages/kubernetes/data_stream/scheduler/_dev/deploy/k8s/.empty new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/packages/kubernetes/data_stream/scheduler/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/scheduler/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..c775d296e6 --- /dev/null +++ b/test/packages/kubernetes/data_stream/scheduler/_dev/test/system/test-default-config.yml @@ -0,0 +1,8 @@ +service: kubernetes +data_stream: + vars: + period: 5s + hosts: + - https://0.0.0.0:10259 + bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token + ssl.verification_mode: "none" diff --git a/test/packages/kubernetes/data_stream/scheduler/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/scheduler/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..99a3b27129 --- /dev/null +++ b/test/packages/kubernetes/data_stream/scheduler/agent/stream/stream.yml.hbs @@ -0,0 +1,11 @@ +metricsets: ["scheduler"] +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} + +{{#if bearer_token_file}} +bearer_token_file: {{bearer_token_file}} +ssl.verification_mode: {{ssl.verification_mode}} +{{/if}} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/scheduler/fields/agent.yml b/test/packages/kubernetes/data_stream/scheduler/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/scheduler/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/scheduler/fields/base-fields.yml b/test/packages/kubernetes/data_stream/scheduler/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/scheduler/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/scheduler/fields/ecs.yml b/test/packages/kubernetes/data_stream/scheduler/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/scheduler/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/scheduler/fields/fields.yml b/test/packages/kubernetes/data_stream/scheduler/fields/fields.yml new file mode 100644 index 0000000000..e6f7f48ee7 --- /dev/null +++ b/test/packages/kubernetes/data_stream/scheduler/fields/fields.yml @@ -0,0 +1,156 @@ +- name: kubernetes.scheduler + type: group + fields: + - name: handler + type: keyword + description: | + Request handler + - name: code + type: keyword + description: | + HTTP code + - name: method + type: keyword + description: | + HTTP method + - name: host + type: keyword + description: | + Request host + - name: name + type: keyword + description: | + Name for the resource + - name: result + type: keyword + description: | + Schedule attempt result + - name: operation + type: keyword + description: | + Scheduling operation + - name: process + type: group + fields: + - name: cpu.sec + type: double + metric_type: counter + description: CPU seconds + - name: memory.resident.bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: Bytes in resident memory + - name: memory.virtual.bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: Bytes in virtual memory + - name: fds.open.count + type: long + metric_type: gauge + description: Number of open file descriptors + - name: started.sec + type: double + metric_type: gauge + description: Seconds since the process started + - name: http + type: group + fields: + - name: request.duration.us.percentile.* + type: object + description: Request duration microseconds percentiles + - name: request.duration.us.sum + type: double + metric_type: counter + unit: micros + description: Request duration microseconds cumulative sum + - name: request.duration.us.count + type: long + metric_type: counter + unit: micros + description: Request count for duration + - name: request.size.bytes.percentile.* + type: object + description: Request size percentiles + - name: request.size.bytes.sum + type: long + format: bytes + unit: byte + metric_type: counter + description: Request size cumulative sum + - name: request.size.bytes.count + type: long + unit: byte + metric_type: counter + description: Request count for size + - name: response.size.bytes.percentile.* + type: object + description: Response size percentiles + - name: response.size.bytes.sum + type: long + format: bytes + unit: byte + metric_type: counter + description: Response size cumulative sum + - name: response.size.bytes.count + type: long + metric_type: counter + description: Response count + - name: request.count + type: long + metric_type: counter + description: Request count + - name: client.request.count + type: long + metric_type: counter + description: | + Number of requests as client + - name: leader.is_master + type: boolean + description: | + Whether the node is master + - name: scheduling + type: group + fields: + - name: e2e.duration.us.bucket.* + type: object + description: End to end scheduling duration microseconds + - name: e2e.duration.us.sum + type: long + unit: micros + metric_type: counter + description: End to end scheduling duration microseconds sum + - name: e2e.duration.us.count + type: long + unit: micros + metric_type: counter + description: End to end scheduling count + - name: pod.preemption.victims.bucket.* + type: long + description: Pod preemption victims + - name: pod.preemption.victims.sum + type: long + metric_type: counter + description: Pod preemption victims sum + - name: pod.preemption.victims.count + type: long + metric_type: counter + description: Pod preemption victims count + - name: pod.attempts.count + type: long + metric_type: counter + description: Pod attempts count + - name: duration.seconds.percentile.* + type: object + description: Scheduling duration percentiles + - name: duration.seconds.sum + type: double + metric_type: counter + description: Scheduling duration cumulative sum + - name: duration.seconds.count + type: long + metric_type: counter + description: Scheduling count diff --git a/test/packages/kubernetes/data_stream/scheduler/manifest.yml b/test/packages/kubernetes/data_stream/scheduler/manifest.yml new file mode 100644 index 0000000000..1a4d69fab0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/scheduler/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes Scheduler metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: bearer_token_file + type: text + title: Bearer Token File + multi: false + required: true + show_user: true + default: /var/run/secrets/kubernetes.io/serviceaccount/token + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - https://0.0.0.0:10259 + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + - name: ssl.verification_mode + type: text + title: SSL Verification Mode + multi: false + required: true + show_user: true + default: none + title: Kubernetes Scheduler metrics + description: Collect Kubernetes Scheduler metrics diff --git a/test/packages/kubernetes/data_stream/scheduler/sample_event.json b/test/packages/kubernetes/data_stream/scheduler/sample_event.json new file mode 100644 index 0000000000..b932b1797d --- /dev/null +++ b/test/packages/kubernetes/data_stream/scheduler/sample_event.json @@ -0,0 +1,79 @@ +{ + "@timestamp": "2020-06-25T12:35:59.624Z", + "agent": { + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a", + "name": "minikube", + "type": "metricbeat" + }, + "host": { + "hostname": "minikube", + "architecture": "x86_64", + "os": { + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux" + }, + "name": "minikube", + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ] + }, + "ecs": { + "version": "1.5.0" + }, + "event": { + "duration": 7245648, + "dataset": "kubernetes.scheduler", + "module": "kubernetes" + }, + "metricset": { + "name": "scheduler", + "period": 10000 + }, + "service": { + "address": "localhost:10251", + "type": "kubernetes" + }, + "kubernetes": { + "scheduler": { + "name": "kube-scheduler", + "leader": { + "is_master": true + } + } + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_container/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_container/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_container/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_container/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_container/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..55bf95929d --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_container/agent/stream/stream.yml.hbs @@ -0,0 +1,10 @@ +metricsets: ["state_container"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} diff --git a/test/packages/kubernetes/data_stream/state_container/fields/agent.yml b/test/packages/kubernetes/data_stream/state_container/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_container/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_container/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_container/fields/base-fields.yml new file mode 100644 index 0000000000..2a6f62ace0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_container/fields/base-fields.yml @@ -0,0 +1,95 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: daemonset.name + type: keyword + description: > + Kubernetes daemonset name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_container/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_container/fields/ecs.yml new file mode 100644 index 0000000000..e60b6629bb --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_container/fields/ecs.yml @@ -0,0 +1,16 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: container.runtime + type: keyword + description: Runtime managing this container +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_container/fields/fields.yml b/test/packages/kubernetes/data_stream/state_container/fields/fields.yml new file mode 100644 index 0000000000..858f894d18 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_container/fields/fields.yml @@ -0,0 +1,66 @@ +- name: kubernetes.container + type: group + fields: + - name: id + type: keyword + description: Container id + - name: status + type: group + fields: + - name: phase + type: keyword + description: | + Container phase (running, waiting, terminated) + - name: ready + type: boolean + description: | + Container ready status + - name: restarts + type: integer + metric_type: counter + description: | + Container restarts count + - name: reason + type: keyword + description: | + Waiting (ContainerCreating, CrashLoopBackoff, ErrImagePull, ImagePullBackoff) or termination (Completed, ContainerCannotRun, Error, OOMKilled) reason. + - name: cpu + type: group + fields: + - name: limit.cores + type: float + metric_type: gauge + description: | + Container CPU cores limit + - name: request.cores + type: float + metric_type: gauge + description: | + Container CPU requested cores + - name: limit.nanocores + type: long + metric_type: gauge + description: | + Container CPU nanocores limit + - name: request.nanocores + type: long + metric_type: gauge + description: | + Container CPU requested nanocores + - name: memory + type: group + fields: + - name: limit.bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Container memory limit in bytes + - name: request.bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Container requested memory in bytes diff --git a/test/packages/kubernetes/data_stream/state_container/manifest.yml b/test/packages/kubernetes/data_stream/state_container/manifest.yml new file mode 100644 index 0000000000..de0873b72c --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_container/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes Container metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes Container metrics + description: Collect Kubernetes Container metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_container/sample_event.json b/test/packages/kubernetes/data_stream/state_container/sample_event.json new file mode 100644 index 0000000000..fda9be9310 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_container/sample_event.json @@ -0,0 +1,80 @@ +{ + "@timestamp": "2020-06-25T12:36:34.469Z", + "host": { + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "os": { + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false + }, + "event": { + "dataset": "kubernetes.container", + "module": "kubernetes", + "duration": 8554499 + }, + "kubernetes": { + "node": { + "name": "minikube" + }, + "labels": { + "component": "kube-scheduler", + "tier": "control-plane" + }, + "container": { + "image": "k8s.gcr.io/kube-scheduler:v1.17.0", + "name": "kube-scheduler", + "cpu": { + "request": { + "cores": 0.1 + } + }, + "status": { + "phase": "running", + "ready": true, + "restarts": 10 + }, + "id": "docker://b00b185f2b304a7ece804d1af28eb232f825255f716bcc85ef5bd20d5a4f45d4" + }, + "pod": { + "name": "kube-scheduler-minikube", + "uid": "9cdbd5ea-7638-4e86-a706-a5b222d86f26" + }, + "namespace": "kube-system" + }, + "ecs": { + "version": "1.5.0" + }, + "agent": { + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0" + }, + "container": { + "runtime": "docker", + "id": "b00b185f2b304a7ece804d1af28eb232f825255f716bcc85ef5bd20d5a4f45d4" + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "metricset": { + "name": "state_container", + "period": 10000 + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_cronjob/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_cronjob/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_cronjob/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_cronjob/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_cronjob/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..08b16b3392 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_cronjob/agent/stream/stream.yml.hbs @@ -0,0 +1,11 @@ +metricsets: ["state_cronjob"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} + diff --git a/test/packages/kubernetes/data_stream/state_cronjob/fields/agent.yml b/test/packages/kubernetes/data_stream/state_cronjob/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_cronjob/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_cronjob/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_cronjob/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_cronjob/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_cronjob/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_cronjob/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_cronjob/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_cronjob/fields/fields.yml b/test/packages/kubernetes/data_stream/state_cronjob/fields/fields.yml new file mode 100644 index 0000000000..53d086b4d1 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_cronjob/fields/fields.yml @@ -0,0 +1,39 @@ +- name: kubernetes.cronjob + type: group + fields: + - name: name + type: keyword + description: Cronjob name + - name: schedule + type: keyword + description: Cronjob schedule + - name: concurrency + type: keyword + description: Concurrency policy + - name: active.count + type: long + metric_type: gauge + description: Number of active pods for the cronjob + - name: is_suspended + type: boolean + description: Whether the cronjob is suspended + - name: created.sec + type: double + unit: s + metric_type: gauge + description: Epoch seconds since the cronjob was created + - name: last_schedule.sec + type: double + unit: s + metric_type: gauge + description: Epoch seconds for last cronjob run + - name: next_schedule.sec + type: double + unit: s + metric_type: gauge + description: Epoch seconds for next cronjob run + - name: deadline.sec + type: long + unit: s + metric_type: gauge + description: Deadline seconds after schedule for considering failed diff --git a/test/packages/kubernetes/data_stream/state_cronjob/manifest.yml b/test/packages/kubernetes/data_stream/state_cronjob/manifest.yml new file mode 100644 index 0000000000..fcc893d16c --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_cronjob/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes Cronjob metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes Cronjob metrics + description: Collect Kubernetes Cronjob metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_cronjob/sample_event.json b/test/packages/kubernetes/data_stream/state_cronjob/sample_event.json new file mode 100644 index 0000000000..f297190ba7 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_cronjob/sample_event.json @@ -0,0 +1,66 @@ +{ + "@timestamp": "2020-06-25T12:43:04.384Z", + "metricset": { + "name": "state_cronjob", + "period": 10000 + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "ecs": { + "version": "1.5.0" + }, + "host": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)" + } + }, + "event": { + "dataset": "kubernetes.cronjob", + "module": "kubernetes", + "duration": 9482053 + }, + "kubernetes": { + "namespace": "default", + "cronjob": { + "active": { + "count": 0 + }, + "is_suspended": false, + "name": "hello", + "next_schedule": { + "sec": 1593088980 + }, + "last_schedule": { + "sec": 1593088920 + }, + "created": { + "sec": 1593088862 + } + } + }, + "agent": { + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_daemonset/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_daemonset/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_daemonset/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_daemonset/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_daemonset/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..f7b98418b7 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_daemonset/agent/stream/stream.yml.hbs @@ -0,0 +1,10 @@ +metricsets: ["state_daemonset"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} diff --git a/test/packages/kubernetes/data_stream/state_daemonset/fields/agent.yml b/test/packages/kubernetes/data_stream/state_daemonset/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_daemonset/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_daemonset/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_daemonset/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_daemonset/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_daemonset/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_daemonset/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_daemonset/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_daemonset/fields/fields.yml b/test/packages/kubernetes/data_stream/state_daemonset/fields/fields.yml new file mode 100644 index 0000000000..0e06111729 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_daemonset/fields/fields.yml @@ -0,0 +1,30 @@ +- name: kubernetes.daemonset + type: group + fields: + - name: name + type: keyword + - name: replicas + type: group + description: | + Kubernetes DaemonSet replica metrics + fields: + - name: available + type: long + metric_type: gauge + description: | + The number of available replicas per DaemonSet + - name: desired + type: long + metric_type: gauge + description: | + The desired number of replicas per DaemonSet + - name: ready + type: long + metric_type: gauge + description: | + The number of ready replicas per DaemonSet + - name: unavailable + type: long + metric_type: gauge + description: | + The number of unavailable replicas per DaemonSet diff --git a/test/packages/kubernetes/data_stream/state_daemonset/manifest.yml b/test/packages/kubernetes/data_stream/state_daemonset/manifest.yml new file mode 100644 index 0000000000..a5c761d75c --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_daemonset/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes Deamonset metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes Deamonset metrics + description: Collect Kubernetes Deamonset metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_daemonset/sample_event.json b/test/packages/kubernetes/data_stream/state_daemonset/sample_event.json new file mode 100644 index 0000000000..54b75c87c9 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_daemonset/sample_event.json @@ -0,0 +1,62 @@ +{ + "@timestamp": "2020-06-25T12:37:04.455Z", + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "event": { + "module": "kubernetes", + "duration": 8648138, + "dataset": "kubernetes.daemonset" + }, + "ecs": { + "version": "1.5.0" + }, + "metricset": { + "name": "state_daemonset", + "period": 10000 + }, + "kubernetes": { + "daemonset": { + "name": "metricbeat", + "replicas": { + "available": 1, + "desired": 1, + "ready": 1, + "unavailable": 0 + } + }, + "labels": { + "k8s-app": "metricbeat" + }, + "namespace": "kube-system" + }, + "host": { + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat" + }, + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ] + }, + "agent": { + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_deployment/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_deployment/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_deployment/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_deployment/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_deployment/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..205bc73767 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_deployment/agent/stream/stream.yml.hbs @@ -0,0 +1,10 @@ +metricsets: ["state_deployment"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} diff --git a/test/packages/kubernetes/data_stream/state_deployment/fields/agent.yml b/test/packages/kubernetes/data_stream/state_deployment/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_deployment/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_deployment/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_deployment/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_deployment/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_deployment/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_deployment/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_deployment/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_deployment/fields/fields.yml b/test/packages/kubernetes/data_stream/state_deployment/fields/fields.yml new file mode 100644 index 0000000000..51b6abb87e --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_deployment/fields/fields.yml @@ -0,0 +1,30 @@ +- name: kubernetes.deployment + type: group + fields: + - name: paused + type: boolean + description: | + Kubernetes deployment paused status + - name: replicas + type: group + fields: + - name: desired + type: integer + metric_type: gauge + description: | + Deployment number of desired replicas (spec) + - name: available + type: integer + metric_type: gauge + description: | + Deployment available replicas + - name: unavailable + type: integer + metric_type: gauge + description: | + Deployment unavailable replicas + - name: updated + type: integer + metric_type: gauge + description: | + Deployment updated replicas diff --git a/test/packages/kubernetes/data_stream/state_deployment/manifest.yml b/test/packages/kubernetes/data_stream/state_deployment/manifest.yml new file mode 100644 index 0000000000..7259b870a5 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_deployment/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes Deployment metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes Deployment metrics + description: Collect Kubernetes Deployment metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_deployment/sample_event.json b/test/packages/kubernetes/data_stream/state_deployment/sample_event.json new file mode 100644 index 0000000000..fd79bacedd --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_deployment/sample_event.json @@ -0,0 +1,63 @@ +{ + "@timestamp": "2020-06-25T12:37:04.455Z", + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "event": { + "module": "kubernetes", + "duration": 8648138, + "dataset": "kubernetes.deployment" + }, + "ecs": { + "version": "1.5.0" + }, + "metricset": { + "name": "state_deployment", + "period": 10000 + }, + "kubernetes": { + "deployment": { + "name": "metricbeat", + "replicas": { + "unavailable": 0, + "desired": 1, + "updated": 1, + "available": 1 + }, + "paused": false + }, + "labels": { + "k8s-app": "metricbeat" + }, + "namespace": "kube-system" + }, + "host": { + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat" + }, + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ] + }, + "agent": { + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_job/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_job/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_job/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_job/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_job/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..b9903eff74 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_job/agent/stream/stream.yml.hbs @@ -0,0 +1,7 @@ +metricsets: ["state_job"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} diff --git a/test/packages/kubernetes/data_stream/state_job/fields/agent.yml b/test/packages/kubernetes/data_stream/state_job/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_job/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_job/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_job/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_job/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_job/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_job/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_job/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_job/fields/fields.yml b/test/packages/kubernetes/data_stream/state_job/fields/fields.yml new file mode 100644 index 0000000000..cbf96e3177 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_job/fields/fields.yml @@ -0,0 +1,75 @@ +- name: kubernetes.job + type: group + fields: + - name: name + type: keyword + description: > + The name of the job resource + + - name: pods + type: group + description: > + Pod metrics for the job + + fields: + - name: active + type: long + metric_type: gauge + description: Number of active pods + - name: failed + type: long + metric_type: gauge + description: Number of failed pods + - name: succeeded + type: long + metric_type: gauge + description: Number of successful pods + - name: time + type: group + description: Kubernetes job timestamps + fields: + - name: created + type: date + description: The time at which the job was created + - name: completed + type: date + description: The time at which the job completed + - name: completions + type: group + description: Kubernetes job completion settings + fields: + - name: desired + type: long + metric_type: gauge + description: The configured completion count for the job (Spec) + - name: parallelism + type: group + description: Kubernetes job parallelism settings + fields: + - name: desired + type: long + metric_type: gauge + description: The configured parallelism of the job (Spec) + - name: owner + type: group + description: Kubernetes job owner information + fields: + - name: name + type: keyword + description: The name of the resource that owns this job + - name: kind + type: keyword + description: The kind of resource that owns this job (eg. "CronJob") + - name: is_controller + type: keyword + description: Owner is controller ("true", "false", or `""`) + - name: status + type: group + description: Kubernetes job status information + fields: + - name: complete + type: keyword + description: Whether the job completed ("true", "false", or "unknown") + - name: failed + type: keyword + description: Whether the job failed ("true", "false", or "unknown") diff --git a/test/packages/kubernetes/data_stream/state_job/manifest.yml b/test/packages/kubernetes/data_stream/state_job/manifest.yml new file mode 100644 index 0000000000..cf8fab958b --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_job/manifest.yml @@ -0,0 +1,31 @@ +title: Kubernetes Job metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes Job metrics + description: Collect Kubernetes Job metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_job/sample_event.json b/test/packages/kubernetes/data_stream/state_job/sample_event.json new file mode 100644 index 0000000000..c9ff59627c --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_job/sample_event.json @@ -0,0 +1,72 @@ +{ + "@timestamp": "2020-06-25T12:43:04.384Z", + "metricset": { + "name": "state_job", + "period": 10000 + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "ecs": { + "version": "1.5.0" + }, + "host": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)" + } + }, + "event": { + "dataset": "kubernetes.job", + "module": "kubernetes", + "duration": 9482053 + }, + "kubernetes": { + "job": { + "completions": { + "desired": 1 + }, + "name": "sleep-30-ok-cron-27075645", + "owner": { + "is_controller": "true", + "kind": "CronJob", + "name": "sleep-30-ok-cron" + }, + "parallelism": { + "desired": 1 + }, + "pods": { + "active": 1, + "failed": 0, + "succeeded": 0 + }, + "time": { + "created": "2021-06-24T12:45:00.000Z" + } + }, + "namespace": "default" + }, + "agent": { + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_node/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_node/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_node/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_node/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_node/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..1ddfb5e92f --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_node/agent/stream/stream.yml.hbs @@ -0,0 +1,10 @@ +metricsets: ["state_node"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} diff --git a/test/packages/kubernetes/data_stream/state_node/fields/agent.yml b/test/packages/kubernetes/data_stream/state_node/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_node/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_node/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_node/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_node/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_node/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_node/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_node/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_node/fields/fields.yml b/test/packages/kubernetes/data_stream/state_node/fields/fields.yml new file mode 100644 index 0000000000..c1eb215244 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_node/fields/fields.yml @@ -0,0 +1,69 @@ +- name: kubernetes.node + type: group + fields: + - name: status + type: group + fields: + - name: ready + type: keyword + description: | + Node ready status (true, false or unknown) + - name: unschedulable + type: boolean + description: | + Node unschedulable status + - name: disk_pressure + type: keyword + description: Node DiskPressure status (true, false or unknown) + - name: memory_pressure + type: keyword + description: Node MemoryPressure status (true, false or unknown) + - name: out_of_disk + type: keyword + description: Node OutOfDisk status (true, false or unknown) + - name: pid_pressure + type: keyword + description: Node PIDPressure status (true, false or unknown) + - name: cpu + type: group + fields: + - name: allocatable.cores + type: float + metric_type: gauge + description: | + Node CPU allocatable cores + - name: capacity.cores + type: long + metric_type: gauge + description: | + Node CPU capacity cores + - name: memory + type: group + fields: + - name: allocatable.bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Node allocatable memory in bytes + - name: capacity.bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Node memory capacity in bytes + - name: pod + type: group + fields: + - name: allocatable.total + type: long + metric_type: gauge + description: | + Node allocatable pods + - name: capacity.total + type: long + metric_type: gauge + description: | + Node pod capacity diff --git a/test/packages/kubernetes/data_stream/state_node/manifest.yml b/test/packages/kubernetes/data_stream/state_node/manifest.yml new file mode 100644 index 0000000000..833041bc2c --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_node/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes Node metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes Node metrics + description: Collect Kubernetes Node metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_node/sample_event.json b/test/packages/kubernetes/data_stream/state_node/sample_event.json new file mode 100644 index 0000000000..9b7f451ab6 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_node/sample_event.json @@ -0,0 +1,88 @@ +{ + "@timestamp": "2020-06-25T12:37:44.457Z", + "ecs": { + "version": "1.5.0" + }, + "host": { + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ] + }, + "metricset": { + "name": "state_node", + "period": 10000 + }, + "kubernetes": { + "node": { + "pod": { + "capacity": { + "total": 110 + }, + "allocatable": { + "total": 110 + } + }, + "memory": { + "capacity": { + "bytes": 16815325184 + }, + "allocatable": { + "bytes": 16815325184 + } + }, + "cpu": { + "allocatable": { + "cores": 4 + }, + "capacity": { + "cores": 4 + } + }, + "name": "minikube", + "status": { + "ready": "true", + "unschedulable": false + } + }, + "labels": { + "kubernetes_io/arch": "amd64", + "kubernetes_io/hostname": "minikube", + "kubernetes_io/os": "linux", + "node-role_kubernetes_io/master": "", + "beta_kubernetes_io/arch": "amd64", + "beta_kubernetes_io/os": "linux" + } + }, + "agent": { + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0" + }, + "service": { + "type": "kubernetes", + "address": "kube-state-metrics:8080" + }, + "event": { + "dataset": "kubernetes.node", + "module": "kubernetes", + "duration": 8194220 + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_persistentvolume/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_persistentvolume/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolume/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_persistentvolume/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_persistentvolume/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..b8556ce98b --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolume/agent/stream/stream.yml.hbs @@ -0,0 +1,11 @@ +metricsets: ["state_persistentvolume"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} + diff --git a/test/packages/kubernetes/data_stream/state_persistentvolume/fields/agent.yml b/test/packages/kubernetes/data_stream/state_persistentvolume/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolume/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_persistentvolume/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_persistentvolume/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolume/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_persistentvolume/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_persistentvolume/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolume/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_persistentvolume/fields/fields.yml b/test/packages/kubernetes/data_stream/state_persistentvolume/fields/fields.yml new file mode 100644 index 0000000000..883ddcbd22 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolume/fields/fields.yml @@ -0,0 +1,17 @@ +- name: kubernetes.persistentvolume + type: group + fields: + - name: name + type: keyword + description: Volume name. + - name: capacity.bytes + type: long + unit: byte + metric_type: gauge + description: Volume capacity + - name: phase + type: keyword + description: Volume phase according to kubernetes + - name: storage_class + type: keyword + description: Storage class for the volume diff --git a/test/packages/kubernetes/data_stream/state_persistentvolume/manifest.yml b/test/packages/kubernetes/data_stream/state_persistentvolume/manifest.yml new file mode 100644 index 0000000000..b7ba3f9d54 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolume/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes PersistentVolume metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes PersistentVolume metrics + description: Collect Kubernetes PersistentVolume metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_persistentvolume/sample_event.json b/test/packages/kubernetes/data_stream/state_persistentvolume/sample_event.json new file mode 100644 index 0000000000..cbc6f7f846 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolume/sample_event.json @@ -0,0 +1,60 @@ +{ + "@timestamp": "2020-06-25T12:43:54.412Z", + "ecs": { + "version": "1.5.0" + }, + "event": { + "module": "kubernetes", + "duration": 12149615, + "dataset": "kubernetes.persistentvolume" + }, + "agent": { + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat" + }, + "kubernetes": { + "persistentvolume": { + "capacity": { + "bytes": 10737418240 + }, + "phase": "Bound", + "storage_class": "manual", + "name": "task-pv-volume" + }, + "labels": { + "type": "local" + } + }, + "host": { + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "containerized": false + }, + "metricset": { + "period": 10000, + "name": "state_persistentvolume" + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..06ae236140 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/agent/stream/stream.yml.hbs @@ -0,0 +1,10 @@ +metricsets: ["state_persistentvolumeclaim"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} diff --git a/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/fields/agent.yml b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/fields/fields.yml b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/fields/fields.yml new file mode 100644 index 0000000000..3b5f813d07 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/fields/fields.yml @@ -0,0 +1,23 @@ +- name: kubernetes.persistentvolumeclaim + type: group + fields: + - name: name + type: keyword + description: PVC name. + - name: volume_name + type: keyword + description: Binded volume name. + - name: request_storage.bytes + type: long + unit: byte + metric_type: gauge + description: Requested capacity. + - name: phase + type: keyword + description: PVC phase. + - name: access_mode + type: keyword + description: Access mode. + - name: storage_class + type: keyword + description: Storage class for the PVC. diff --git a/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/manifest.yml b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/manifest.yml new file mode 100644 index 0000000000..2c7264c612 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes PersistentVolumeClaim metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes PersistentVolumeClaim metrics + description: Collect Kubernetes PersistentVolumeClaim metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/sample_event.json b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/sample_event.json new file mode 100644 index 0000000000..0a1204964a --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_persistentvolumeclaim/sample_event.json @@ -0,0 +1,60 @@ +{ + "@timestamp": "2020-06-25T12:44:44.418Z", + "event": { + "dataset": "kubernetes.persistentvolumeclaim", + "module": "kubernetes", + "duration": 5698588 + }, + "metricset": { + "name": "state_persistentvolumeclaim", + "period": 10000 + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "kubernetes": { + "namespace": "default", + "persistentvolumeclaim": { + "phase": "Bound", + "storage_class": "manual", + "volume_name": "task-pv-volume", + "name": "task-pv-claim", + "request_storage": { + "bytes": 3221225472 + }, + "access_mode": "ReadWriteOnce" + } + }, + "agent": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee" + }, + "ecs": { + "version": "1.5.0" + }, + "host": { + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_pod/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_pod/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_pod/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_pod/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_pod/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..b35f091f9a --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_pod/agent/stream/stream.yml.hbs @@ -0,0 +1,10 @@ +metricsets: ["state_pod"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} diff --git a/test/packages/kubernetes/data_stream/state_pod/fields/agent.yml b/test/packages/kubernetes/data_stream/state_pod/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_pod/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_pod/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_pod/fields/base-fields.yml new file mode 100644 index 0000000000..bd2abd6f22 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_pod/fields/base-fields.yml @@ -0,0 +1,95 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: daemonset.name + type: keyword + description: > + Kubernetes daemonset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_pod/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_pod/fields/ecs.yml new file mode 100644 index 0000000000..e60b6629bb --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_pod/fields/ecs.yml @@ -0,0 +1,16 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: container.runtime + type: keyword + description: Runtime managing this container +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_pod/fields/fields.yml b/test/packages/kubernetes/data_stream/state_pod/fields/fields.yml new file mode 100644 index 0000000000..67d4cb4b22 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_pod/fields/fields.yml @@ -0,0 +1,26 @@ +- name: kubernetes.pod + type: group + fields: + - name: ip + type: ip + description: | + Kubernetes pod IP + - name: host_ip + type: ip + description: | + Kubernetes pod host IP + - name: status + type: group + fields: + - name: phase + type: keyword + description: | + Kubernetes pod phase (Running, Pending...) + - name: ready + type: keyword + description: | + Kubernetes pod ready status (true, false or unknown) + - name: scheduled + type: keyword + description: | + Kubernetes pod scheduled status (true, false, unknown) diff --git a/test/packages/kubernetes/data_stream/state_pod/manifest.yml b/test/packages/kubernetes/data_stream/state_pod/manifest.yml new file mode 100644 index 0000000000..f000ea29cd --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_pod/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes Pod metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes Pod metrics + description: Collect Kubernetes Pod metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_pod/sample_event.json b/test/packages/kubernetes/data_stream/state_pod/sample_event.json new file mode 100644 index 0000000000..1b868887bd --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_pod/sample_event.json @@ -0,0 +1,69 @@ +{ + "@timestamp": "2020-06-25T12:38:34.469Z", + "ecs": { + "version": "1.5.0" + }, + "host": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux" + } + }, + "event": { + "duration": 10777415, + "dataset": "kubernetes.pod", + "module": "kubernetes" + }, + "service": { + "type": "kubernetes", + "address": "kube-state-metrics:8080" + }, + "kubernetes": { + "pod": { + "name": "filebeat-dqzzz", + "status": { + "ready": "true", + "scheduled": "true", + "phase": "running" + }, + "host_ip": "192.168.64.10", + "ip": "192.168.64.10", + "uid": "a5f1d3c9-40b6-4182-823b-dd5ff9832279" + }, + "namespace": "kube-system", + "node": { + "name": "minikube" + }, + "labels": { + "controller-revision-hash": "85649b9ddb", + "k8s-app": "filebeat", + "pod-template-generation": "1" + } + }, + "agent": { + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487" + }, + "metricset": { + "period": 10000, + "name": "state_pod" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_replicaset/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_replicaset/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_replicaset/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_replicaset/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_replicaset/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..8b4cdd5828 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_replicaset/agent/stream/stream.yml.hbs @@ -0,0 +1,10 @@ +metricsets: ["state_replicaset"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} diff --git a/test/packages/kubernetes/data_stream/state_replicaset/fields/agent.yml b/test/packages/kubernetes/data_stream/state_replicaset/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_replicaset/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_replicaset/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_replicaset/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_replicaset/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_replicaset/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_replicaset/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_replicaset/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_replicaset/fields/fields.yml b/test/packages/kubernetes/data_stream/state_replicaset/fields/fields.yml new file mode 100644 index 0000000000..40928a7713 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_replicaset/fields/fields.yml @@ -0,0 +1,31 @@ +- name: kubernetes.replicaset + type: group + fields: + - name: replicas + type: group + fields: + - name: available + type: long + metric_type: gauge + description: | + The number of replicas per ReplicaSet + - name: desired + type: long + metric_type: gauge + description: | + The number of replicas per ReplicaSet + - name: ready + type: long + metric_type: gauge + description: | + The number of ready replicas per ReplicaSet + - name: observed + type: long + metric_type: gauge + description: | + The generation observed by the ReplicaSet controller + - name: labeled + type: long + metric_type: gauge + description: | + The number of fully labeled replicas per ReplicaSet diff --git a/test/packages/kubernetes/data_stream/state_replicaset/manifest.yml b/test/packages/kubernetes/data_stream/state_replicaset/manifest.yml new file mode 100644 index 0000000000..6727fd45bd --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_replicaset/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes state_replicaset metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes Replicaset metrics + description: Collect Kubernetes Replicaset metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_replicaset/sample_event.json b/test/packages/kubernetes/data_stream/state_replicaset/sample_event.json new file mode 100644 index 0000000000..e5506863d7 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_replicaset/sample_event.json @@ -0,0 +1,69 @@ +{ + "@timestamp": "2020-06-25T12:38:54.482Z", + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "metricset": { + "period": 10000, + "name": "state_replicaset" + }, + "event": { + "module": "kubernetes", + "duration": 5456128, + "dataset": "kubernetes.replicaset" + }, + "kubernetes": { + "namespace": "kube-system", + "replicaset": { + "name": "nginx-ingress-controller-6fc5bcc8c9", + "replicas": { + "labeled": 1, + "ready": 1, + "available": 1, + "observed": 1, + "desired": 1 + } + }, + "deployment": { + "name": "nginx-ingress-controller" + }, + "labels": { + "app_kubernetes_io/part-of": "kube-system", + "pod-template-hash": "6fc5bcc8c9", + "addonmanager_kubernetes_io/mode": "Reconcile", + "app_kubernetes_io/name": "nginx-ingress-controller" + } + }, + "agent": { + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0" + }, + "ecs": { + "version": "1.5.0" + }, + "host": { + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + }, + "id": "b0e83d397c054b8a99a431072fe4617b" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_resourcequota/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_resourcequota/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_resourcequota/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_resourcequota/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_resourcequota/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..2b7da2e386 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_resourcequota/agent/stream/stream.yml.hbs @@ -0,0 +1,10 @@ +metricsets: ["state_resourcequota"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} diff --git a/test/packages/kubernetes/data_stream/state_resourcequota/fields/agent.yml b/test/packages/kubernetes/data_stream/state_resourcequota/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_resourcequota/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_resourcequota/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_resourcequota/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_resourcequota/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_resourcequota/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_resourcequota/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_resourcequota/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_resourcequota/fields/fields.yml b/test/packages/kubernetes/data_stream/state_resourcequota/fields/fields.yml new file mode 100644 index 0000000000..015f40048a --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_resourcequota/fields/fields.yml @@ -0,0 +1,21 @@ +- name: kubernetes.resourcequota + type: group + fields: + - name: created.sec + type: double + unit: s + metric_type: gauge + description: Epoch seconds since the ResourceQuota was created + - name: quota + type: double + metric_type: gauge + description: Quota informed (hard or used) for the resource + - name: name + type: keyword + description: ResourceQuota name + - name: type + type: keyword + description: Quota information type, `hard` or `used` + - name: resource + type: keyword + description: Resource name the quota applies to diff --git a/test/packages/kubernetes/data_stream/state_resourcequota/manifest.yml b/test/packages/kubernetes/data_stream/state_resourcequota/manifest.yml new file mode 100644 index 0000000000..b5813ff9c1 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_resourcequota/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes ResourceQuota metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes ResourceQuota metrics + description: Collect Kubernetes ResourceQuota metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_resourcequota/sample_event.json b/test/packages/kubernetes/data_stream/state_resourcequota/sample_event.json new file mode 100644 index 0000000000..d3603943f0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_resourcequota/sample_event.json @@ -0,0 +1,56 @@ +{ + "@timestamp": "2020-06-25T12:45:04.416Z", + "metricset": { + "name": "state_resourcequota", + "period": 10000 + }, + "host": { + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ] + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "event": { + "dataset": "kubernetes.resourcequota", + "module": "kubernetes", + "duration": 6324269 + }, + "agent": { + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487" + }, + "ecs": { + "version": "1.5.0" + }, + "kubernetes": { + "namespace": "quota-object-example", + "resourcequota": { + "name": "object-quota-demo", + "resource": "persistentvolumeclaims", + "type": "hard", + "quota": 1 + } + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_service/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_service/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..1343496786 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_service/agent/stream/stream.yml.hbs @@ -0,0 +1,10 @@ +metricsets: ["state_service"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} diff --git a/test/packages/kubernetes/data_stream/state_service/fields/agent.yml b/test/packages/kubernetes/data_stream/state_service/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_service/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_service/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_service/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_service/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_service/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_service/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_service/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_service/fields/fields.yml b/test/packages/kubernetes/data_stream/state_service/fields/fields.yml new file mode 100644 index 0000000000..85100326ea --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_service/fields/fields.yml @@ -0,0 +1,30 @@ +- name: kubernetes.service + type: group + fields: + - name: name + type: keyword + description: Service name. + - name: cluster_ip + type: ip + description: Internal IP for the service. + - name: external_name + type: keyword + description: Service external DNS name + - name: external_ip + type: keyword + description: Service external IP + - name: load_balancer_ip + type: keyword + description: Load Balancer service IP + - name: type + type: keyword + description: Service type + - name: ingress_ip + type: keyword + description: Ingress IP + - name: ingress_hostname + type: keyword + description: Ingress Hostname + - name: created + type: date + description: Service creation date diff --git a/test/packages/kubernetes/data_stream/state_service/manifest.yml b/test/packages/kubernetes/data_stream/state_service/manifest.yml new file mode 100644 index 0000000000..1827542fde --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_service/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes Service metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes Service metrics + description: Collect Kubernetes Service metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_service/sample_event.json b/test/packages/kubernetes/data_stream/state_service/sample_event.json new file mode 100644 index 0000000000..a6cd05ed17 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_service/sample_event.json @@ -0,0 +1,62 @@ +{ + "@timestamp": "2020-06-25T12:39:24.389Z", + "kubernetes": { + "labels": { + "kubernetes_io_minikube_addons_endpoint": "metrics-server", + "kubernetes_io_name": "Metrics-server", + "addonmanager_kubernetes_io_mode": "Reconcile", + "kubernetes_io_minikube_addons": "metrics-server" + }, + "service": { + "name": "metrics-server", + "created": "2020-06-10T09:02:27.000Z", + "cluster_ip": "10.96.124.248", + "type": "ClusterIP" + }, + "namespace": "kube-system" + }, + "event": { + "dataset": "kubernetes.service", + "module": "kubernetes", + "duration": 10966648 + }, + "metricset": { + "name": "state_service", + "period": 10000 + }, + "host": { + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + } + }, + "agent": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee" + }, + "ecs": { + "version": "1.5.0" + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_statefulset/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_statefulset/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_statefulset/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_statefulset/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_statefulset/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..2ecf8cc5f3 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_statefulset/agent/stream/stream.yml.hbs @@ -0,0 +1,10 @@ +metricsets: ["state_statefulset"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} diff --git a/test/packages/kubernetes/data_stream/state_statefulset/fields/agent.yml b/test/packages/kubernetes/data_stream/state_statefulset/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_statefulset/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_statefulset/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_statefulset/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_statefulset/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_statefulset/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_statefulset/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_statefulset/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_statefulset/fields/fields.yml b/test/packages/kubernetes/data_stream/state_statefulset/fields/fields.yml new file mode 100644 index 0000000000..e28adddcaa --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_statefulset/fields/fields.yml @@ -0,0 +1,40 @@ +- name: kubernetes.statefulset + type: group + fields: + - name: created + type: long + metric_type: gauge + description: | + The creation timestamp (epoch) for StatefulSet + - name: replicas + type: group + fields: + - name: observed + type: long + metric_type: gauge + description: | + The number of observed replicas per StatefulSet + - name: desired + type: long + metric_type: gauge + description: | + The number of desired replicas per StatefulSet + - name: ready + type: long + metric_type: gauge + description: > + The number of ready replicas per StatefulSet + + - name: generation + type: group + fields: + - name: observed + type: long + metric_type: gauge + description: | + The observed generation per StatefulSet + - name: desired + type: long + metric_type: gauge + description: | + The desired generation per StatefulSet diff --git a/test/packages/kubernetes/data_stream/state_statefulset/manifest.yml b/test/packages/kubernetes/data_stream/state_statefulset/manifest.yml new file mode 100644 index 0000000000..4bab01e842 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_statefulset/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes StatefulSet metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes StatefulSet metrics + description: Collect Kubernetes StatefulSet metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_statefulset/sample_event.json b/test/packages/kubernetes/data_stream/state_statefulset/sample_event.json new file mode 100644 index 0000000000..9a87bc3df2 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_statefulset/sample_event.json @@ -0,0 +1,62 @@ +{ + "@timestamp": "2020-06-25T12:39:24.389Z", + "kubernetes": { + "namespace": "default", + "statefulset": { + "created": 1511989697, + "generation": { + "desired": 4, + "observed": 2 + }, + "name": "mysql", + "replicas": { + "desired": 5, + "observed": 2 + } + } + }, + "event": { + "dataset": "kubernetes.statefulset", + "module": "kubernetes", + "duration": 10966648 + }, + "metricset": { + "name": "state_statefulset", + "period": 10000 + }, + "host": { + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + } + }, + "agent": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee" + }, + "ecs": { + "version": "1.5.0" + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/state_storageclass/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/state_storageclass/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..0f4bd620c0 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_storageclass/_dev/test/system/test-default-config.yml @@ -0,0 +1,6 @@ +service: kubernetes +data_stream: + vars: + hosts: + # this is the DNS name of the k8s service for kube-state-metrics deployment + - http://kube-state-metrics:8080 diff --git a/test/packages/kubernetes/data_stream/state_storageclass/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/state_storageclass/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..acf1b2e2c6 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_storageclass/agent/stream/stream.yml.hbs @@ -0,0 +1,10 @@ +metricsets: ["state_storageclass"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if leaderelection}} +condition: ${kubernetes_leaderelection.leader} == true +{{/if}} diff --git a/test/packages/kubernetes/data_stream/state_storageclass/fields/agent.yml b/test/packages/kubernetes/data_stream/state_storageclass/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_storageclass/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/state_storageclass/fields/base-fields.yml b/test/packages/kubernetes/data_stream/state_storageclass/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_storageclass/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/state_storageclass/fields/ecs.yml b/test/packages/kubernetes/data_stream/state_storageclass/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_storageclass/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/state_storageclass/fields/fields.yml b/test/packages/kubernetes/data_stream/state_storageclass/fields/fields.yml new file mode 100644 index 0000000000..c190f1dbb1 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_storageclass/fields/fields.yml @@ -0,0 +1,18 @@ +- name: kubernetes.storageclass + type: group + fields: + - name: name + type: keyword + description: Storage class name. + - name: provisioner + type: keyword + description: Volume provisioner for the storage class. + - name: reclaim_policy + type: keyword + description: Reclaim policy for dynamically created volumes + - name: volume_binding_mode + type: keyword + description: Mode for default provisioning and binding + - name: created + type: date + description: Storage class creation date diff --git a/test/packages/kubernetes/data_stream/state_storageclass/manifest.yml b/test/packages/kubernetes/data_stream/state_storageclass/manifest.yml new file mode 100644 index 0000000000..95b7e9d631 --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_storageclass/manifest.yml @@ -0,0 +1,38 @@ +title: Kubernetes StorageClass metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + enabled: false + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - kube-state-metrics:8080 + - name: leaderelection + type: bool + title: Leader Election + multi: false + required: true + show_user: true + default: true + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + title: Kubernetes StorageClass metrics + description: Collect Kubernetes StorageClass metrics from kube_state_metrics diff --git a/test/packages/kubernetes/data_stream/state_storageclass/sample_event.json b/test/packages/kubernetes/data_stream/state_storageclass/sample_event.json new file mode 100644 index 0000000000..de074d381d --- /dev/null +++ b/test/packages/kubernetes/data_stream/state_storageclass/sample_event.json @@ -0,0 +1,59 @@ +{ + "@timestamp": "2020-06-25T12:39:44.399Z", + "agent": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee" + }, + "kubernetes": { + "storageclass": { + "provisioner": "k8s.io/minikube-hostpath", + "reclaim_policy": "Delete", + "volume_binding_mode": "Immediate", + "name": "standard", + "created": "2020-06-10T09:02:27.000Z" + }, + "labels": { + "addonmanager_kubernetes_io_mode": "EnsureExists" + } + }, + "host": { + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ] + }, + "event": { + "module": "kubernetes", + "duration": 5713503, + "dataset": "kubernetes.storageclass" + }, + "metricset": { + "name": "state_storageclass", + "period": 10000 + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "ecs": { + "version": "1.5.0" + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/system/_dev/deploy/k8s/.empty b/test/packages/kubernetes/data_stream/system/_dev/deploy/k8s/.empty new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/packages/kubernetes/data_stream/system/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/system/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..6c69562d49 --- /dev/null +++ b/test/packages/kubernetes/data_stream/system/_dev/test/system/test-default-config.yml @@ -0,0 +1,7 @@ +service: kubernetes +data_stream: + vars: + bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token + period: 5s + hosts: + - https://{{Hostname}}:10250 diff --git a/test/packages/kubernetes/data_stream/system/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/system/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..7157b762c7 --- /dev/null +++ b/test/packages/kubernetes/data_stream/system/agent/stream/stream.yml.hbs @@ -0,0 +1,11 @@ +metricsets: ["system"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if bearer_token_file}} +bearer_token_file: {{bearer_token_file}} +ssl.verification_mode: {{ssl.verification_mode}} +{{/if}} diff --git a/test/packages/kubernetes/data_stream/system/fields/agent.yml b/test/packages/kubernetes/data_stream/system/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/system/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/system/fields/base-fields.yml b/test/packages/kubernetes/data_stream/system/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/system/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/system/fields/ecs.yml b/test/packages/kubernetes/data_stream/system/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/system/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/system/fields/fields.yml b/test/packages/kubernetes/data_stream/system/fields/fields.yml new file mode 100644 index 0000000000..67f7f21604 --- /dev/null +++ b/test/packages/kubernetes/data_stream/system/fields/fields.yml @@ -0,0 +1,73 @@ +- name: kubernetes.system + type: group + fields: + - name: container + type: keyword + description: | + Container name + - name: start_time + type: date + description: | + Start time + - name: cpu + type: group + fields: + - name: usage + type: group + fields: + - name: core + type: group + fields: + - name: ns + type: long + metric_type: gauge + description: | + CPU Core usage nanoseconds + - name: nanocores + type: long + metric_type: gauge + description: | + CPU used nanocores + - name: memory + type: group + fields: + - name: usage + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Total memory usage + - name: rss + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + RSS memory usage + - name: workingset + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Working set memory usage + - name: pagefaults + type: long + metric_type: counter + description: | + Number of page faults + - name: majorpagefaults + type: long + metric_type: counter + description: | + Number of major page faults diff --git a/test/packages/kubernetes/data_stream/system/manifest.yml b/test/packages/kubernetes/data_stream/system/manifest.yml new file mode 100644 index 0000000000..a75c2ab4fc --- /dev/null +++ b/test/packages/kubernetes/data_stream/system/manifest.yml @@ -0,0 +1,44 @@ +title: Kubernetes System metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: bearer_token_file + type: text + title: Bearer Token File + multi: false + required: true + show_user: true + default: /var/run/secrets/kubernetes.io/serviceaccount/token + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - https://${env.NODE_NAME}:10250 + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + - name: ssl.verification_mode + type: text + title: SSL Verification Mode + multi: false + required: true + show_user: true + default: none + title: Kubernetes System metrics + description: Collect Kubernetes system metrics diff --git a/test/packages/kubernetes/data_stream/system/sample_event.json b/test/packages/kubernetes/data_stream/system/sample_event.json new file mode 100644 index 0000000000..1c490ef754 --- /dev/null +++ b/test/packages/kubernetes/data_stream/system/sample_event.json @@ -0,0 +1,101 @@ +{ + "@timestamp": "2020-06-25T12:39:59.647Z", + "service": { + "address": "minikube:10250", + "type": "kubernetes" + }, + "event": { + "duration": 20012905, + "dataset": "kubernetes.system", + "module": "kubernetes" + }, + "ecs": { + "version": "1.5.0" + }, + "host": { + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ], + "hostname": "minikube", + "name": "minikube", + "architecture": "x86_64", + "os": { + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ] + }, + "agent": { + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a", + "name": "minikube", + "type": "metricbeat", + "version": "8.0.0" + }, + "kubernetes": { + "node": { + "name": "minikube" + }, + "system": { + "container": "runtime", + "cpu": { + "usage": { + "nanocores": 35779815, + "core": { + "ns": 530899961233 + } + } + }, + "memory": { + "pagefaults": 12944019, + "majorpagefaults": 99, + "usage": { + "bytes": 198279168 + }, + "workingset": { + "bytes": 178794496 + }, + "rss": { + "bytes": 125259776 + } + }, + "start_time": "2020-06-25T07:19:32Z" + } + }, + "metricset": { + "name": "system", + "period": 10000 + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/data_stream/volume/_dev/deploy/k8s/.empty b/test/packages/kubernetes/data_stream/volume/_dev/deploy/k8s/.empty new file mode 100644 index 0000000000..e69de29bb2 diff --git a/test/packages/kubernetes/data_stream/volume/_dev/test/system/test-default-config.yml b/test/packages/kubernetes/data_stream/volume/_dev/test/system/test-default-config.yml new file mode 100644 index 0000000000..6c69562d49 --- /dev/null +++ b/test/packages/kubernetes/data_stream/volume/_dev/test/system/test-default-config.yml @@ -0,0 +1,7 @@ +service: kubernetes +data_stream: + vars: + bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token + period: 5s + hosts: + - https://{{Hostname}}:10250 diff --git a/test/packages/kubernetes/data_stream/volume/agent/stream/stream.yml.hbs b/test/packages/kubernetes/data_stream/volume/agent/stream/stream.yml.hbs new file mode 100644 index 0000000000..d546a66630 --- /dev/null +++ b/test/packages/kubernetes/data_stream/volume/agent/stream/stream.yml.hbs @@ -0,0 +1,11 @@ +metricsets: ["volume"] +add_metadata: {{add_metadata}} +hosts: +{{#each hosts}} + - {{this}} +{{/each}} +period: {{period}} +{{#if bearer_token_file}} +bearer_token_file: {{bearer_token_file}} +ssl.verification_mode: {{ssl.verification_mode}} +{{/if}} diff --git a/test/packages/kubernetes/data_stream/volume/fields/agent.yml b/test/packages/kubernetes/data_stream/volume/fields/agent.yml new file mode 100644 index 0000000000..da4e652c53 --- /dev/null +++ b/test/packages/kubernetes/data_stream/volume/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/test/packages/kubernetes/data_stream/volume/fields/base-fields.yml b/test/packages/kubernetes/data_stream/volume/fields/base-fields.yml new file mode 100644 index 0000000000..d43ffd7964 --- /dev/null +++ b/test/packages/kubernetes/data_stream/volume/fields/base-fields.yml @@ -0,0 +1,90 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: kubernetes + type: group + fields: + - name: pod.name + type: keyword + description: > + Kubernetes pod name + + - name: pod.uid + type: keyword + description: > + Kubernetes pod UID + + - name: pod.ip + type: ip + description: > + Kubernetes pod IP + + - name: namespace + type: keyword + description: > + Kubernetes namespace + + - name: node.name + type: keyword + description: > + Kubernetes node name + + - name: node.hostname + type: keyword + description: > + Kubernetes hostname as reported by the node’s kernel + + - name: labels.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes labels map + + - name: annotations.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes annotations map + + - name: selectors.* + type: object + object_type: keyword + object_type_mapping_type: "*" + description: > + Kubernetes Service selectors map + + - name: replicaset.name + type: keyword + description: > + Kubernetes replicaset name + + - name: deployment.name + type: keyword + description: > + Kubernetes deployment name + + - name: statefulset.name + type: keyword + description: > + Kubernetes statefulset name + + - name: container.name + type: keyword + description: > + Kubernetes container name + + - name: container.image + type: keyword + description: >- + Kubernetes container image diff --git a/test/packages/kubernetes/data_stream/volume/fields/ecs.yml b/test/packages/kubernetes/data_stream/volume/fields/ecs.yml new file mode 100644 index 0000000000..adbab0f791 --- /dev/null +++ b/test/packages/kubernetes/data_stream/volume/fields/ecs.yml @@ -0,0 +1,13 @@ +- name: ecs.version + type: keyword + description: ECS version +- name: service.address + type: keyword + description: Service address +- name: service.type + type: keyword + description: Service type +- name: orchestrator.cluster.name + external: ecs +- name: orchestrator.cluster.url + external: ecs diff --git a/test/packages/kubernetes/data_stream/volume/fields/fields.yml b/test/packages/kubernetes/data_stream/volume/fields/fields.yml new file mode 100644 index 0000000000..49e510f1b4 --- /dev/null +++ b/test/packages/kubernetes/data_stream/volume/fields/fields.yml @@ -0,0 +1,64 @@ +- name: kubernetes.volume + type: group + fields: + - name: name + type: keyword + description: | + Volume name + - name: fs + type: group + fields: + - name: capacity + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Filesystem total capacity in bytes + - name: available + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Filesystem total available in bytes + - name: used + type: group + fields: + - name: bytes + type: long + format: bytes + unit: byte + metric_type: gauge + description: | + Filesystem total used in bytes + - name: pct + type: long + unit: percent + metric_type: gauge + description: | + Percentage of filesystem total used + - name: inodes + type: group + fields: + - name: used + type: long + metric_type: gauge + description: | + Used inodes + - name: free + type: long + metric_type: gauge + description: | + Free inodes + - name: count + type: long + metric_type: gauge + description: | + Total inodes diff --git a/test/packages/kubernetes/data_stream/volume/manifest.yml b/test/packages/kubernetes/data_stream/volume/manifest.yml new file mode 100644 index 0000000000..d5da20b984 --- /dev/null +++ b/test/packages/kubernetes/data_stream/volume/manifest.yml @@ -0,0 +1,44 @@ +title: Kubernetes Volume metrics +release: experimental +type: metrics +streams: + - input: kubernetes/metrics + vars: + - name: add_metadata + type: bool + title: Add Metadata + multi: false + required: true + show_user: true + default: true + - name: bearer_token_file + type: text + title: Bearer Token File + multi: false + required: true + show_user: true + default: /var/run/secrets/kubernetes.io/serviceaccount/token + - name: hosts + type: text + title: Hosts + multi: true + required: true + show_user: true + default: + - https://${env.NODE_NAME}:10250 + - name: period + type: text + title: Period + multi: false + required: true + show_user: true + default: 10s + - name: ssl.verification_mode + type: text + title: SSL Verification Mode + multi: false + required: true + show_user: true + default: none + title: Kubernetes Volume metrics + description: Collect Kubernetes Volume metrics diff --git a/test/packages/kubernetes/data_stream/volume/sample_event.json b/test/packages/kubernetes/data_stream/volume/sample_event.json new file mode 100644 index 0000000000..700d089f1b --- /dev/null +++ b/test/packages/kubernetes/data_stream/volume/sample_event.json @@ -0,0 +1,99 @@ +{ + "@timestamp": "2020-06-25T12:40:19.649Z", + "ecs": { + "version": "1.5.0" + }, + "metricset": { + "name": "volume", + "period": 10000 + }, + "service": { + "type": "kubernetes", + "address": "minikube:10250" + }, + "kubernetes": { + "pod": { + "name": "metricbeat-g9fc6" + }, + "volume": { + "name": "config", + "fs": { + "inodes": { + "used": 5, + "free": 9549949, + "count": 9768928 + }, + "available": { + "bytes": 7719858176 + }, + "capacity": { + "bytes": 17361141760 + }, + "used": { + "bytes": 12288 + } + } + }, + "namespace": "kube-system", + "node": { + "name": "minikube" + } + }, + "host": { + "architecture": "x86_64", + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "name": "minikube", + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ], + "hostname": "minikube" + }, + "agent": { + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a", + "name": "minikube" + }, + "event": { + "dataset": "kubernetes.volume", + "module": "kubernetes", + "duration": 12481688 + } +} \ No newline at end of file diff --git a/test/packages/kubernetes/docs/README.md b/test/packages/kubernetes/docs/README.md index e182b19427..f7dc53d855 100644 --- a/test/packages/kubernetes/docs/README.md +++ b/test/packages/kubernetes/docs/README.md @@ -1 +1,84 @@ -# Kubernetes package \ No newline at end of file +# Kubernetes integration + +This integration is used to collect metrics from +[Kubernetes clusters](https://kubernetes.io/). + +As one of the main pieces provided for Kubernetes monitoring, this integration is capable of fetching metrics from several components: + +- [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) +- [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) +- [apiserver](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) +- [controller-manager](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/) +- [scheduler](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/) +- [proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) + +Some of the previous components are running on each of the Kubernetes nodes (like `kubelet` or `proxy`) while others provide +a single cluster-wide endpoint. This is important to determine the optimal configuration and running strategy +for the different datasets included in the integration. + +For a complete reference on how to configure and run this package on Kubernetes as part of a `DaemonSet` and a `Deployment`, +there's a complete [example manifest](https://github.com/elastic/beats/blob/master/deploy/kubernetes/elastic-agent-kubernetes.yaml) +available. + +#### Kubernetes endpoints and metricsets + +Kubernetes module is a bit complex as its internal datasets require access to a wide variety of endpoints. + +This section highlights and introduces some groups of datasets with similar endpoint access needs. +For more details on the datasets see `configuration example` and the `datasets` sections below. + + +#### node / system / pod / container / module / volume + +The datasets `container`, `node`, `pod`, `system` and `volume` require access to the `kubelet endpoint` in each of +the Kubernetes nodes, hence it's recommended to include them as part +of an `Agent DaemonSet` or standalone Agents running on the hosts. + +Depending on the version and configuration of Kubernetes nodes, `kubelet` might provide a read only http port (typically 10255), +which is used in some configuration examples. But in general, and lately, this endpoint requires SSL (`https`) access +(to port 10250 by default) and token based authentication. + + +##### state_* and event + +All datasets with the `state_` prefix require `hosts` field pointing to `kube-state-metrics` +service within the cluster. As the service provides cluster-wide metrics, there's no need to fetch them per node, +hence the recommendation is to run these datasets as part of an `Agent Deployment` with one only replica. + +Note: Kube-state-metrics is not deployed by default in Kubernetes. For these cases the instructions for its +deployment are available [here](https://github.com/kubernetes/kube-state-metrics#kubernetes-deployment). +Generally `kube-state-metrics` runs a `Deployment` and is accessible via a service called `kube-state-metrics` on +`kube-system` namespace, which will be the service to use in our configuration. + +state_* datasets are not enabled by default. + +#### apiserver + +The apiserver dataset requires access to the Kubernetes API, which should be easily available in all Kubernetes +environments. Depending on the Kubernetes configuration, the API access might require SSL (`https`) and token +based authentication. + +#### proxy + +The proxy dataset requires access to the proxy endpoint in each of Kubernetes nodes, hence it's recommended +to configure it as a part of an `Agent DaemonSet`. + +#### scheduler and controllermanager + +These datasets require access to the Kubernetes `controller-manager` and `scheduler` endpoints. By default, these pods +run only on master nodes, and they are not exposed via a Service, but there are different strategies +available for its configuration: + +- Create `Kubernetes Services` to make `kube-controller-manager` and `kube-scheduler` available and configure + the datasets to point to these services as part of an `Agent Deployment`. +- Run these datasets as part an `Agent Daemonset` (with HostNetwork setting) with a `nodeSelector` to only run on Master nodes. + +These datasets are not enabled by default. + +Note: In some "As a Service" Kubernetes implementations, like `GKE`, the master nodes or even the pods running on +the masters won't be visible. In these cases it won't be possible to use `scheduler` and `controllermanager` metricsets. + +## Compatibility + +The Kubernetes package is tested with Kubernetes 1.13.x, 1.14.x, 1.15.x, 1.16.x, 1.17.x, and 1.18.x + diff --git a/test/packages/kubernetes/docs/events.md b/test/packages/kubernetes/docs/events.md new file mode 100644 index 0000000000..1ea6c84f6c --- /dev/null +++ b/test/packages/kubernetes/docs/events.md @@ -0,0 +1,171 @@ +# events + +## Metrics + +### event + +This is the `event` dataset of the Kubernetes package. It collects Kubernetes events +related metrics. + +If Leader Election is activated (default behaviour) only the `elastic agent` which holds the leadership lock +will retrieve events related metrics. +This is relevant in multi-node kubernetes cluster and prevents duplicate data. + +An example event for `event` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:30:27.575Z", + "metricset": { + "name": "event" + }, + "agent": { + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc" + }, + "ecs": { + "version": "1.5.0" + }, + "event": { + "dataset": "kubernetes.event", + "module": "kubernetes" + }, + "service": { + "type": "kubernetes" + }, + "kubernetes": { + "event": { + "metadata": { + "uid": "604e39e0-862f-4615-9cec-8cb62299dea3", + "resource_version": "485630", + "timestamp": { + "created": "2020-06-25T07:20:25.000Z" + }, + "name": "monitor.161bb862545e3099", + "namespace": "beats", + "self_link": "/api/v1/namespaces/beats/events/monitor.161bb862545e3099", + "generate_name": "" + }, + "timestamp": { + "first_occurrence": "2020-06-25T07:20:25.000Z", + "last_occurrence": "2020-06-25T12:30:27.000Z" + }, + "message": "Failed to find referenced backend beats/monitor: Elasticsearch.elasticsearch.k8s.elastic.co \"monitor\" not found", + "reason": "AssociationError", + "type": "Warning", + "count": 1861, + "source": { + "host": "", + "component": "kibana-association-controller" + }, + "involved_object": { + "api_version": "kibana.k8s.elastic.co/v1", + "resource_version": "101842", + "name": "monitor", + "kind": "Kibana", + "uid": "45a19de5-5eef-4090-a2d3-dbceb0a28af8" + } + } + }, + "host": { + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + } + } +} +``` + +**Exported fields** + +| Field | Description | Type | Metric Type | +|---|---|---|---| +| @timestamp | Event timestamp. | date | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | +| cloud.instance.name | Instance name of the host machine. | keyword | | +| cloud.machine.type | Machine type of the host machine. | keyword | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | +| cloud.region | Region in which this host is running. | keyword | | +| container.id | Unique container id. | keyword | | +| container.image.name | Name of the image the container was built on. | keyword | | +| container.labels | Image labels. | object | | +| container.name | Container name. | keyword | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | +| data_stream.type | Data stream type. | constant_keyword | | +| ecs.version | ECS version | keyword | | +| host.architecture | Operating system architecture. | keyword | | +| host.containerized | If the host is a container. | boolean | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | +| host.ip | Host ip addresses. | ip | | +| host.mac | Host mac addresses. | keyword | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | +| host.os.build | OS build information. | keyword | | +| host.os.codename | OS codename, if any. | keyword | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | +| host.os.name | Operating system name, without the version. | keyword | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | +| host.os.version | Operating system version as a raw string. | keyword | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | +| kubernetes.container.image | Kubernetes container image | keyword | | +| kubernetes.container.name | Kubernetes container name | keyword | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | +| kubernetes.event.count | Count field records the number of times the particular event has occurred | long | counter | +| kubernetes.event.involved_object.api_version | API version of the object | keyword | | +| kubernetes.event.involved_object.kind | API kind of the object | keyword | | +| kubernetes.event.involved_object.name | name of the object | keyword | | +| kubernetes.event.involved_object.resource_version | resource version of the object | keyword | | +| kubernetes.event.involved_object.uid | UUID version of the object | keyword | | +| kubernetes.event.message | Message recorded for the given event | text | | +| kubernetes.event.metadata.generate_name | Generate name of the event | keyword | | +| kubernetes.event.metadata.name | Name of the event | keyword | | +| kubernetes.event.metadata.namespace | Namespace in which event was generated | keyword | | +| kubernetes.event.metadata.resource_version | Version of the event resource | keyword | | +| kubernetes.event.metadata.self_link | URL representing the event | keyword | | +| kubernetes.event.metadata.timestamp.created | Timestamp of creation of the given event | date | | +| kubernetes.event.metadata.uid | Unique identifier to the event object | keyword | | +| kubernetes.event.reason | Reason recorded for the given event | keyword | | +| kubernetes.event.source.component | Component from which the event is generated | keyword | | +| kubernetes.event.source.host | Node name on which the event is generated | keyword | | +| kubernetes.event.timestamp.first_occurrence | Timestamp of first occurrence of event | date | | +| kubernetes.event.timestamp.last_occurrence | Timestamp of last occurrence of event | date | | +| kubernetes.event.type | Type of the given event | keyword | | +| kubernetes.labels.\* | Kubernetes labels map | object | | +| kubernetes.namespace | Kubernetes namespace | keyword | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | +| kubernetes.node.name | Kubernetes node name | keyword | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | +| service.type | Service type | keyword | | diff --git a/test/packages/kubernetes/docs/kube-apiserver.md b/test/packages/kubernetes/docs/kube-apiserver.md new file mode 100644 index 0000000000..52a5bec309 --- /dev/null +++ b/test/packages/kubernetes/docs/kube-apiserver.md @@ -0,0 +1,186 @@ +# kube-apiserver + +## Metrics + +### apiserver + +This is the `apiserver` dataset of the Kubernetes package, in charge of retrieving metrics +from the Kubernetes API (available at `/metrics`). + +This metricset needs access to the `apiserver` component of Kubernetes, accessible typically +by any POD via the `kubernetes.default` service or via environment +variables (`KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT`). + +If Leader Election is activated (default behaviour) only the `elastic agent` which holds the leadership lock +will retrieve metrics from the `apiserver`. +This is relevant in multi-node kubernetes cluster and prevents duplicate data. + +When the API uses https, the pod will need to authenticate using its default token and trust +the server using the appropriate CA file. + +Configuration example using https and token based authentication: + + +In order to access the `/metrics` path of the API service, some Kubernetes environments might +require the following permission to be added to a ClusterRole. + +```yaml +rules: +- nonResourceURLs: + - /metrics + verbs: + - get +``` + +An example event for `apiserver` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:30:34.616Z", + "metricset": { + "name": "apiserver", + "period": 30000 + }, + "service": { + "address": "10.96.0.1:443", + "type": "kubernetes" + }, + "event": { + "dataset": "kubernetes.apiserver", + "module": "kubernetes", + "duration": 114780772 + }, + "kubernetes": { + "apiserver": { + "request": { + "client": "metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format", + "version": "v1", + "count": 3, + "scope": "cluster", + "content_type": "application/vnd.kubernetes.protobuf", + "code": "200", + "verb": "LIST", + "component": "apiserver", + "resource": "nodes" + } + } + }, + "ecs": { + "version": "1.5.0" + }, + "agent": { + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat" + }, + "host": { + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "os": { + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)" + } + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.apiserver.audit.event.count | Number of audit events | long | | counter | +| kubernetes.apiserver.audit.rejected.count | Number of audit rejected events | long | | counter | +| kubernetes.apiserver.client.request.count | Number of requests as client | long | | counter | +| kubernetes.apiserver.etcd.object.count | Number of kubernetes objects at etcd | long | | gauge | +| kubernetes.apiserver.http.request.count | Request count for response | long | | counter | +| kubernetes.apiserver.http.request.duration.us.count | Request count for duration | long | micros | counter | +| kubernetes.apiserver.http.request.duration.us.percentile.\* | Request duration microseconds percentiles | object | | | +| kubernetes.apiserver.http.request.duration.us.sum | Request duration microseconds cumulative sum | double | micros | counter | +| kubernetes.apiserver.http.request.size.bytes.count | Request count for size | long | byte | counter | +| kubernetes.apiserver.http.request.size.bytes.percentile.\* | Request size percentiles | object | | | +| kubernetes.apiserver.http.request.size.bytes.sum | Request size cumulative sum | long | byte | counter | +| kubernetes.apiserver.http.response.size.bytes.count | Response count | long | | counter | +| kubernetes.apiserver.http.response.size.bytes.percentile.\* | Response size percentiles | object | | | +| kubernetes.apiserver.http.response.size.bytes.sum | Response size cumulative sum | long | byte | counter | +| kubernetes.apiserver.process.cpu.sec | CPU seconds | double | | counter | +| kubernetes.apiserver.process.fds.open.count | Number of open file descriptors | long | | gauge | +| kubernetes.apiserver.process.memory.resident.bytes | Bytes in resident memory | long | byte | gauge | +| kubernetes.apiserver.process.memory.virtual.bytes | Bytes in virtual memory | long | byte | gauge | +| kubernetes.apiserver.process.started.sec | Seconds since the process started | double | | gauge | +| kubernetes.apiserver.request.client | Client executing requests | keyword | | | +| kubernetes.apiserver.request.code | HTTP code | keyword | | | +| kubernetes.apiserver.request.component | Component handling the request | keyword | | | +| kubernetes.apiserver.request.content_type | Request HTTP content type | keyword | | | +| kubernetes.apiserver.request.count | Number of requests | long | | counter | +| kubernetes.apiserver.request.current.count | Inflight requests | long | | counter | +| kubernetes.apiserver.request.dry_run | Wether the request uses dry run | keyword | | | +| kubernetes.apiserver.request.duration.us.bucket.\* | Request duration, histogram buckets | object | | | +| kubernetes.apiserver.request.duration.us.count | Request duration, number of operations | long | | counter | +| kubernetes.apiserver.request.duration.us.sum | Request duration, sum in microseconds | long | | counter | +| kubernetes.apiserver.request.group | API group for the resource | keyword | | | +| kubernetes.apiserver.request.handler | Request handler | keyword | | | +| kubernetes.apiserver.request.host | Request host | keyword | | | +| kubernetes.apiserver.request.kind | Kind of request | keyword | | | +| kubernetes.apiserver.request.latency.bucket.\* | Request latency histogram buckets | object | | | +| kubernetes.apiserver.request.latency.count | Request latency, number of requests | long | | counter | +| kubernetes.apiserver.request.latency.sum | Requests latency, sum of latencies in microseconds | long | | counter | +| kubernetes.apiserver.request.longrunning.count | Number of requests active long running requests | long | | counter | +| kubernetes.apiserver.request.method | HTTP method | keyword | | | +| kubernetes.apiserver.request.resource | Requested resource | keyword | | | +| kubernetes.apiserver.request.scope | Request scope (cluster, namespace, resource) | keyword | | | +| kubernetes.apiserver.request.subresource | Requested subresource | keyword | | | +| kubernetes.apiserver.request.verb | HTTP verb | keyword | | | +| kubernetes.apiserver.request.version | version for the group | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | diff --git a/test/packages/kubernetes/docs/kube-controller-manager.md b/test/packages/kubernetes/docs/kube-controller-manager.md new file mode 100644 index 0000000000..f987fdc6bb --- /dev/null +++ b/test/packages/kubernetes/docs/kube-controller-manager.md @@ -0,0 +1,195 @@ +# kube-controller-manager + +## Metrics + +### controllermanager + +This is the `controllermanager` dataset for the Kubernetes package. It collects from +Kubernetes controller component `metrics` endpoint. + +An example event for `controllermanager` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:33:29.643Z", + "kubernetes": { + "controllermanager": { + "workqueue": { + "unfinished": { + "sec": 0 + }, + "adds": { + "count": 0 + }, + "depth": { + "count": 0 + }, + "longestrunning": { + "sec": 0 + }, + "retries": { + "count": 0 + } + }, + "name": "certificate" + } + }, + "event": { + "dataset": "kubernetes.controllermanager", + "module": "kubernetes", + "duration": 8893806 + }, + "ecs": { + "version": "1.5.0" + }, + "host": { + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ], + "hostname": "minikube", + "architecture": "x86_64", + "os": { + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "name": "minikube" + }, + "agent": { + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a", + "name": "minikube", + "type": "metricbeat" + }, + "metricset": { + "period": 10000, + "name": "controllermanager" + }, + "service": { + "address": "localhost:10252", + "type": "kubernetes" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.controllermanager.client.request.count | Number of requests as client | long | | counter | +| kubernetes.controllermanager.code | HTTP code | keyword | | | +| kubernetes.controllermanager.handler | Request handler | keyword | | | +| kubernetes.controllermanager.host | Request host | keyword | | | +| kubernetes.controllermanager.http.request.count | Request count for response | long | | counter | +| kubernetes.controllermanager.http.request.duration.us.count | Request count for duration | long | micros | counter | +| kubernetes.controllermanager.http.request.duration.us.percentile.\* | Request duration microseconds percentiles | object | | | +| kubernetes.controllermanager.http.request.duration.us.sum | Request duration microseconds cumulative sum | double | micros | counter | +| kubernetes.controllermanager.http.request.size.bytes.count | Request count for size | long | byte | counter | +| kubernetes.controllermanager.http.request.size.bytes.percentile.\* | Request size percentiles | object | | | +| kubernetes.controllermanager.http.request.size.bytes.sum | Request size cumulative sum | long | byte | counter | +| kubernetes.controllermanager.http.response.size.bytes.count | Response count | long | byte | counter | +| kubernetes.controllermanager.http.response.size.bytes.percentile.\* | Response size percentiles | object | | | +| kubernetes.controllermanager.http.response.size.bytes.sum | Response size cumulative sum | long | byte | counter | +| kubernetes.controllermanager.leader.is_master | Whether the node is master | boolean | | | +| kubernetes.controllermanager.method | HTTP method | keyword | | | +| kubernetes.controllermanager.name | Name for the resource | keyword | | | +| kubernetes.controllermanager.node.collector.count | Number of nodes | long | | gauge | +| kubernetes.controllermanager.node.collector.eviction.count | Number of node evictions | long | | counter | +| kubernetes.controllermanager.node.collector.health.pct | Percentage of healthy nodes | long | | gauge | +| kubernetes.controllermanager.node.collector.unhealthy.count | Number of unhealthy nodes | long | | gauge | +| kubernetes.controllermanager.process.cpu.sec | CPU seconds | double | | counter | +| kubernetes.controllermanager.process.fds.open.count | Number of open file descriptors | long | | gauge | +| kubernetes.controllermanager.process.memory.resident.bytes | Bytes in resident memory | long | byte | gauge | +| kubernetes.controllermanager.process.memory.virtual.bytes | Bytes in virtual memory | long | byte | gauge | +| kubernetes.controllermanager.process.started.sec | Seconds since the process started | double | | gauge | +| kubernetes.controllermanager.workqueue.adds.count | Workqueue add count | long | | counter | +| kubernetes.controllermanager.workqueue.depth.count | Workqueue depth count | long | | gauge | +| kubernetes.controllermanager.workqueue.longestrunning.sec | Longest running processors | double | | gauge | +| kubernetes.controllermanager.workqueue.retries.count | Workqueue number of retries | long | | counter | +| kubernetes.controllermanager.workqueue.unfinished.sec | Unfinished processors | double | | gauge | +| kubernetes.controllermanager.zone | Infrastructure zone | keyword | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | diff --git a/test/packages/kubernetes/docs/kube-proxy.md b/test/packages/kubernetes/docs/kube-proxy.md new file mode 100644 index 0000000000..675550ee6c --- /dev/null +++ b/test/packages/kubernetes/docs/kube-proxy.md @@ -0,0 +1,298 @@ +# kube-proxy + +## Metrics + +### proxy + +This is the `proxy` dataset of the Kubernetes package. It collects metrics +from Kubernetes Proxy component. + +An example event for `proxy` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:35:29.639Z", + "agent": { + "name": "minikube", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a" + }, + "host": { + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "name": "minikube", + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ], + "hostname": "minikube", + "architecture": "x86_64", + "os": { + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false + }, + "kubernetes": { + "proxy": { + "sync": { + "rules": { + "duration": { + "us": { + "sum": 763620.9329999998, + "count": 18, + "bucket": { + "1000": 0, + "2000": 0, + "4000": 0, + "8000": 0, + "16000": 0, + "32000": 10, + "64000": 16, + "128000": 17, + "256000": 18, + "512000": 18, + "1024000": 18, + "2048000": 18, + "4096000": 18, + "8192000": 18, + "16384000": 18, + "+Inf": 18 + } + } + } + }, + "networkprogramming": { + "duration": { + "us": { + "count": 19, + "bucket": { + "0": 0, + "250000": 4, + "500000": 8, + "1000000": 11, + "2000000": 11, + "3000000": 11, + "4000000": 11, + "5000000": 11, + "6000000": 11, + "7000000": 11, + "8000000": 11, + "9000000": 11, + "10000000": 11, + "11000000": 11, + "12000000": 11, + "13000000": 11, + "14000000": 11, + "15000000": 11, + "16000000": 11, + "17000000": 11, + "18000000": 11, + "19000000": 11, + "20000000": 11, + "21000000": 11, + "22000000": 11, + "23000000": 11, + "24000000": 11, + "25000000": 11, + "26000000": 11, + "27000000": 11, + "28000000": 11, + "29000000": 11, + "30000000": 11, + "31000000": 11, + "32000000": 11, + "33000000": 11, + "34000000": 11, + "35000000": 11, + "36000000": 11, + "37000000": 11, + "38000000": 11, + "39000000": 11, + "40000000": 11, + "41000000": 11, + "42000000": 11, + "43000000": 11, + "44000000": 11, + "45000000": 11, + "46000000": 11, + "47000000": 11, + "48000000": 11, + "49000000": 11, + "50000000": 11, + "51000000": 11, + "52000000": 11, + "53000000": 11, + "54000000": 11, + "55000000": 11, + "56000000": 11, + "57000000": 11, + "58000000": 11, + "59000000": 11, + "60000000": 11, + "65000000": 11, + "70000000": 11, + "75000000": 11, + "80000000": 11, + "85000000": 11, + "90000000": 11, + "95000000": 11, + "100000000": 11, + "105000000": 11, + "110000000": 11, + "115000000": 11, + "120000000": 11, + "150000000": 11, + "180000000": 11, + "210000000": 11, + "240000000": 11, + "270000000": 11, + "300000000": 11, + "+Inf": 19 + }, + "sum": 5571080914163.27 + } + } + } + }, + "process": { + "cpu": { + "sec": 8 + }, + "memory": { + "resident": { + "bytes": 37609472 + }, + "virtual": { + "bytes": 143990784 + } + }, + "started": { + "sec": 1593069580.69 + }, + "fds": { + "open": { + "count": 17 + } + } + } + } + }, + "ecs": { + "version": "1.5.0" + }, + "event": { + "module": "kubernetes", + "duration": 2031254, + "dataset": "kubernetes.proxy" + }, + "metricset": { + "name": "proxy", + "period": 10000 + }, + "service": { + "address": "localhost:10249", + "type": "kubernetes" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.proxy.client.request.count | Number of requests as client | long | | counter | +| kubernetes.proxy.code | HTTP code | keyword | | | +| kubernetes.proxy.handler | Request handler | keyword | | | +| kubernetes.proxy.host | Request host | keyword | | | +| kubernetes.proxy.http.request.count | Request count | long | | counter | +| kubernetes.proxy.http.request.duration.us.count | Request count for duration | long | micros | counter | +| kubernetes.proxy.http.request.duration.us.percentile.\* | Request duration microseconds percentiles | object | | | +| kubernetes.proxy.http.request.duration.us.sum | Request duration microseconds cumulative sum | double | micros | counter | +| kubernetes.proxy.http.request.size.bytes.count | Request count for size | long | byte | counter | +| kubernetes.proxy.http.request.size.bytes.percentile.\* | Request size percentiles | object | | | +| kubernetes.proxy.http.request.size.bytes.sum | Request size cumulative sum | long | byte | counter | +| kubernetes.proxy.http.response.size.bytes.count | Response count | long | | counter | +| kubernetes.proxy.http.response.size.bytes.percentile.\* | Response size percentiles | object | | | +| kubernetes.proxy.http.response.size.bytes.sum | Response size cumulative sum | long | byte | counter | +| kubernetes.proxy.method | HTTP method | keyword | | | +| kubernetes.proxy.process.cpu.sec | CPU seconds | double | | counter | +| kubernetes.proxy.process.fds.open.count | Number of open file descriptors | long | | gauge | +| kubernetes.proxy.process.memory.resident.bytes | Bytes in resident memory | long | byte | gauge | +| kubernetes.proxy.process.memory.virtual.bytes | Bytes in virtual memory | long | byte | gauge | +| kubernetes.proxy.process.started.sec | Seconds since the process started | double | | gauge | +| kubernetes.proxy.sync.networkprogramming.duration.us.bucket.\* | Network programming duration, histogram buckets | object | | | +| kubernetes.proxy.sync.networkprogramming.duration.us.count | Network programming duration, number of operations | long | | counter | +| kubernetes.proxy.sync.networkprogramming.duration.us.sum | Network programming duration, sum in microseconds | long | | counter | +| kubernetes.proxy.sync.rules.duration.us.bucket.\* | SyncProxyRules duration, histogram buckets | object | | | +| kubernetes.proxy.sync.rules.duration.us.count | SyncProxyRules duration, number of operations | long | | counter | +| kubernetes.proxy.sync.rules.duration.us.sum | SyncProxyRules duration, sum of durations in microseconds | long | | counter | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | diff --git a/test/packages/kubernetes/docs/kube-scheduler.md b/test/packages/kubernetes/docs/kube-scheduler.md new file mode 100644 index 0000000000..5c239728f9 --- /dev/null +++ b/test/packages/kubernetes/docs/kube-scheduler.md @@ -0,0 +1,183 @@ +# kube-scheduler + +## Metrics + +### scheduler + +This is the `scheduler` dataset of the Kubernetes package. It collects metrics +from Kubernetes Scheduler component. + +An example event for `scheduler` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:35:59.624Z", + "agent": { + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a", + "name": "minikube", + "type": "metricbeat" + }, + "host": { + "hostname": "minikube", + "architecture": "x86_64", + "os": { + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux" + }, + "name": "minikube", + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ] + }, + "ecs": { + "version": "1.5.0" + }, + "event": { + "duration": 7245648, + "dataset": "kubernetes.scheduler", + "module": "kubernetes" + }, + "metricset": { + "name": "scheduler", + "period": 10000 + }, + "service": { + "address": "localhost:10251", + "type": "kubernetes" + }, + "kubernetes": { + "scheduler": { + "name": "kube-scheduler", + "leader": { + "is_master": true + } + } + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.scheduler.client.request.count | Number of requests as client | long | | counter | +| kubernetes.scheduler.code | HTTP code | keyword | | | +| kubernetes.scheduler.handler | Request handler | keyword | | | +| kubernetes.scheduler.host | Request host | keyword | | | +| kubernetes.scheduler.http.request.count | Request count | long | | counter | +| kubernetes.scheduler.http.request.duration.us.count | Request count for duration | long | micros | counter | +| kubernetes.scheduler.http.request.duration.us.percentile.\* | Request duration microseconds percentiles | object | | | +| kubernetes.scheduler.http.request.duration.us.sum | Request duration microseconds cumulative sum | double | micros | counter | +| kubernetes.scheduler.http.request.size.bytes.count | Request count for size | long | byte | counter | +| kubernetes.scheduler.http.request.size.bytes.percentile.\* | Request size percentiles | object | | | +| kubernetes.scheduler.http.request.size.bytes.sum | Request size cumulative sum | long | byte | counter | +| kubernetes.scheduler.http.response.size.bytes.count | Response count | long | | counter | +| kubernetes.scheduler.http.response.size.bytes.percentile.\* | Response size percentiles | object | | | +| kubernetes.scheduler.http.response.size.bytes.sum | Response size cumulative sum | long | byte | counter | +| kubernetes.scheduler.leader.is_master | Whether the node is master | boolean | | | +| kubernetes.scheduler.method | HTTP method | keyword | | | +| kubernetes.scheduler.name | Name for the resource | keyword | | | +| kubernetes.scheduler.operation | Scheduling operation | keyword | | | +| kubernetes.scheduler.process.cpu.sec | CPU seconds | double | | counter | +| kubernetes.scheduler.process.fds.open.count | Number of open file descriptors | long | | gauge | +| kubernetes.scheduler.process.memory.resident.bytes | Bytes in resident memory | long | byte | gauge | +| kubernetes.scheduler.process.memory.virtual.bytes | Bytes in virtual memory | long | byte | gauge | +| kubernetes.scheduler.process.started.sec | Seconds since the process started | double | | gauge | +| kubernetes.scheduler.result | Schedule attempt result | keyword | | | +| kubernetes.scheduler.scheduling.duration.seconds.count | Scheduling count | long | | counter | +| kubernetes.scheduler.scheduling.duration.seconds.percentile.\* | Scheduling duration percentiles | object | | | +| kubernetes.scheduler.scheduling.duration.seconds.sum | Scheduling duration cumulative sum | double | | counter | +| kubernetes.scheduler.scheduling.e2e.duration.us.bucket.\* | End to end scheduling duration microseconds | object | | | +| kubernetes.scheduler.scheduling.e2e.duration.us.count | End to end scheduling count | long | micros | counter | +| kubernetes.scheduler.scheduling.e2e.duration.us.sum | End to end scheduling duration microseconds sum | long | micros | counter | +| kubernetes.scheduler.scheduling.pod.attempts.count | Pod attempts count | long | | counter | +| kubernetes.scheduler.scheduling.pod.preemption.victims.bucket.\* | Pod preemption victims | long | | | +| kubernetes.scheduler.scheduling.pod.preemption.victims.count | Pod preemption victims count | long | | counter | +| kubernetes.scheduler.scheduling.pod.preemption.victims.sum | Pod preemption victims sum | long | | counter | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | diff --git a/test/packages/kubernetes/docs/kube-state-metrics.md b/test/packages/kubernetes/docs/kube-state-metrics.md new file mode 100644 index 0000000000..e94360c85e --- /dev/null +++ b/test/packages/kubernetes/docs/kube-state-metrics.md @@ -0,0 +1,1988 @@ +# kube-state-metrics + +## Metrics + +If Leader Election is activated (default behaviour) only the `elastic agent` which holds the leadership lock +will retrieve metrics from the `kube_state_metrics`. +This is relevant in multi-node kubernetes cluster and prevents duplicate data. + +### state_container + +This is the `state_container` dataset of the Kubernetes package. It collects container related +metrics from `kube_state_metrics`. + +An example event for `state_container` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:36:34.469Z", + "host": { + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "os": { + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false + }, + "event": { + "dataset": "kubernetes.container", + "module": "kubernetes", + "duration": 8554499 + }, + "kubernetes": { + "node": { + "name": "minikube" + }, + "labels": { + "component": "kube-scheduler", + "tier": "control-plane" + }, + "container": { + "image": "k8s.gcr.io/kube-scheduler:v1.17.0", + "name": "kube-scheduler", + "cpu": { + "request": { + "cores": 0.1 + } + }, + "status": { + "phase": "running", + "ready": true, + "restarts": 10 + }, + "id": "docker://b00b185f2b304a7ece804d1af28eb232f825255f716bcc85ef5bd20d5a4f45d4" + }, + "pod": { + "name": "kube-scheduler-minikube", + "uid": "9cdbd5ea-7638-4e86-a706-a5b222d86f26" + }, + "namespace": "kube-system" + }, + "ecs": { + "version": "1.5.0" + }, + "agent": { + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0" + }, + "container": { + "runtime": "docker", + "id": "b00b185f2b304a7ece804d1af28eb232f825255f716bcc85ef5bd20d5a4f45d4" + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "metricset": { + "name": "state_container", + "period": 10000 + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| container.runtime | Runtime managing this container | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.cpu.limit.cores | Container CPU cores limit | float | | gauge | +| kubernetes.container.cpu.limit.nanocores | Container CPU nanocores limit | long | | gauge | +| kubernetes.container.cpu.request.cores | Container CPU requested cores | float | | gauge | +| kubernetes.container.cpu.request.nanocores | Container CPU requested nanocores | long | | gauge | +| kubernetes.container.id | Container id | keyword | | | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.memory.limit.bytes | Container memory limit in bytes | long | byte | gauge | +| kubernetes.container.memory.request.bytes | Container requested memory in bytes | long | byte | gauge | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.container.status.phase | Container phase (running, waiting, terminated) | keyword | | | +| kubernetes.container.status.ready | Container ready status | boolean | | | +| kubernetes.container.status.reason | Waiting (ContainerCreating, CrashLoopBackoff, ErrImagePull, ImagePullBackoff) or termination (Completed, ContainerCannotRun, Error, OOMKilled) reason. | keyword | | | +| kubernetes.container.status.restarts | Container restarts count | integer | | counter | +| kubernetes.daemonset.name | Kubernetes daemonset name | keyword | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | + + +### state_cronjob + +This is the `state_cronjob` dataset of the Kubernetes package. It collects cronjob related +metrics from `kube_state_metrics`. + +An example event for `state_cronjob` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:43:04.384Z", + "metricset": { + "name": "state_cronjob", + "period": 10000 + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "ecs": { + "version": "1.5.0" + }, + "host": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)" + } + }, + "event": { + "dataset": "kubernetes.cronjob", + "module": "kubernetes", + "duration": 9482053 + }, + "kubernetes": { + "namespace": "default", + "cronjob": { + "active": { + "count": 0 + }, + "is_suspended": false, + "name": "hello", + "next_schedule": { + "sec": 1593088980 + }, + "last_schedule": { + "sec": 1593088920 + }, + "created": { + "sec": 1593088862 + } + } + }, + "agent": { + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.cronjob.active.count | Number of active pods for the cronjob | long | | gauge | +| kubernetes.cronjob.concurrency | Concurrency policy | keyword | | | +| kubernetes.cronjob.created.sec | Epoch seconds since the cronjob was created | double | s | gauge | +| kubernetes.cronjob.deadline.sec | Deadline seconds after schedule for considering failed | long | s | gauge | +| kubernetes.cronjob.is_suspended | Whether the cronjob is suspended | boolean | | | +| kubernetes.cronjob.last_schedule.sec | Epoch seconds for last cronjob run | double | s | gauge | +| kubernetes.cronjob.name | Cronjob name | keyword | | | +| kubernetes.cronjob.next_schedule.sec | Epoch seconds for next cronjob run | double | s | gauge | +| kubernetes.cronjob.schedule | Cronjob schedule | keyword | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | + + +### state_daemonset + +This is the `state_daemonset` dataset of the Kubernetes package. It collects daemonset related +metrics from `kube_state_metrics`. + +An example event for `state_daemonset` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:37:04.455Z", + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "event": { + "module": "kubernetes", + "duration": 8648138, + "dataset": "kubernetes.daemonset" + }, + "ecs": { + "version": "1.5.0" + }, + "metricset": { + "name": "state_daemonset", + "period": 10000 + }, + "kubernetes": { + "daemonset": { + "name": "metricbeat", + "replicas": { + "available": 1, + "desired": 1, + "ready": 1, + "unavailable": 0 + } + }, + "labels": { + "k8s-app": "metricbeat" + }, + "namespace": "kube-system" + }, + "host": { + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat" + }, + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ] + }, + "agent": { + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Metric Type | +|---|---|---|---| +| @timestamp | Event timestamp. | date | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | +| cloud.instance.name | Instance name of the host machine. | keyword | | +| cloud.machine.type | Machine type of the host machine. | keyword | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | +| cloud.region | Region in which this host is running. | keyword | | +| container.id | Unique container id. | keyword | | +| container.image.name | Name of the image the container was built on. | keyword | | +| container.labels | Image labels. | object | | +| container.name | Container name. | keyword | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | +| data_stream.type | Data stream type. | constant_keyword | | +| ecs.version | ECS version | keyword | | +| host.architecture | Operating system architecture. | keyword | | +| host.containerized | If the host is a container. | boolean | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | +| host.ip | Host ip addresses. | ip | | +| host.mac | Host mac addresses. | keyword | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | +| host.os.build | OS build information. | keyword | | +| host.os.codename | OS codename, if any. | keyword | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | +| host.os.name | Operating system name, without the version. | keyword | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | +| host.os.version | Operating system version as a raw string. | keyword | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | +| kubernetes.container.image | Kubernetes container image | keyword | | +| kubernetes.container.name | Kubernetes container name | keyword | | +| kubernetes.daemonset.name | | keyword | | +| kubernetes.daemonset.replicas.available | The number of available replicas per DaemonSet | long | gauge | +| kubernetes.daemonset.replicas.desired | The desired number of replicas per DaemonSet | long | gauge | +| kubernetes.daemonset.replicas.ready | The number of ready replicas per DaemonSet | long | gauge | +| kubernetes.daemonset.replicas.unavailable | The number of unavailable replicas per DaemonSet | long | gauge | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | +| kubernetes.labels.\* | Kubernetes labels map | object | | +| kubernetes.namespace | Kubernetes namespace | keyword | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | +| kubernetes.node.name | Kubernetes node name | keyword | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | +| service.address | Service address | keyword | | +| service.type | Service type | keyword | | + + +### state_deployment + +This is the `state_deployment` dataset of the Kubernetes package. It collects deployment related +metrics from `kube_state_metrics`. + +An example event for `state_deployment` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:37:04.455Z", + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "event": { + "module": "kubernetes", + "duration": 8648138, + "dataset": "kubernetes.deployment" + }, + "ecs": { + "version": "1.5.0" + }, + "metricset": { + "name": "state_deployment", + "period": 10000 + }, + "kubernetes": { + "deployment": { + "name": "metricbeat", + "replicas": { + "unavailable": 0, + "desired": 1, + "updated": 1, + "available": 1 + }, + "paused": false + }, + "labels": { + "k8s-app": "metricbeat" + }, + "namespace": "kube-system" + }, + "host": { + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat" + }, + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ] + }, + "agent": { + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Metric Type | +|---|---|---|---| +| @timestamp | Event timestamp. | date | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | +| cloud.instance.name | Instance name of the host machine. | keyword | | +| cloud.machine.type | Machine type of the host machine. | keyword | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | +| cloud.region | Region in which this host is running. | keyword | | +| container.id | Unique container id. | keyword | | +| container.image.name | Name of the image the container was built on. | keyword | | +| container.labels | Image labels. | object | | +| container.name | Container name. | keyword | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | +| data_stream.type | Data stream type. | constant_keyword | | +| ecs.version | ECS version | keyword | | +| host.architecture | Operating system architecture. | keyword | | +| host.containerized | If the host is a container. | boolean | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | +| host.ip | Host ip addresses. | ip | | +| host.mac | Host mac addresses. | keyword | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | +| host.os.build | OS build information. | keyword | | +| host.os.codename | OS codename, if any. | keyword | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | +| host.os.name | Operating system name, without the version. | keyword | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | +| host.os.version | Operating system version as a raw string. | keyword | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | +| kubernetes.container.image | Kubernetes container image | keyword | | +| kubernetes.container.name | Kubernetes container name | keyword | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | +| kubernetes.deployment.paused | Kubernetes deployment paused status | boolean | | +| kubernetes.deployment.replicas.available | Deployment available replicas | integer | gauge | +| kubernetes.deployment.replicas.desired | Deployment number of desired replicas (spec) | integer | gauge | +| kubernetes.deployment.replicas.unavailable | Deployment unavailable replicas | integer | gauge | +| kubernetes.deployment.replicas.updated | Deployment updated replicas | integer | gauge | +| kubernetes.labels.\* | Kubernetes labels map | object | | +| kubernetes.namespace | Kubernetes namespace | keyword | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | +| kubernetes.node.name | Kubernetes node name | keyword | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | +| service.address | Service address | keyword | | +| service.type | Service type | keyword | | + + +### state_job + +This is the `state_job` dataset of the Kubernetes package. It collects job related +metrics from `kube_state_metrics`. + +An example event for `state_job` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:43:04.384Z", + "metricset": { + "name": "state_job", + "period": 10000 + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "ecs": { + "version": "1.5.0" + }, + "host": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)" + } + }, + "event": { + "dataset": "kubernetes.job", + "module": "kubernetes", + "duration": 9482053 + }, + "kubernetes": { + "job": { + "completions": { + "desired": 1 + }, + "name": "sleep-30-ok-cron-27075645", + "owner": { + "is_controller": "true", + "kind": "CronJob", + "name": "sleep-30-ok-cron" + }, + "parallelism": { + "desired": 1 + }, + "pods": { + "active": 1, + "failed": 0, + "succeeded": 0 + }, + "time": { + "created": "2021-06-24T12:45:00.000Z" + } + }, + "namespace": "default" + }, + "agent": { + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Metric Type | +|---|---|---|---| +| @timestamp | Event timestamp. | date | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | +| cloud.instance.name | Instance name of the host machine. | keyword | | +| cloud.machine.type | Machine type of the host machine. | keyword | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | +| cloud.region | Region in which this host is running. | keyword | | +| container.id | Unique container id. | keyword | | +| container.image.name | Name of the image the container was built on. | keyword | | +| container.labels | Image labels. | object | | +| container.name | Container name. | keyword | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | +| data_stream.type | Data stream type. | constant_keyword | | +| ecs.version | ECS version | keyword | | +| host.architecture | Operating system architecture. | keyword | | +| host.containerized | If the host is a container. | boolean | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | +| host.ip | Host ip addresses. | ip | | +| host.mac | Host mac addresses. | keyword | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | +| host.os.build | OS build information. | keyword | | +| host.os.codename | OS codename, if any. | keyword | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | +| host.os.name | Operating system name, without the version. | keyword | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | +| host.os.version | Operating system version as a raw string. | keyword | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | +| kubernetes.container.image | Kubernetes container image | keyword | | +| kubernetes.container.name | Kubernetes container name | keyword | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | +| kubernetes.job.completions.desired | The configured completion count for the job (Spec) | long | gauge | +| kubernetes.job.name | The name of the job resource | keyword | | +| kubernetes.job.owner.is_controller | Owner is controller ("true", "false", or `"\"`) | keyword | | +| kubernetes.job.owner.kind | The kind of resource that owns this job (eg. "CronJob") | keyword | | +| kubernetes.job.owner.name | The name of the resource that owns this job | keyword | | +| kubernetes.job.parallelism.desired | The configured parallelism of the job (Spec) | long | gauge | +| kubernetes.job.pods.active | Number of active pods | long | gauge | +| kubernetes.job.pods.failed | Number of failed pods | long | gauge | +| kubernetes.job.pods.succeeded | Number of successful pods | long | gauge | +| kubernetes.job.status.complete | Whether the job completed ("true", "false", or "unknown") | keyword | | +| kubernetes.job.status.failed | Whether the job failed ("true", "false", or "unknown") | keyword | | +| kubernetes.job.time.completed | The time at which the job completed | date | | +| kubernetes.job.time.created | The time at which the job was created | date | | +| kubernetes.labels.\* | Kubernetes labels map | object | | +| kubernetes.namespace | Kubernetes namespace | keyword | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | +| kubernetes.node.name | Kubernetes node name | keyword | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | +| service.address | Service address | keyword | | +| service.type | Service type | keyword | | + + +### state_node + +This is the `state_node` dataset of the Kubernetes package. It collects node related +metrics from `kube_state_metrics`. + +An example event for `state_node` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:37:44.457Z", + "ecs": { + "version": "1.5.0" + }, + "host": { + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ] + }, + "metricset": { + "name": "state_node", + "period": 10000 + }, + "kubernetes": { + "node": { + "pod": { + "capacity": { + "total": 110 + }, + "allocatable": { + "total": 110 + } + }, + "memory": { + "capacity": { + "bytes": 16815325184 + }, + "allocatable": { + "bytes": 16815325184 + } + }, + "cpu": { + "allocatable": { + "cores": 4 + }, + "capacity": { + "cores": 4 + } + }, + "name": "minikube", + "status": { + "ready": "true", + "unschedulable": false + } + }, + "labels": { + "kubernetes_io/arch": "amd64", + "kubernetes_io/hostname": "minikube", + "kubernetes_io/os": "linux", + "node-role_kubernetes_io/master": "", + "beta_kubernetes_io/arch": "amd64", + "beta_kubernetes_io/os": "linux" + } + }, + "agent": { + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0" + }, + "service": { + "type": "kubernetes", + "address": "kube-state-metrics:8080" + }, + "event": { + "dataset": "kubernetes.node", + "module": "kubernetes", + "duration": 8194220 + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.cpu.allocatable.cores | Node CPU allocatable cores | float | | gauge | +| kubernetes.node.cpu.capacity.cores | Node CPU capacity cores | long | | gauge | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.memory.allocatable.bytes | Node allocatable memory in bytes | long | byte | gauge | +| kubernetes.node.memory.capacity.bytes | Node memory capacity in bytes | long | byte | gauge | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.node.pod.allocatable.total | Node allocatable pods | long | | gauge | +| kubernetes.node.pod.capacity.total | Node pod capacity | long | | gauge | +| kubernetes.node.status.disk_pressure | Node DiskPressure status (true, false or unknown) | keyword | | | +| kubernetes.node.status.memory_pressure | Node MemoryPressure status (true, false or unknown) | keyword | | | +| kubernetes.node.status.out_of_disk | Node OutOfDisk status (true, false or unknown) | keyword | | | +| kubernetes.node.status.pid_pressure | Node PIDPressure status (true, false or unknown) | keyword | | | +| kubernetes.node.status.ready | Node ready status (true, false or unknown) | keyword | | | +| kubernetes.node.status.unschedulable | Node unschedulable status | boolean | | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | + + +### state_persistentvolume + +This is the `state_persistentvolume` dataset of the Kubernetes package. It collects +PersistentVolume related metrics from `kube_state_metrics`. + +An example event for `state_persistentvolume` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:43:54.412Z", + "ecs": { + "version": "1.5.0" + }, + "event": { + "module": "kubernetes", + "duration": 12149615, + "dataset": "kubernetes.persistentvolume" + }, + "agent": { + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat" + }, + "kubernetes": { + "persistentvolume": { + "capacity": { + "bytes": 10737418240 + }, + "phase": "Bound", + "storage_class": "manual", + "name": "task-pv-volume" + }, + "labels": { + "type": "local" + } + }, + "host": { + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "containerized": false + }, + "metricset": { + "period": 10000, + "name": "state_persistentvolume" + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.persistentvolume.capacity.bytes | Volume capacity | long | byte | gauge | +| kubernetes.persistentvolume.name | Volume name. | keyword | | | +| kubernetes.persistentvolume.phase | Volume phase according to kubernetes | keyword | | | +| kubernetes.persistentvolume.storage_class | Storage class for the volume | keyword | | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | + + +### state_persistentvolumeclaim + +This is the `state_persistentvolumeclaim` dataset of the Kubernetes package. It collects +PersistentVolumeClaim related metrics from `kube_state_metrics`. + +An example event for `state_persistentvolumeclaim` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:44:44.418Z", + "event": { + "dataset": "kubernetes.persistentvolumeclaim", + "module": "kubernetes", + "duration": 5698588 + }, + "metricset": { + "name": "state_persistentvolumeclaim", + "period": 10000 + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "kubernetes": { + "namespace": "default", + "persistentvolumeclaim": { + "phase": "Bound", + "storage_class": "manual", + "volume_name": "task-pv-volume", + "name": "task-pv-claim", + "request_storage": { + "bytes": 3221225472 + }, + "access_mode": "ReadWriteOnce" + } + }, + "agent": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee" + }, + "ecs": { + "version": "1.5.0" + }, + "host": { + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.persistentvolumeclaim.access_mode | Access mode. | keyword | | | +| kubernetes.persistentvolumeclaim.name | PVC name. | keyword | | | +| kubernetes.persistentvolumeclaim.phase | PVC phase. | keyword | | | +| kubernetes.persistentvolumeclaim.request_storage.bytes | Requested capacity. | long | byte | gauge | +| kubernetes.persistentvolumeclaim.storage_class | Storage class for the PVC. | keyword | | | +| kubernetes.persistentvolumeclaim.volume_name | Binded volume name. | keyword | | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | + + +### state_pod + +This is the `state_pod` dataset of the Kubernetes package. It collects +Pod related metrics from `kube_state_metrics`. + +An example event for `state_pod` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:38:34.469Z", + "ecs": { + "version": "1.5.0" + }, + "host": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux" + } + }, + "event": { + "duration": 10777415, + "dataset": "kubernetes.pod", + "module": "kubernetes" + }, + "service": { + "type": "kubernetes", + "address": "kube-state-metrics:8080" + }, + "kubernetes": { + "pod": { + "name": "filebeat-dqzzz", + "status": { + "ready": "true", + "scheduled": "true", + "phase": "running" + }, + "host_ip": "192.168.64.10", + "ip": "192.168.64.10", + "uid": "a5f1d3c9-40b6-4182-823b-dd5ff9832279" + }, + "namespace": "kube-system", + "node": { + "name": "minikube" + }, + "labels": { + "controller-revision-hash": "85649b9ddb", + "k8s-app": "filebeat", + "pod-template-generation": "1" + } + }, + "agent": { + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487" + }, + "metricset": { + "period": 10000, + "name": "state_pod" + } +} +``` + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | +| cloud.image.id | Image ID for the cloud instance. | keyword | +| cloud.instance.id | Instance ID of the host machine. | keyword | +| cloud.instance.name | Instance name of the host machine. | keyword | +| cloud.machine.type | Machine type of the host machine. | keyword | +| cloud.project.id | Name of the project in Google Cloud. | keyword | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | +| cloud.region | Region in which this host is running. | keyword | +| container.id | Unique container id. | keyword | +| container.image.name | Name of the image the container was built on. | keyword | +| container.labels | Image labels. | object | +| container.name | Container name. | keyword | +| container.runtime | Runtime managing this container | keyword | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| ecs.version | ECS version | keyword | +| host.architecture | Operating system architecture. | keyword | +| host.containerized | If the host is a container. | boolean | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | +| host.ip | Host ip addresses. | ip | +| host.mac | Host mac addresses. | keyword | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | +| host.os.build | OS build information. | keyword | +| host.os.codename | OS codename, if any. | keyword | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | +| host.os.name | Operating system name, without the version. | keyword | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | +| host.os.version | Operating system version as a raw string. | keyword | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | +| kubernetes.annotations.\* | Kubernetes annotations map | object | +| kubernetes.container.image | Kubernetes container image | keyword | +| kubernetes.container.name | Kubernetes container name | keyword | +| kubernetes.daemonset.name | Kubernetes daemonset name | keyword | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | +| kubernetes.labels.\* | Kubernetes labels map | object | +| kubernetes.namespace | Kubernetes namespace | keyword | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | +| kubernetes.node.name | Kubernetes node name | keyword | +| kubernetes.pod.host_ip | Kubernetes pod host IP | ip | +| kubernetes.pod.ip | Kubernetes pod IP | ip | +| kubernetes.pod.name | Kubernetes pod name | keyword | +| kubernetes.pod.status.phase | Kubernetes pod phase (Running, Pending...) | keyword | +| kubernetes.pod.status.ready | Kubernetes pod ready status (true, false or unknown) | keyword | +| kubernetes.pod.status.scheduled | Kubernetes pod scheduled status (true, false, unknown) | keyword | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | +| orchestrator.cluster.name | Name of the cluster. | keyword | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | +| service.address | Service address | keyword | +| service.type | Service type | keyword | + + +### state_replicaset + +This is the `state_replicaset` dataset of the Kubernetes package. It collects +Replicaset related metrics from `kube_state_metrics`. + +An example event for `state_replicaset` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:38:54.482Z", + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "metricset": { + "period": 10000, + "name": "state_replicaset" + }, + "event": { + "module": "kubernetes", + "duration": 5456128, + "dataset": "kubernetes.replicaset" + }, + "kubernetes": { + "namespace": "kube-system", + "replicaset": { + "name": "nginx-ingress-controller-6fc5bcc8c9", + "replicas": { + "labeled": 1, + "ready": 1, + "available": 1, + "observed": 1, + "desired": 1 + } + }, + "deployment": { + "name": "nginx-ingress-controller" + }, + "labels": { + "app_kubernetes_io/part-of": "kube-system", + "pod-template-hash": "6fc5bcc8c9", + "addonmanager_kubernetes_io/mode": "Reconcile", + "app_kubernetes_io/name": "nginx-ingress-controller" + } + }, + "agent": { + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0" + }, + "ecs": { + "version": "1.5.0" + }, + "host": { + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + }, + "id": "b0e83d397c054b8a99a431072fe4617b" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Metric Type | +|---|---|---|---| +| @timestamp | Event timestamp. | date | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | +| cloud.instance.name | Instance name of the host machine. | keyword | | +| cloud.machine.type | Machine type of the host machine. | keyword | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | +| cloud.region | Region in which this host is running. | keyword | | +| container.id | Unique container id. | keyword | | +| container.image.name | Name of the image the container was built on. | keyword | | +| container.labels | Image labels. | object | | +| container.name | Container name. | keyword | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | +| data_stream.type | Data stream type. | constant_keyword | | +| ecs.version | ECS version | keyword | | +| host.architecture | Operating system architecture. | keyword | | +| host.containerized | If the host is a container. | boolean | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | +| host.ip | Host ip addresses. | ip | | +| host.mac | Host mac addresses. | keyword | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | +| host.os.build | OS build information. | keyword | | +| host.os.codename | OS codename, if any. | keyword | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | +| host.os.name | Operating system name, without the version. | keyword | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | +| host.os.version | Operating system version as a raw string. | keyword | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | +| kubernetes.container.image | Kubernetes container image | keyword | | +| kubernetes.container.name | Kubernetes container name | keyword | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | +| kubernetes.labels.\* | Kubernetes labels map | object | | +| kubernetes.namespace | Kubernetes namespace | keyword | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | +| kubernetes.node.name | Kubernetes node name | keyword | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | +| kubernetes.replicaset.replicas.available | The number of replicas per ReplicaSet | long | gauge | +| kubernetes.replicaset.replicas.desired | The number of replicas per ReplicaSet | long | gauge | +| kubernetes.replicaset.replicas.labeled | The number of fully labeled replicas per ReplicaSet | long | gauge | +| kubernetes.replicaset.replicas.observed | The generation observed by the ReplicaSet controller | long | gauge | +| kubernetes.replicaset.replicas.ready | The number of ready replicas per ReplicaSet | long | gauge | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | +| service.address | Service address | keyword | | +| service.type | Service type | keyword | | + + +### state_resourcequota + +This is the `state_resourcequota` dataset of the Kubernetes package. It collects ResourceQuota related metrics +from `kube_state_metrics`. + +An example event for `state_resourcequota` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:45:04.416Z", + "metricset": { + "name": "state_resourcequota", + "period": 10000 + }, + "host": { + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ] + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "event": { + "dataset": "kubernetes.resourcequota", + "module": "kubernetes", + "duration": 6324269 + }, + "agent": { + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487" + }, + "ecs": { + "version": "1.5.0" + }, + "kubernetes": { + "namespace": "quota-object-example", + "resourcequota": { + "name": "object-quota-demo", + "resource": "persistentvolumeclaims", + "type": "hard", + "quota": 1 + } + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.resourcequota.created.sec | Epoch seconds since the ResourceQuota was created | double | s | gauge | +| kubernetes.resourcequota.name | ResourceQuota name | keyword | | | +| kubernetes.resourcequota.quota | Quota informed (hard or used) for the resource | double | | gauge | +| kubernetes.resourcequota.resource | Resource name the quota applies to | keyword | | | +| kubernetes.resourcequota.type | Quota information type, `hard` or `used` | keyword | | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | + + +### state_service + +This is the `state_service` dataset of the Kubernetes package. It collects +Service related metrics from `kube_state_metrics`. + +An example event for `state_service` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:39:24.389Z", + "kubernetes": { + "labels": { + "kubernetes_io_minikube_addons_endpoint": "metrics-server", + "kubernetes_io_name": "Metrics-server", + "addonmanager_kubernetes_io_mode": "Reconcile", + "kubernetes_io_minikube_addons": "metrics-server" + }, + "service": { + "name": "metrics-server", + "created": "2020-06-10T09:02:27.000Z", + "cluster_ip": "10.96.124.248", + "type": "ClusterIP" + }, + "namespace": "kube-system" + }, + "event": { + "dataset": "kubernetes.service", + "module": "kubernetes", + "duration": 10966648 + }, + "metricset": { + "name": "state_service", + "period": 10000 + }, + "host": { + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + } + }, + "agent": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee" + }, + "ecs": { + "version": "1.5.0" + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + } +} +``` + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | +| cloud.image.id | Image ID for the cloud instance. | keyword | +| cloud.instance.id | Instance ID of the host machine. | keyword | +| cloud.instance.name | Instance name of the host machine. | keyword | +| cloud.machine.type | Machine type of the host machine. | keyword | +| cloud.project.id | Name of the project in Google Cloud. | keyword | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | +| cloud.region | Region in which this host is running. | keyword | +| container.id | Unique container id. | keyword | +| container.image.name | Name of the image the container was built on. | keyword | +| container.labels | Image labels. | object | +| container.name | Container name. | keyword | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| ecs.version | ECS version | keyword | +| host.architecture | Operating system architecture. | keyword | +| host.containerized | If the host is a container. | boolean | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | +| host.ip | Host ip addresses. | ip | +| host.mac | Host mac addresses. | keyword | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | +| host.os.build | OS build information. | keyword | +| host.os.codename | OS codename, if any. | keyword | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | +| host.os.name | Operating system name, without the version. | keyword | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | +| host.os.version | Operating system version as a raw string. | keyword | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | +| kubernetes.annotations.\* | Kubernetes annotations map | object | +| kubernetes.container.image | Kubernetes container image | keyword | +| kubernetes.container.name | Kubernetes container name | keyword | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | +| kubernetes.labels.\* | Kubernetes labels map | object | +| kubernetes.namespace | Kubernetes namespace | keyword | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | +| kubernetes.node.name | Kubernetes node name | keyword | +| kubernetes.pod.ip | Kubernetes pod IP | ip | +| kubernetes.pod.name | Kubernetes pod name | keyword | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | +| kubernetes.service.cluster_ip | Internal IP for the service. | ip | +| kubernetes.service.created | Service creation date | date | +| kubernetes.service.external_ip | Service external IP | keyword | +| kubernetes.service.external_name | Service external DNS name | keyword | +| kubernetes.service.ingress_hostname | Ingress Hostname | keyword | +| kubernetes.service.ingress_ip | Ingress IP | keyword | +| kubernetes.service.load_balancer_ip | Load Balancer service IP | keyword | +| kubernetes.service.name | Service name. | keyword | +| kubernetes.service.type | Service type | keyword | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | +| orchestrator.cluster.name | Name of the cluster. | keyword | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | +| service.address | Service address | keyword | +| service.type | Service type | keyword | + + +### state_statefulset + +This is the `state_statefulset` dataset of the Kubernetes package. + +An example event for `state_statefulset` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:39:24.389Z", + "kubernetes": { + "namespace": "default", + "statefulset": { + "created": 1511989697, + "generation": { + "desired": 4, + "observed": 2 + }, + "name": "mysql", + "replicas": { + "desired": 5, + "observed": 2 + } + } + }, + "event": { + "dataset": "kubernetes.statefulset", + "module": "kubernetes", + "duration": 10966648 + }, + "metricset": { + "name": "state_statefulset", + "period": 10000 + }, + "host": { + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ], + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + } + }, + "agent": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee" + }, + "ecs": { + "version": "1.5.0" + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Metric Type | +|---|---|---|---| +| @timestamp | Event timestamp. | date | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | +| cloud.instance.name | Instance name of the host machine. | keyword | | +| cloud.machine.type | Machine type of the host machine. | keyword | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | +| cloud.region | Region in which this host is running. | keyword | | +| container.id | Unique container id. | keyword | | +| container.image.name | Name of the image the container was built on. | keyword | | +| container.labels | Image labels. | object | | +| container.name | Container name. | keyword | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | +| data_stream.type | Data stream type. | constant_keyword | | +| ecs.version | ECS version | keyword | | +| host.architecture | Operating system architecture. | keyword | | +| host.containerized | If the host is a container. | boolean | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | +| host.ip | Host ip addresses. | ip | | +| host.mac | Host mac addresses. | keyword | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | +| host.os.build | OS build information. | keyword | | +| host.os.codename | OS codename, if any. | keyword | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | +| host.os.name | Operating system name, without the version. | keyword | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | +| host.os.version | Operating system version as a raw string. | keyword | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | +| kubernetes.container.image | Kubernetes container image | keyword | | +| kubernetes.container.name | Kubernetes container name | keyword | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | +| kubernetes.labels.\* | Kubernetes labels map | object | | +| kubernetes.namespace | Kubernetes namespace | keyword | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | +| kubernetes.node.name | Kubernetes node name | keyword | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | +| kubernetes.statefulset.created | The creation timestamp (epoch) for StatefulSet | long | gauge | +| kubernetes.statefulset.generation.desired | The desired generation per StatefulSet | long | gauge | +| kubernetes.statefulset.generation.observed | The observed generation per StatefulSet | long | gauge | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | +| kubernetes.statefulset.replicas.desired | The number of desired replicas per StatefulSet | long | gauge | +| kubernetes.statefulset.replicas.observed | The number of observed replicas per StatefulSet | long | gauge | +| kubernetes.statefulset.replicas.ready | The number of ready replicas per StatefulSet | long | gauge | +| orchestrator.cluster.name | Name of the cluster. | keyword | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | +| service.address | Service address | keyword | | +| service.type | Service type | keyword | | + + +### state_storageclass + +This is the `state_storageclass` dataset of the Kubernetes package. It collects +StorageClass related metrics from `kube_state_metrics`. + +An example event for `state_storageclass` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:39:44.399Z", + "agent": { + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "644323b5-5d6a-4dfb-92dd-35ca602db487", + "id": "a6147a6e-6626-4a84-9907-f372f6c61eee" + }, + "kubernetes": { + "storageclass": { + "provisioner": "k8s.io/minikube-hostpath", + "reclaim_policy": "Delete", + "volume_binding_mode": "Immediate", + "name": "standard", + "created": "2020-06-10T09:02:27.000Z" + }, + "labels": { + "addonmanager_kubernetes_io_mode": "EnsureExists" + } + }, + "host": { + "hostname": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "architecture": "x86_64", + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "name": "agent-ingest-management-clusterscope-674dbb75df-rp8cc", + "containerized": false, + "ip": [ + "172.17.0.11" + ], + "mac": [ + "02:42:ac:11:00:0b" + ] + }, + "event": { + "module": "kubernetes", + "duration": 5713503, + "dataset": "kubernetes.storageclass" + }, + "metricset": { + "name": "state_storageclass", + "period": 10000 + }, + "service": { + "address": "kube-state-metrics:8080", + "type": "kubernetes" + }, + "ecs": { + "version": "1.5.0" + } +} +``` + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | +| cloud.image.id | Image ID for the cloud instance. | keyword | +| cloud.instance.id | Instance ID of the host machine. | keyword | +| cloud.instance.name | Instance name of the host machine. | keyword | +| cloud.machine.type | Machine type of the host machine. | keyword | +| cloud.project.id | Name of the project in Google Cloud. | keyword | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | +| cloud.region | Region in which this host is running. | keyword | +| container.id | Unique container id. | keyword | +| container.image.name | Name of the image the container was built on. | keyword | +| container.labels | Image labels. | object | +| container.name | Container name. | keyword | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| ecs.version | ECS version | keyword | +| host.architecture | Operating system architecture. | keyword | +| host.containerized | If the host is a container. | boolean | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | +| host.ip | Host ip addresses. | ip | +| host.mac | Host mac addresses. | keyword | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | +| host.os.build | OS build information. | keyword | +| host.os.codename | OS codename, if any. | keyword | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | +| host.os.name | Operating system name, without the version. | keyword | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | +| host.os.version | Operating system version as a raw string. | keyword | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | +| kubernetes.annotations.\* | Kubernetes annotations map | object | +| kubernetes.container.image | Kubernetes container image | keyword | +| kubernetes.container.name | Kubernetes container name | keyword | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | +| kubernetes.labels.\* | Kubernetes labels map | object | +| kubernetes.namespace | Kubernetes namespace | keyword | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | +| kubernetes.node.name | Kubernetes node name | keyword | +| kubernetes.pod.ip | Kubernetes pod IP | ip | +| kubernetes.pod.name | Kubernetes pod name | keyword | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | +| kubernetes.storageclass.created | Storage class creation date | date | +| kubernetes.storageclass.name | Storage class name. | keyword | +| kubernetes.storageclass.provisioner | Volume provisioner for the storage class. | keyword | +| kubernetes.storageclass.reclaim_policy | Reclaim policy for dynamically created volumes | keyword | +| kubernetes.storageclass.volume_binding_mode | Mode for default provisioning and binding | keyword | +| orchestrator.cluster.name | Name of the cluster. | keyword | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | +| service.address | Service address | keyword | +| service.type | Service type | keyword | diff --git a/test/packages/kubernetes/docs/kubelet.md b/test/packages/kubernetes/docs/kubelet.md new file mode 100644 index 0000000000..13e0b8facf --- /dev/null +++ b/test/packages/kubernetes/docs/kubelet.md @@ -0,0 +1,1043 @@ +# kubelet + +## Metrics + +### container + +This is the `container` dataset of the Kubernetes package. It collects container related metrics +from Kubelet's monitoring APIs. + +An example event for `container` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:32:29.748Z", + "kubernetes": { + "namespace": "kube-system", + "node": { + "name": "minikube" + }, + "pod": { + "name": "metricbeat-g9fc6" + }, + "container": { + "rootfs": { + "used": { + "bytes": 61440 + }, + "inodes": { + "used": 17 + }, + "available": { + "bytes": 6724222976 + }, + "capacity": { + "bytes": 17361141760 + } + }, + "logs": { + "used": { + "bytes": 1617920 + }, + "inodes": { + "count": 9768928, + "used": 223910, + "free": 9545018 + }, + "available": { + "bytes": 6724222976 + }, + "capacity": { + "bytes": 17361141760 + } + }, + "start_time": "2020-06-25T07:19:37Z", + "name": "metricbeat", + "cpu": { + "usage": { + "node": { + "pct": 0.00015289625 + }, + "limit": { + "pct": 0.00015289625 + }, + "nanocores": 611585, + "core": { + "ns": 12206519774 + } + } + }, + "memory": { + "pagefaults": 10164, + "majorpagefaults": 528, + "available": { + "bytes": 188600320 + }, + "usage": { + "limit": { + "pct": 0.005608354460473573 + }, + "bytes": 94306304, + "node": { + "pct": 0.005608354460473573 + } + }, + "workingset": { + "bytes": 21114880 + }, + "rss": { + "bytes": 18386944 + } + } + } + }, + "host": { + "containerized": false, + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ], + "hostname": "minikube", + "architecture": "x86_64", + "os": { + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)" + }, + "name": "minikube", + "id": "b0e83d397c054b8a99a431072fe4617b" + }, + "agent": { + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a", + "name": "minikube" + }, + "metricset": { + "period": 10000, + "name": "container" + }, + "service": { + "address": "minikube:10250", + "type": "kubernetes" + }, + "event": { + "dataset": "kubernetes.container", + "module": "kubernetes", + "duration": 11091346 + }, + "ecs": { + "version": "1.5.0" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.cpu.usage.core.ns | Container CPU Core usage nanoseconds | long | | gauge | +| kubernetes.container.cpu.usage.limit.pct | CPU usage as a percentage of the defined limit for the container (or total node allocatable CPU if unlimited) | scaled_float | percent | gauge | +| kubernetes.container.cpu.usage.nanocores | CPU used nanocores | long | | gauge | +| kubernetes.container.cpu.usage.node.pct | CPU usage as a percentage of the total node allocatable CPU | scaled_float | percent | gauge | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.logs.available.bytes | Logs available capacity in bytes | long | byte | gauge | +| kubernetes.container.logs.capacity.bytes | Logs total capacity in bytes | long | byte | gauge | +| kubernetes.container.logs.inodes.count | Total available inodes | long | | gauge | +| kubernetes.container.logs.inodes.free | Total free inodes | long | | gauge | +| kubernetes.container.logs.inodes.used | Total used inodes | long | | gauge | +| kubernetes.container.logs.used.bytes | Logs used capacity in bytes | long | byte | gauge | +| kubernetes.container.memory.available.bytes | Total available memory | long | byte | gauge | +| kubernetes.container.memory.majorpagefaults | Number of major page faults | long | | counter | +| kubernetes.container.memory.pagefaults | Number of page faults | long | | counter | +| kubernetes.container.memory.rss.bytes | RSS memory usage | long | byte | gauge | +| kubernetes.container.memory.usage.bytes | Total memory usage | long | byte | gauge | +| kubernetes.container.memory.usage.limit.pct | Memory usage as a percentage of the defined limit for the container (or total node allocatable memory if unlimited) | scaled_float | percent | gauge | +| kubernetes.container.memory.usage.node.pct | Memory usage as a percentage of the total node allocatable memory | scaled_float | percent | gauge | +| kubernetes.container.memory.workingset.bytes | Working set memory usage | long | byte | gauge | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.container.rootfs.available.bytes | Root filesystem total available in bytes | long | byte | gauge | +| kubernetes.container.rootfs.capacity.bytes | Root filesystem total capacity in bytes | long | byte | gauge | +| kubernetes.container.rootfs.inodes.used | Used inodes | long | | gauge | +| kubernetes.container.rootfs.used.bytes | Root filesystem total used in bytes | long | byte | gauge | +| kubernetes.container.start_time | Start time | date | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | + + +### node + +This is the `node` dataset of the Kubernetes package. It collects Node related metrics +from Kubelet's monitoring APIs. + +An example event for `node` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:34:39.723Z", + "event": { + "dataset": "kubernetes.node", + "module": "kubernetes", + "duration": 13042307 + }, + "service": { + "type": "kubernetes", + "address": "minikube:10250" + }, + "host": { + "containerized": false, + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ], + "name": "minikube", + "hostname": "minikube", + "architecture": "x86_64", + "os": { + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81" + }, + "id": "b0e83d397c054b8a99a431072fe4617b" + }, + "metricset": { + "name": "node", + "period": 10000 + }, + "kubernetes": { + "labels": { + "beta_kubernetes_io/os": "linux", + "kubernetes_io/arch": "amd64", + "kubernetes_io/hostname": "minikube", + "kubernetes_io/os": "linux", + "node-role_kubernetes_io/master": "", + "beta_kubernetes_io/arch": "amd64" + }, + "node": { + "memory": { + "available": { + "bytes": 12746428416 + }, + "usage": { + "bytes": 5670916096 + }, + "workingset": { + "bytes": 4068896768 + }, + "rss": { + "bytes": 3252125696 + }, + "pagefaults": 31680, + "majorpagefaults": 0 + }, + "network": { + "rx": { + "bytes": 107077476, + "errors": 0 + }, + "tx": { + "bytes": 67457933, + "errors": 0 + } + }, + "fs": { + "available": { + "bytes": 6655090688 + }, + "capacity": { + "bytes": 17361141760 + }, + "used": { + "bytes": 9689358336 + }, + "inodes": { + "count": 9768928, + "used": 224151, + "free": 9544777 + } + }, + "runtime": { + "imagefs": { + "capacity": { + "bytes": 17361141760 + }, + "used": { + "bytes": 8719928568 + }, + "available": { + "bytes": 6655090688 + } + } + }, + "start_time": "2020-06-25T07:18:38Z", + "name": "minikube", + "cpu": { + "usage": { + "core": { + "ns": 6136184971873 + }, + "nanocores": 455263291 + } + } + } + }, + "agent": { + "name": "minikube", + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a" + }, + "ecs": { + "version": "1.5.0" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.cpu.usage.core.ns | Node CPU Core usage nanoseconds | long | | gauge | +| kubernetes.node.cpu.usage.nanocores | CPU used nanocores | long | | gauge | +| kubernetes.node.fs.available.bytes | Filesystem total available in bytes | long | byte | gauge | +| kubernetes.node.fs.capacity.bytes | Filesystem total capacity in bytes | long | byte | gauge | +| kubernetes.node.fs.inodes.count | Number of inodes | long | | gauge | +| kubernetes.node.fs.inodes.free | Number of free inodes | long | | gauge | +| kubernetes.node.fs.inodes.used | Number of used inodes | long | | gauge | +| kubernetes.node.fs.used.bytes | Filesystem total used in bytes | long | byte | gauge | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.memory.available.bytes | Total available memory | long | byte | gauge | +| kubernetes.node.memory.majorpagefaults | Number of major page faults | long | | counter | +| kubernetes.node.memory.pagefaults | Number of page faults | long | | counter | +| kubernetes.node.memory.rss.bytes | RSS memory usage | long | byte | gauge | +| kubernetes.node.memory.usage.bytes | Total memory usage | long | byte | gauge | +| kubernetes.node.memory.workingset.bytes | Working set memory usage | long | byte | gauge | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.node.network.rx.bytes | Received bytes | long | byte | counter | +| kubernetes.node.network.rx.errors | Rx errors | long | | | +| kubernetes.node.network.tx.bytes | Transmitted bytes | long | byte | counter | +| kubernetes.node.network.tx.errors | Tx errors | long | | counter | +| kubernetes.node.runtime.imagefs.available.bytes | Image filesystem total available in bytes | long | byte | gauge | +| kubernetes.node.runtime.imagefs.capacity.bytes | Image filesystem total capacity in bytes | long | byte | gauge | +| kubernetes.node.runtime.imagefs.used.bytes | Image filesystem total used in bytes | long | byte | gauge | +| kubernetes.node.start_time | Start time | date | | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | + + +### pod + +This is the `pod` dataset of the Kubernetes package. It collects Pod related metrics +from Kubelet's monitoring APIs. + +An example event for `pod` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:34:59.729Z", + "kubernetes": { + "pod": { + "memory": { + "rss": { + "bytes": 7823360 + }, + "page_faults": 5742, + "major_page_faults": 0, + "usage": { + "limit": { + "pct": 0.0008033509820466402 + }, + "bytes": 13508608, + "node": { + "pct": 0.0008033509820466402 + } + }, + "available": { + "bytes": 0 + }, + "working_set": { + "bytes": 8556544 + } + }, + "network": { + "rx": { + "bytes": 25671624, + "errors": 0 + }, + "tx": { + "errors": 0, + "bytes": 1092900259 + } + }, + "start_time": "2020-06-18T11:12:58Z", + "name": "kube-state-metrics-57cd6fdf9-hd959", + "uid": "a7c61334-dd52-4a12-bed5-4daee4c74139", + "cpu": { + "usage": { + "nanocores": 2811918, + "node": { + "pct": 0.0007029795 + }, + "limit": { + "pct": 0.0007029795 + } + } + } + }, + "namespace": "kube-system", + "node": { + "name": "minikube" + } + }, + "event": { + "duration": 20735189, + "dataset": "kubernetes.pod", + "module": "kubernetes" + }, + "ecs": { + "version": "1.5.0" + }, + "metricset": { + "period": 10000, + "name": "pod" + }, + "service": { + "type": "kubernetes", + "address": "minikube:10250" + }, + "host": { + "name": "minikube", + "hostname": "minikube", + "architecture": "x86_64", + "os": { + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ] + }, + "agent": { + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a", + "name": "minikube" + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.pod.cpu.usage.limit.pct | CPU usage as a percentage of the defined limit for the pod containers (or total node CPU if one or more containers of the pod are unlimited) | scaled_float | percent | gauge | +| kubernetes.pod.cpu.usage.nanocores | CPU used nanocores | long | byte | gauge | +| kubernetes.pod.cpu.usage.node.pct | CPU usage as a percentage of the total node CPU | scaled_float | percent | gauge | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.memory.available.bytes | Total memory available | long | percent | gauge | +| kubernetes.pod.memory.major_page_faults | Total major page faults | long | | counter | +| kubernetes.pod.memory.page_faults | Total page faults | long | | counter | +| kubernetes.pod.memory.rss.bytes | Total resident set size memory | long | percent | gauge | +| kubernetes.pod.memory.usage.bytes | Total memory usage | long | byte | gauge | +| kubernetes.pod.memory.usage.limit.pct | Memory usage as a percentage of the defined limit for the pod containers (or total node allocatable memory if unlimited) | scaled_float | percent | gauge | +| kubernetes.pod.memory.usage.node.pct | Memory usage as a percentage of the total node allocatable memory | scaled_float | percent | gauge | +| kubernetes.pod.memory.working_set.bytes | Total working set memory | long | percent | gauge | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.network.rx.bytes | Received bytes | long | byte | counter | +| kubernetes.pod.network.rx.errors | Rx errors | long | | counter | +| kubernetes.pod.network.tx.bytes | Transmitted bytes | long | byte | counter | +| kubernetes.pod.network.tx.errors | Tx errors | long | | counter | +| kubernetes.pod.start_time | Start time | date | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | + + +### system + +This is the `system` dataset of the Kubernetes package. It collects System related metrics +from Kubelet's monitoring APIs. + +An example event for `system` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:39:59.647Z", + "service": { + "address": "minikube:10250", + "type": "kubernetes" + }, + "event": { + "duration": 20012905, + "dataset": "kubernetes.system", + "module": "kubernetes" + }, + "ecs": { + "version": "1.5.0" + }, + "host": { + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ], + "hostname": "minikube", + "name": "minikube", + "architecture": "x86_64", + "os": { + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core", + "platform": "centos" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ] + }, + "agent": { + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a", + "name": "minikube", + "type": "metricbeat", + "version": "8.0.0" + }, + "kubernetes": { + "node": { + "name": "minikube" + }, + "system": { + "container": "runtime", + "cpu": { + "usage": { + "nanocores": 35779815, + "core": { + "ns": 530899961233 + } + } + }, + "memory": { + "pagefaults": 12944019, + "majorpagefaults": 99, + "usage": { + "bytes": 198279168 + }, + "workingset": { + "bytes": 178794496 + }, + "rss": { + "bytes": 125259776 + } + }, + "start_time": "2020-06-25T07:19:32Z" + } + }, + "metricset": { + "name": "system", + "period": 10000 + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| kubernetes.system.container | Container name | keyword | | | +| kubernetes.system.cpu.usage.core.ns | CPU Core usage nanoseconds | long | | gauge | +| kubernetes.system.cpu.usage.nanocores | CPU used nanocores | long | | gauge | +| kubernetes.system.memory.majorpagefaults | Number of major page faults | long | | counter | +| kubernetes.system.memory.pagefaults | Number of page faults | long | | counter | +| kubernetes.system.memory.rss.bytes | RSS memory usage | long | byte | gauge | +| kubernetes.system.memory.usage.bytes | Total memory usage | long | byte | gauge | +| kubernetes.system.memory.workingset.bytes | Working set memory usage | long | byte | gauge | +| kubernetes.system.start_time | Start time | date | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | + + +### volume + +This is the `volume` dataset of the Kubernetes package. It collects Volume related metrics +from Kubelet's monitoring APIs. + +An example event for `volume` looks as following: + +```json +{ + "@timestamp": "2020-06-25T12:40:19.649Z", + "ecs": { + "version": "1.5.0" + }, + "metricset": { + "name": "volume", + "period": 10000 + }, + "service": { + "type": "kubernetes", + "address": "minikube:10250" + }, + "kubernetes": { + "pod": { + "name": "metricbeat-g9fc6" + }, + "volume": { + "name": "config", + "fs": { + "inodes": { + "used": 5, + "free": 9549949, + "count": 9768928 + }, + "available": { + "bytes": 7719858176 + }, + "capacity": { + "bytes": 17361141760 + }, + "used": { + "bytes": 12288 + } + } + }, + "namespace": "kube-system", + "node": { + "name": "minikube" + } + }, + "host": { + "architecture": "x86_64", + "os": { + "platform": "centos", + "version": "7 (Core)", + "family": "redhat", + "name": "CentOS Linux", + "kernel": "4.19.81", + "codename": "Core" + }, + "id": "b0e83d397c054b8a99a431072fe4617b", + "containerized": false, + "ip": [ + "192.168.64.10", + "fe80::a883:2fff:fe7f:6b12", + "172.17.0.1", + "fe80::42:d4ff:fe8c:9493", + "fe80::2859:80ff:fe9e:fcd6", + "fe80::d83a:d9ff:fee9:7052", + "fe80::880a:b6ff:fe18:ba76", + "fe80::f447:faff:fe80:e88b", + "fe80::9cc3:ffff:fe95:e48e", + "fe80::6c1c:29ff:fe50:d40c", + "fe80::b4f3:11ff:fe60:14ed", + "fe80::20f2:2aff:fe96:1e7b", + "fe80::5434:baff:fede:5720", + "fe80::a878:91ff:fe29:81f7" + ], + "name": "minikube", + "mac": [ + "aa:83:2f:7f:6b:12", + "02:42:d4:8c:94:93", + "2a:59:80:9e:fc:d6", + "da:3a:d9:e9:70:52", + "8a:0a:b6:18:ba:76", + "f6:47:fa:80:e8:8b", + "9e:c3:ff:95:e4:8e", + "6e:1c:29:50:d4:0c", + "b6:f3:11:60:14:ed", + "22:f2:2a:96:1e:7b", + "56:34:ba:de:57:20", + "aa:78:91:29:81:f7" + ], + "hostname": "minikube" + }, + "agent": { + "type": "metricbeat", + "version": "8.0.0", + "ephemeral_id": "b964a246-96c0-456a-a5c2-8c8b1040ecaf", + "id": "f7ec69f9-4997-4e76-b6c7-0c75206b727a", + "name": "minikube" + }, + "event": { + "dataset": "kubernetes.volume", + "module": "kubernetes", + "duration": 12481688 + } +} +``` + +**Exported fields** + +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | | | +| cloud.image.id | Image ID for the cloud instance. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.instance.name | Instance name of the host machine. | keyword | | | +| cloud.machine.type | Machine type of the host machine. | keyword | | | +| cloud.project.id | Name of the project in Google Cloud. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host is running. | keyword | | | +| container.id | Unique container id. | keyword | | | +| container.image.name | Name of the image the container was built on. | keyword | | | +| container.labels | Image labels. | object | | | +| container.name | Container name. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| ecs.version | ECS version | keyword | | | +| host.architecture | Operating system architecture. | keyword | | | +| host.containerized | If the host is a container. | boolean | | | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | | | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | | | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | | | +| host.ip | Host ip addresses. | ip | | | +| host.mac | Host mac addresses. | keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| host.os.build | OS build information. | keyword | | | +| host.os.codename | OS codename, if any. | keyword | | | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | | | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | | | +| host.os.name | Operating system name, without the version. | keyword | | | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | | +| host.os.version | Operating system version as a raw string. | keyword | | | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | | +| kubernetes.annotations.\* | Kubernetes annotations map | object | | | +| kubernetes.container.image | Kubernetes container image | keyword | | | +| kubernetes.container.name | Kubernetes container name | keyword | | | +| kubernetes.deployment.name | Kubernetes deployment name | keyword | | | +| kubernetes.labels.\* | Kubernetes labels map | object | | | +| kubernetes.namespace | Kubernetes namespace | keyword | | | +| kubernetes.node.hostname | Kubernetes hostname as reported by the node’s kernel | keyword | | | +| kubernetes.node.name | Kubernetes node name | keyword | | | +| kubernetes.pod.ip | Kubernetes pod IP | ip | | | +| kubernetes.pod.name | Kubernetes pod name | keyword | | | +| kubernetes.pod.uid | Kubernetes pod UID | keyword | | | +| kubernetes.replicaset.name | Kubernetes replicaset name | keyword | | | +| kubernetes.selectors.\* | Kubernetes Service selectors map | object | | | +| kubernetes.statefulset.name | Kubernetes statefulset name | keyword | | | +| kubernetes.volume.fs.available.bytes | Filesystem total available in bytes | long | byte | gauge | +| kubernetes.volume.fs.capacity.bytes | Filesystem total capacity in bytes | long | byte | gauge | +| kubernetes.volume.fs.inodes.count | Total inodes | long | | gauge | +| kubernetes.volume.fs.inodes.free | Free inodes | long | | gauge | +| kubernetes.volume.fs.inodes.used | Used inodes | long | | gauge | +| kubernetes.volume.fs.used.bytes | Filesystem total used in bytes | long | byte | gauge | +| kubernetes.volume.fs.used.pct | Percentage of filesystem total used | long | percent | gauge | +| kubernetes.volume.name | Volume name | keyword | | | +| orchestrator.cluster.name | Name of the cluster. | keyword | | | +| orchestrator.cluster.url | URL of the API used to manage the cluster. | keyword | | | +| service.address | Service address | keyword | | | +| service.type | Service type | keyword | | | diff --git a/test/packages/kubernetes/img/logo_kubernetes.svg b/test/packages/kubernetes/img/logo_kubernetes.svg new file mode 100644 index 0000000000..7f3e86f548 --- /dev/null +++ b/test/packages/kubernetes/img/logo_kubernetes.svg @@ -0,0 +1 @@ + diff --git a/test/packages/kubernetes/img/metricbeat-kubernetes-controllermanager.png b/test/packages/kubernetes/img/metricbeat-kubernetes-controllermanager.png new file mode 100644 index 0000000000..ba9120f263 Binary files /dev/null and b/test/packages/kubernetes/img/metricbeat-kubernetes-controllermanager.png differ diff --git a/test/packages/kubernetes/img/metricbeat-kubernetes-proxy.png b/test/packages/kubernetes/img/metricbeat-kubernetes-proxy.png new file mode 100644 index 0000000000..439473e0ca Binary files /dev/null and b/test/packages/kubernetes/img/metricbeat-kubernetes-proxy.png differ diff --git a/test/packages/kubernetes/img/metricbeat_kubernetes_overview.png b/test/packages/kubernetes/img/metricbeat_kubernetes_overview.png new file mode 100644 index 0000000000..16ff532d27 Binary files /dev/null and b/test/packages/kubernetes/img/metricbeat_kubernetes_overview.png differ diff --git a/test/packages/kubernetes/img/metricbeat_kubernetes_scheduler.png b/test/packages/kubernetes/img/metricbeat_kubernetes_scheduler.png new file mode 100644 index 0000000000..8f296b7990 Binary files /dev/null and b/test/packages/kubernetes/img/metricbeat_kubernetes_scheduler.png differ diff --git a/test/packages/kubernetes/kibana/dashboard/kubernetes-5e649d60-9901-11e9-ba57-b7ab4e2d4b58.json b/test/packages/kubernetes/kibana/dashboard/kubernetes-5e649d60-9901-11e9-ba57-b7ab4e2d4b58.json new file mode 100644 index 0000000000..233c4a4aad --- /dev/null +++ b/test/packages/kubernetes/kibana/dashboard/kubernetes-5e649d60-9901-11e9-ba57-b7ab4e2d4b58.json @@ -0,0 +1,188 @@ +{ + "attributes": { + "description": "Kubernetes Proxy metrics", + "hits": 0, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "kubernetes.proxy" + }, + "type": "phrase", + "value": "kubernetes.proxy" + }, + "query": { + "match": { + "data_stream.dataset": { + "query": "kubernetes.proxy", + "type": "phrase" + } + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "1", + "w": 23, + "x": 0, + "y": 12 + }, + "panelIndex": "1", + "panelRefName": "panel_0", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "2", + "w": 25, + "x": 23, + "y": 12 + }, + "panelIndex": "2", + "panelRefName": "panel_1", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "3", + "w": 9, + "x": 12, + "y": 0 + }, + "panelIndex": "3", + "panelRefName": "panel_2", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "4", + "w": 27, + "x": 21, + "y": 0 + }, + "panelIndex": "4", + "panelRefName": "panel_3", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 13, + "i": "5", + "w": 48, + "x": 0, + "y": 24 + }, + "panelIndex": "5", + "panelRefName": "panel_4", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 10, + "i": "6", + "w": 48, + "x": 0, + "y": 37 + }, + "panelIndex": "6", + "panelRefName": "panel_5", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "7", + "w": 12, + "x": 0, + "y": 0 + }, + "panelIndex": "7", + "panelRefName": "panel_6", + "version": "7.3.0" + } + ], + "timeRestore": false, + "title": "[Metrics Kubernetes] Proxy", + "version": 1 + }, + "id": "kubernetes-5e649d60-9901-11e9-ba57-b7ab4e2d4b58", + "migrationVersion": { + "dashboard": "7.3.0" + }, + "references": [ + { + "id": "metrics-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "kubernetes-e0ddd3e0-98fe-11e9-ba57-b7ab4e2d4b58", + "name": "panel_0", + "type": "visualization" + }, + { + "id": "kubernetes-cac89fb0-9906-11e9-ba57-b7ab4e2d4b58", + "name": "panel_1", + "type": "visualization" + }, + { + "id": "kubernetes-09b404f0-99af-11e9-ba57-b7ab4e2d4b58", + "name": "panel_2", + "type": "visualization" + }, + { + "id": "kubernetes-8c6c2690-9bd8-11e9-9dc8-fd27291d427f", + "name": "panel_3", + "type": "visualization" + }, + { + "id": "kubernetes-b8a24790-9bf0-11e9-9dc8-fd27291d427f", + "name": "panel_4", + "type": "visualization" + }, + { + "id": "kubernetes-ba7bf750-9bf5-11e9-9dc8-fd27291d427f", + "name": "panel_5", + "type": "visualization" + }, + { + "id": "kubernetes-bcb194a0-9bf8-11e9-9dc8-fd27291d427f", + "name": "panel_6", + "type": "visualization" + } + ], + "type": "dashboard" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/dashboard/kubernetes-97312060-9c1b-11e9-9dc8-fd27291d427f.json b/test/packages/kubernetes/kibana/dashboard/kubernetes-97312060-9c1b-11e9-9dc8-fd27291d427f.json new file mode 100644 index 0000000000..682f5713db --- /dev/null +++ b/test/packages/kubernetes/kibana/dashboard/kubernetes-97312060-9c1b-11e9-9dc8-fd27291d427f.json @@ -0,0 +1,224 @@ +{ + "attributes": { + "description": "Kubernetes Controller Manager metrics", + "hits": 0, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "kubernetes.controllermanager" + }, + "type": "phrase", + "value": "kubernetes.controllermanager" + }, + "query": { + "match": { + "data_stream.dataset": { + "query": "kubernetes.controllermanager", + "type": "phrase" + } + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": {}, + "gridData": { + "h": 14, + "i": "1", + "w": 11, + "x": 0, + "y": 0 + }, + "panelIndex": "1", + "panelRefName": "panel_0", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 14, + "i": "2", + "w": 11, + "x": 11, + "y": 0 + }, + "panelIndex": "2", + "panelRefName": "panel_1", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 14, + "i": "3", + "w": 26, + "x": 22, + "y": 0 + }, + "panelIndex": "3", + "panelRefName": "panel_2", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "5", + "w": 24, + "x": 0, + "y": 39 + }, + "panelIndex": "5", + "panelRefName": "panel_3", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "6", + "w": 24, + "x": 24, + "y": 39 + }, + "panelIndex": "6", + "panelRefName": "panel_4", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 13, + "i": "7", + "w": 24, + "x": 24, + "y": 14 + }, + "panelIndex": "7", + "panelRefName": "panel_5", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "8", + "w": 24, + "x": 0, + "y": 27 + }, + "panelIndex": "8", + "panelRefName": "panel_6", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "9", + "w": 24, + "x": 24, + "y": 27 + }, + "panelIndex": "9", + "panelRefName": "panel_7", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 13, + "i": "10", + "w": 24, + "x": 0, + "y": 14 + }, + "panelIndex": "10", + "panelRefName": "panel_8", + "version": "7.3.0" + } + ], + "timeRestore": false, + "title": "[Metrics Kubernetes] Controller Manager", + "version": 1 + }, + "id": "kubernetes-97312060-9c1b-11e9-9dc8-fd27291d427f", + "migrationVersion": { + "dashboard": "7.3.0" + }, + "references": [ + { + "id": "metrics-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "kubernetes-bcb194a0-9bf8-11e9-9dc8-fd27291d427f", + "name": "panel_0", + "type": "visualization" + }, + { + "id": "kubernetes-3dbf6230-9c20-11e9-9dc8-fd27291d427f", + "name": "panel_1", + "type": "visualization" + }, + { + "id": "kubernetes-d86b2da0-9c20-11e9-9dc8-fd27291d427f", + "name": "panel_2", + "type": "visualization" + }, + { + "id": "kubernetes-0ca95350-9c24-11e9-9dc8-fd27291d427f", + "name": "panel_3", + "type": "visualization" + }, + { + "id": "kubernetes-3e1e1fd0-9c27-11e9-9dc8-fd27291d427f", + "name": "panel_4", + "type": "visualization" + }, + { + "id": "kubernetes-2ba628e0-9c2a-11e9-9dc8-fd27291d427f", + "name": "panel_5", + "type": "visualization" + }, + { + "id": "kubernetes-8a95de50-9c38-11e9-9dc8-fd27291d427f", + "name": "panel_6", + "type": "visualization" + }, + { + "id": "kubernetes-485c8550-9c3a-11e9-9dc8-fd27291d427f", + "name": "panel_7", + "type": "visualization" + }, + { + "id": "kubernetes-7d80f790-9d96-11e9-b2ae-49acc4cbcea9", + "name": "panel_8", + "type": "visualization" + } + ], + "type": "dashboard" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/dashboard/kubernetes-AV4RGUqo5NkDleZmzKuZ.json b/test/packages/kubernetes/kibana/dashboard/kubernetes-AV4RGUqo5NkDleZmzKuZ.json new file mode 100644 index 0000000000..0e6477a380 --- /dev/null +++ b/test/packages/kubernetes/kibana/dashboard/kubernetes-AV4RGUqo5NkDleZmzKuZ.json @@ -0,0 +1,267 @@ +{ + "attributes": { + "description": "Overview of Kubernetes cluster metrics", + "hits": 0, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "highlightAll": true, + "query": { + "language": "kuery", + "query": "" + }, + "version": true + } + }, + "optionsJSON": { + "darkTheme": false, + "useMargins": false + }, + "panelsJSON": [ + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "1", + "w": 24, + "x": 24, + "y": 0 + }, + "panelIndex": "1", + "panelRefName": "panel_0", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "2", + "w": 24, + "x": 0, + "y": 24 + }, + "panelIndex": "2", + "panelRefName": "panel_1", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "5", + "w": 12, + "x": 12, + "y": 0 + }, + "panelIndex": "5", + "panelRefName": "panel_2", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "6", + "w": 8, + "x": 0, + "y": 12 + }, + "panelIndex": "6", + "panelRefName": "panel_3", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "7", + "w": 24, + "x": 24, + "y": 24 + }, + "panelIndex": "7", + "panelRefName": "panel_4", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "8", + "w": 24, + "x": 24, + "y": 36 + }, + "panelIndex": "8", + "panelRefName": "panel_5", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "9", + "w": 24, + "x": 0, + "y": 36 + }, + "panelIndex": "9", + "panelRefName": "panel_6", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "10", + "w": 12, + "x": 0, + "y": 0 + }, + "panelIndex": "10", + "panelRefName": "panel_7", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "11", + "w": 24, + "x": 0, + "y": 48 + }, + "panelIndex": "11", + "panelRefName": "panel_8", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "12", + "w": 24, + "x": 24, + "y": 48 + }, + "panelIndex": "12", + "panelRefName": "panel_9", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "13", + "w": 8, + "x": 16, + "y": 12 + }, + "panelIndex": "13", + "panelRefName": "panel_10", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "14", + "w": 24, + "x": 24, + "y": 12 + }, + "panelIndex": "14", + "panelRefName": "panel_11", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 12, + "i": "15", + "w": 8, + "x": 8, + "y": 12 + }, + "panelIndex": "15", + "panelRefName": "panel_12", + "version": "7.3.0" + } + ], + "timeRestore": false, + "title": "[Metrics Kubernetes] Overview", + "version": 1 + }, + "id": "kubernetes-AV4RGUqo5NkDleZmzKuZ", + "migrationVersion": { + "dashboard": "7.3.0" + }, + "references": [ + { + "id": "kubernetes-022a54c0-2bf5-11e7-859b-f78b612cde28", + "name": "panel_0", + "type": "visualization" + }, + { + "id": "kubernetes-44f12b40-2bf4-11e7-859b-f78b612cde28", + "name": "panel_1", + "type": "visualization" + }, + { + "id": "kubernetes-cd059410-2bfb-11e7-859b-f78b612cde28", + "name": "panel_2", + "type": "visualization" + }, + { + "id": "kubernetes-e1018b90-2bfb-11e7-859b-f78b612cde28", + "name": "panel_3", + "type": "visualization" + }, + { + "id": "kubernetes-d6564360-2bfc-11e7-859b-f78b612cde28", + "name": "panel_4", + "type": "visualization" + }, + { + "id": "kubernetes-16fa4470-2bfd-11e7-859b-f78b612cde28", + "name": "panel_5", + "type": "visualization" + }, + { + "id": "kubernetes-294546b0-30d6-11e7-8df8-6d3604a72912", + "name": "panel_6", + "type": "visualization" + }, + { + "id": "kubernetes-408fccf0-30d6-11e7-8df8-6d3604a72912", + "name": "panel_7", + "type": "visualization" + }, + { + "id": "kubernetes-58e644f0-30d6-11e7-8df8-6d3604a72912", + "name": "panel_8", + "type": "visualization" + }, + { + "id": "kubernetes-a4c9d360-30df-11e7-8df8-6d3604a72912", + "name": "panel_9", + "type": "visualization" + }, + { + "id": "kubernetes-174a6ad0-30e0-11e7-8df8-6d3604a72912", + "name": "panel_10", + "type": "visualization" + }, + { + "id": "kubernetes-7aac4fd0-30e0-11e7-8df8-6d3604a72912", + "name": "panel_11", + "type": "visualization" + }, + { + "id": "kubernetes-da1ff7c0-30ed-11e7-b9e5-2b5b07213ab3", + "name": "panel_12", + "type": "visualization" + } + ], + "type": "dashboard" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/dashboard/kubernetes-af7225b0-5794-11e8-afa2-e9067ea62228.json b/test/packages/kubernetes/kibana/dashboard/kubernetes-af7225b0-5794-11e8-afa2-e9067ea62228.json new file mode 100644 index 0000000000..ccc9e8e7fe --- /dev/null +++ b/test/packages/kubernetes/kibana/dashboard/kubernetes-af7225b0-5794-11e8-afa2-e9067ea62228.json @@ -0,0 +1,88 @@ +{ + "attributes": { + "description": "Kubernetes API server metrics", + "hits": 0, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "highlightAll": true, + "query": { + "language": "kuery", + "query": "" + }, + "version": true + } + }, + "optionsJSON": { + "darkTheme": false, + "hidePanelTitles": false, + "useMargins": false + }, + "panelsJSON": [ + { + "embeddableConfig": {}, + "gridData": { + "h": 24, + "i": "1", + "w": 24, + "x": 0, + "y": 22 + }, + "panelIndex": "1", + "panelRefName": "panel_0", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 22, + "i": "3", + "w": 48, + "x": 0, + "y": 0 + }, + "panelIndex": "3", + "panelRefName": "panel_1", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 24, + "i": "4", + "w": 24, + "x": 24, + "y": 22 + }, + "panelIndex": "4", + "panelRefName": "panel_2", + "version": "7.3.0" + } + ], + "timeRestore": false, + "title": "[Metrics Kubernetes] API server", + "version": 1 + }, + "id": "kubernetes-af7225b0-5794-11e8-afa2-e9067ea62228", + "migrationVersion": { + "dashboard": "7.3.0" + }, + "references": [ + { + "id": "kubernetes-7cbeb750-5794-11e8-afa2-e9067ea62228", + "name": "panel_0", + "type": "visualization" + }, + { + "id": "kubernetes-ec360ff0-57a0-11e8-afa2-e9067ea62228", + "name": "panel_1", + "type": "visualization" + }, + { + "id": "kubernetes-95a7f110-57a2-11e8-afa2-e9067ea62228", + "name": "panel_2", + "type": "visualization" + } + ], + "type": "dashboard" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/dashboard/kubernetes-f5ab5510-9c94-11e9-94fd-c91206cd5249.json b/test/packages/kubernetes/kibana/dashboard/kubernetes-f5ab5510-9c94-11e9-94fd-c91206cd5249.json new file mode 100644 index 0000000000..ce3372ed6d --- /dev/null +++ b/test/packages/kubernetes/kibana/dashboard/kubernetes-f5ab5510-9c94-11e9-94fd-c91206cd5249.json @@ -0,0 +1,188 @@ +{ + "attributes": { + "description": "Kubernetes Scheduler metrics", + "hits": 0, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "kubernetes.scheduler" + }, + "type": "phrase", + "value": "kubernetes.scheduler" + }, + "query": { + "match": { + "data_stream.dataset": { + "query": "kubernetes.scheduler", + "type": "phrase" + } + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": {}, + "gridData": { + "h": 14, + "i": "1", + "w": 14, + "x": 0, + "y": 0 + }, + "panelIndex": "1", + "panelRefName": "panel_0", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 14, + "i": "2", + "w": 11, + "x": 14, + "y": 0 + }, + "panelIndex": "2", + "panelRefName": "panel_1", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 14, + "i": "3", + "w": 23, + "x": 25, + "y": 0 + }, + "panelIndex": "3", + "panelRefName": "panel_2", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 11, + "i": "4", + "w": 24, + "x": 0, + "y": 29 + }, + "panelIndex": "4", + "panelRefName": "panel_3", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 11, + "i": "5", + "w": 24, + "x": 24, + "y": 29 + }, + "panelIndex": "5", + "panelRefName": "panel_4", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 15, + "i": "6", + "w": 24, + "x": 0, + "y": 14 + }, + "panelIndex": "6", + "panelRefName": "panel_5", + "version": "7.3.0" + }, + { + "embeddableConfig": {}, + "gridData": { + "h": 15, + "i": "7", + "w": 24, + "x": 24, + "y": 14 + }, + "panelIndex": "7", + "panelRefName": "panel_6", + "version": "7.3.0" + } + ], + "timeRestore": false, + "title": "[Metrics Kubernetes] Scheduler", + "version": 1 + }, + "id": "kubernetes-f5ab5510-9c94-11e9-94fd-c91206cd5249", + "migrationVersion": { + "dashboard": "7.3.0" + }, + "references": [ + { + "id": "metrics-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "kubernetes-bcb194a0-9bf8-11e9-9dc8-fd27291d427f", + "name": "panel_0", + "type": "visualization" + }, + { + "id": "kubernetes-826d80c0-9c97-11e9-94fd-c91206cd5249", + "name": "panel_1", + "type": "visualization" + }, + { + "id": "kubernetes-15bd4420-9c9b-11e9-94fd-c91206cd5249", + "name": "panel_2", + "type": "visualization" + }, + { + "id": "kubernetes-d9fc1b80-9c9c-11e9-94fd-c91206cd5249", + "name": "panel_3", + "type": "visualization" + }, + { + "id": "kubernetes-95595810-9ca8-11e9-94fd-c91206cd5249", + "name": "panel_4", + "type": "visualization" + }, + { + "id": "kubernetes-382ace30-9d98-11e9-b2ae-49acc4cbcea9", + "name": "panel_5", + "type": "visualization" + }, + { + "id": "kubernetes-84d9b200-9d98-11e9-b2ae-49acc4cbcea9", + "name": "panel_6", + "type": "visualization" + } + ], + "type": "dashboard" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-022a54c0-2bf5-11e7-859b-f78b612cde28.json b/test/packages/kubernetes/kibana/visualization/kubernetes-022a54c0-2bf5-11e7-859b-f78b612cde28.json new file mode 100644 index 0000000000..f4bbac9730 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-022a54c0-2bf5-11e7-859b-f78b612cde28.json @@ -0,0 +1,75 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.state_deployment" + } + } + }, + "title": "Available pods per deployment [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "filter": { + "language": "lucene", + "query": "data_stream.dataset:kubernetes.state_deployment" + }, + "id": "117fadf0-30df-11e7-8df8-6d3604a72912", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(104,188,0,1)", + "fill": 0.5, + "formatter": "number", + "id": "64456840-30df-11e7-8df8-6d3604a72912", + "label": "Available pods", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.deployment.replicas.available", + "id": "64456841-30df-11e7-8df8-6d3604a72912", + "type": "avg" + } + ], + "point_size": 1, + "seperate_axis": 0, + "split_color_mode": "gradient", + "split_filters": [ + { + "color": "#68BC00", + "id": "53d35ad0-30df-11e7-8df8-6d3604a72912" + } + ], + "split_mode": "terms", + "stacked": "stacked", + "terms_field": "kubernetes.deployment.name", + "terms_size": "10000" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Available pods per deployment [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-022a54c0-2bf5-11e7-859b-f78b612cde28", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-09b404f0-99af-11e9-ba57-b7ab4e2d4b58.json b/test/packages/kubernetes/kibana/visualization/kubernetes-09b404f0-99af-11e9-ba57-b7ab4e2d4b58.json new file mode 100644 index 0000000000..ddea509f5b --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-09b404f0-99af-11e9-ba57-b7ab4e2d4b58.json @@ -0,0 +1,159 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.proxy" + } + } + }, + "title": "Proxy process [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "30s", + "markdown": "\n| | |\n|---|---|\n|**Days running**|{{ math.started.last.raw }}|\n|**File descriptors open**|{{ average_of_kubernetes_proxy_process_fds_open_count.fds.last.raw }}|\n|**Resident Memory**|{{ average_of_kubernetes_proxy_process_memory_resident_bytes.resident_memory.last.formatted }}|\n|**Virtual Memory**|{{ average_of_kubernetes_proxy_process_memory_virtual_bytes.virtual_memory.last.formatted }}|\n\n", + "markdown_css": "#markdown-61ca57f0-469d-11e7-af02-69e470af7417 table,#markdown-61ca57f0-469d-11e7-af02-69e470af7417 tr,#markdown-61ca57f0-469d-11e7-af02-69e470af7417 td,#markdown-61ca57f0-469d-11e7-af02-69e470af7417 th{border:none}", + "markdown_less": "\ntable, tr, td, th {\n border: none;\n}\n\n", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.process.started.sec", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "script": "", + "type": "max", + "variables": [ + { + "id": "1bbc4a20-99ac-11e9-8beb-c3bf9b9dfc43", + "name": "v" + } + ] + }, + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "1f34f620-99ac-11e9-8beb-c3bf9b9dfc43", + "script": "round( (params._timestamp /1000 - params.started) / 86400, 2)", + "type": "math", + "variables": [ + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "1f371900-99ac-11e9-8beb-c3bf9b9dfc43", + "name": "started" + } + ] + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "var_name": "started" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "cc3b1700-99ac-11e9-8beb-c3bf9b9dfc43", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.process.fds.open.count", + "id": "cc3b1701-99ac-11e9-8beb-c3bf9b9dfc43", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "var_name": "fds" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "bytes", + "id": "da1b1270-99ad-11e9-8beb-c3bf9b9dfc43", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.process.memory.resident.bytes", + "id": "da1b1271-99ad-11e9-8beb-c3bf9b9dfc43", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "var_name": "resident_memory" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "bytes", + "id": "50f54000-99ae-11e9-8beb-c3bf9b9dfc43", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.process.memory.virtual.bytes", + "id": "50f54001-99ae-11e9-8beb-c3bf9b9dfc43", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "var_name": "virtual_memory" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "markdown" + }, + "title": "Proxy process [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-09b404f0-99af-11e9-ba57-b7ab4e2d4b58", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-0ca95350-9c24-11e9-9dc8-fd27291d427f.json b/test/packages/kubernetes/kibana/visualization/kubernetes-0ca95350-9c24-11e9-9dc8-fd27291d427f.json new file mode 100644 index 0000000000..9a70405e11 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-0ca95350-9c24-11e9-9dc8-fd27291d427f.json @@ -0,0 +1,71 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.controllermanager" + } + } + }, + "title": "Controller Longest running processor [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_min": "0", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.workqueue.longestrunning.sec", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "rainbow", + "split_mode": "terms", + "stacked": "none", + "terms_direction": "desc", + "terms_field": "kubernetes.controllermanager.name", + "terms_order_by": "61ca57f2-469d-11e7-af02-69e470af7417", + "terms_size": "5" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Controller Longest running processor [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-0ca95350-9c24-11e9-9dc8-fd27291d427f", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-15bd4420-9c9b-11e9-94fd-c91206cd5249.json b/test/packages/kubernetes/kibana/visualization/kubernetes-15bd4420-9c9b-11e9-94fd-c91206cd5249.json new file mode 100644 index 0000000000..03675f8b08 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-15bd4420-9c9b-11e9-94fd-c91206cd5249.json @@ -0,0 +1,110 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.scheduler" + } + } + }, + "title": "Scheduler HTTP request duration [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(165,228,85,1)", + "fill": 0.5, + "formatter": "us,ms,2", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "P99", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.http.request.duration.us.percentile.99", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "us,ms,2", + "id": "708a5a50-9c9b-11e9-b248-53caaa33a2c5", + "label": "P90", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.http.request.duration.us.percentile.90", + "id": "708a8160-9c9b-11e9-b248-53caaa33a2c5", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(65,117,0,1)", + "fill": 0.5, + "formatter": "us,ms,2", + "id": "9ba449d0-9c9b-11e9-b248-53caaa33a2c5", + "label": "P50", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.http.request.duration.us.percentile.50", + "id": "9ba449d1-9c9b-11e9-b248-53caaa33a2c5", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "timeseries" + }, + "title": "Scheduler HTTP request duration [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-15bd4420-9c9b-11e9-94fd-c91206cd5249", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-16fa4470-2bfd-11e7-859b-f78b612cde28.json b/test/packages/kubernetes/kibana/visualization/kubernetes-16fa4470-2bfd-11e7-859b-f78b612cde28.json new file mode 100644 index 0000000000..585c28cd1f --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-16fa4470-2bfd-11e7-859b-f78b612cde28.json @@ -0,0 +1,100 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.pod" + } + } + }, + "title": "Network in by node [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "listeners": {}, + "params": { + "axis_formatter": "number", + "axis_position": "left", + "background_color_rules": [ + { + "id": "12c1f2f0-2bf2-11e7-859b-f78b612cde28" + } + ], + "bar_color_rules": [ + { + "id": "1373ddd0-2bf2-11e7-859b-f78b612cde28" + } + ], + "filter": { + "language": "lucene", + "query": "data_stream.dataset:kubernetes.pod" + }, + "gauge_color_rules": [ + { + "id": "140e4910-2bf2-11e7-859b-f78b612cde28" + } + ], + "gauge_inner_width": 10, + "gauge_style": "half", + "gauge_width": 10, + "id": "0d5c6b10-2bf2-11e7-859b-f78b612cde28", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(104,188,0,1)", + "fill": 0.5, + "formatter": "bytes", + "id": "0d5c9220-2bf2-11e7-859b-f78b612cde28", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.pod.network.rx.bytes", + "id": "0d5c9221-2bf2-11e7-859b-f78b612cde28", + "type": "max" + }, + { + "field": "0d5c9221-2bf2-11e7-859b-f78b612cde28", + "id": "494fc310-2bf7-11e7-859b-f78b612cde28", + "type": "derivative", + "unit": "" + }, + { + "field": "494fc310-2bf7-11e7-859b-f78b612cde28", + "id": "37c72a70-3598-11e7-aa4a-8313a0c92a88", + "type": "positive_only", + "unit": "" + } + ], + "point_size": 1, + "seperate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "terms", + "stacked": "stacked", + "terms_field": "kubernetes.node.name", + "terms_order_by": "0d5c9221-2bf2-11e7-859b-f78b612cde28", + "terms_size": "100000" + } + ], + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Network in by node [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-16fa4470-2bfd-11e7-859b-f78b612cde28", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-174a6ad0-30e0-11e7-8df8-6d3604a72912.json b/test/packages/kubernetes/kibana/visualization/kubernetes-174a6ad0-30e0-11e7-8df8-6d3604a72912.json new file mode 100644 index 0000000000..1474b18284 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-174a6ad0-30e0-11e7-8df8-6d3604a72912.json @@ -0,0 +1,88 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.state_deployment" + } + } + }, + "title": "Unavailable pods [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "listeners": {}, + "params": { + "axis_formatter": "number", + "axis_position": "left", + "background_color_rules": [ + { + "id": "508ffb30-30d5-11e7-8df8-6d3604a72912" + } + ], + "bar_color_rules": [ + { + "id": "674d83b0-30d5-11e7-8df8-6d3604a72912" + } + ], + "filter": { + "language": "lucene", + "query": "data_stream.dataset:kubernetes.state_deployment" + }, + "gauge_color_rules": [ + { + "id": "50f9b980-30d5-11e7-8df8-6d3604a72912" + } + ], + "gauge_inner_width": "10", + "gauge_max": "", + "gauge_style": "half", + "gauge_width": "10", + "id": "2fe9d3b0-30d5-11e7-8df8-6d3604a72912", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "2fe9d3b1-30d5-11e7-8df8-6d3604a72912", + "label": "Unavailable Pods", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.deployment.replicas.unavailable", + "id": "54cf79a0-30d5-11e7-8df8-6d3604a72912", + "type": "sum" + } + ], + "override_index_pattern": 1, + "point_size": 1, + "seperate_axis": 0, + "series_interval": "10s", + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_legend": 1, + "time_field": "@timestamp", + "type": "metric" + }, + "title": "Unavailable pods [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-174a6ad0-30e0-11e7-8df8-6d3604a72912", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-294546b0-30d6-11e7-8df8-6d3604a72912.json b/test/packages/kubernetes/kibana/visualization/kubernetes-294546b0-30d6-11e7-8df8-6d3604a72912.json new file mode 100644 index 0000000000..b4e7814492 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-294546b0-30d6-11e7-8df8-6d3604a72912.json @@ -0,0 +1,100 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.pod" + } + } + }, + "title": "Network out by node [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "listeners": {}, + "params": { + "axis_formatter": "number", + "axis_position": "left", + "background_color_rules": [ + { + "id": "12c1f2f0-2bf2-11e7-859b-f78b612cde28" + } + ], + "bar_color_rules": [ + { + "id": "1373ddd0-2bf2-11e7-859b-f78b612cde28" + } + ], + "filter": { + "language": "lucene", + "query": "data_stream.dataset:kubernetes.pod" + }, + "gauge_color_rules": [ + { + "id": "140e4910-2bf2-11e7-859b-f78b612cde28" + } + ], + "gauge_inner_width": 10, + "gauge_style": "half", + "gauge_width": 10, + "id": "0d5c6b10-2bf2-11e7-859b-f78b612cde28", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(104,188,0,1)", + "fill": 0.5, + "formatter": "bytes", + "id": "0d5c9220-2bf2-11e7-859b-f78b612cde28", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.pod.network.tx.bytes", + "id": "0d5c9221-2bf2-11e7-859b-f78b612cde28", + "type": "max" + }, + { + "field": "0d5c9221-2bf2-11e7-859b-f78b612cde28", + "id": "494fc310-2bf7-11e7-859b-f78b612cde28", + "type": "derivative", + "unit": "" + }, + { + "field": "494fc310-2bf7-11e7-859b-f78b612cde28", + "id": "244c70e0-3598-11e7-aa4a-8313a0c92a88", + "type": "positive_only", + "unit": "" + } + ], + "point_size": 1, + "seperate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "terms", + "stacked": "stacked", + "terms_field": "kubernetes.node.name", + "terms_order_by": "0d5c9221-2bf2-11e7-859b-f78b612cde28", + "terms_size": "10000" + } + ], + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Network out by node [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-294546b0-30d6-11e7-8df8-6d3604a72912", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-2ba628e0-9c2a-11e9-9dc8-fd27291d427f.json b/test/packages/kubernetes/kibana/visualization/kubernetes-2ba628e0-9c2a-11e9-9dc8-fd27291d427f.json new file mode 100644 index 0000000000..d1c48c69aa --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-2ba628e0-9c2a-11e9-9dc8-fd27291d427f.json @@ -0,0 +1,88 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.controllermanager" + } + } + }, + "title": "Controller Memory [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(251,158,0,1)", + "fill": 0.5, + "formatter": "bytes", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "Resident", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.process.memory.resident.bytes", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(164,221,0,1)", + "fill": 0.5, + "formatter": "bytes", + "id": "ac896b30-9c29-11e9-92c1-f7d03186c592", + "label": "Virtual", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.process.memory.virtual.bytes", + "id": "ac896b31-9c29-11e9-92c1-f7d03186c592", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Controller Memory [Metricset Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-2ba628e0-9c2a-11e9-9dc8-fd27291d427f", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-382ace30-9d98-11e9-b2ae-49acc4cbcea9.json b/test/packages/kubernetes/kibana/visualization/kubernetes-382ace30-9d98-11e9-b2ae-49acc4cbcea9.json new file mode 100644 index 0000000000..7c9c31c295 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-382ace30-9d98-11e9-b2ae-49acc4cbcea9.json @@ -0,0 +1,74 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.scheduler" + } + } + }, + "title": "Scheduler CPU [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_min": "0", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "CPU time", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.process.cpu.sec", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + }, + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "11331120-9d98-11e9-9e81-115d18bcfeaa", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "value_template": "{{value}} s" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "timeseries" + }, + "title": "Scheduler CPU [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-382ace30-9d98-11e9-b2ae-49acc4cbcea9", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-3dbf6230-9c20-11e9-9dc8-fd27291d427f.json b/test/packages/kubernetes/kibana/visualization/kubernetes-3dbf6230-9c20-11e9-9dc8-fd27291d427f.json new file mode 100644 index 0000000000..451510aca1 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-3dbf6230-9c20-11e9-9dc8-fd27291d427f.json @@ -0,0 +1,150 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.controllermanager" + } + } + }, + "title": "Controller Process [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "drop_last_bucket": 1, + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "30s", + "markdown": "| | |\n|---|---|\n|**Days running**|{{ math.started.last.raw }}|\n|**File descriptors open**|{{ average_of_kubernetes_controllermanager_process_fds_open_count.fds.last.raw }}|\n|**Resident Memory**|{{ average_of_kubernetes_controllermanager_process_memory_resident_bytes.resident_memory.last.formatted }}|\n|**Virtual Memory**|{{ average_of_kubernetes_controllermanager_process_memory_virtual_bytes.virtual_memory.last.formatted }}|\n", + "markdown_css": "#markdown-61ca57f0-469d-11e7-af02-69e470af7417 table,#markdown-61ca57f0-469d-11e7-af02-69e470af7417 tr,#markdown-61ca57f0-469d-11e7-af02-69e470af7417 td,#markdown-61ca57f0-469d-11e7-af02-69e470af7417 th{border:none}", + "markdown_less": "\ntable, tr, td, th {\n border: none;\n}\n\n", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.process.started.sec", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "max" + }, + { + "id": "ebfdb1c0-9c1c-11e9-b29f-d55be9348723", + "script": "round( (params._timestamp /1000 - params.started) / 86400, 2)", + "type": "math", + "variables": [ + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "ef30a230-9c1c-11e9-b29f-d55be9348723", + "name": "started" + } + ] + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "var_name": "started" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "1212ba80-9c1e-11e9-b29f-d55be9348723", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.process.fds.open.count", + "id": "1212ba81-9c1e-11e9-b29f-d55be9348723", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "var_name": "fds" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "bytes", + "id": "3f3b17a0-9c1e-11e9-b29f-d55be9348723", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.process.memory.resident.bytes", + "id": "3f3b17a1-9c1e-11e9-b29f-d55be9348723", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "var_name": "resident_memory" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "bytes", + "id": "b1a01b50-9c1f-11e9-b29f-d55be9348723", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.process.memory.virtual.bytes", + "id": "b1a01b51-9c1f-11e9-b29f-d55be9348723", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "var_name": "virtual_memory" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "markdown" + }, + "title": "Controller Process [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-3dbf6230-9c20-11e9-9dc8-fd27291d427f", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-3e1e1fd0-9c27-11e9-9dc8-fd27291d427f.json b/test/packages/kubernetes/kibana/visualization/kubernetes-3e1e1fd0-9c27-11e9-9dc8-fd27291d427f.json new file mode 100644 index 0000000000..bcc0b17026 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-3e1e1fd0-9c27-11e9-9dc8-fd27291d427f.json @@ -0,0 +1,69 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.controllermanager" + } + } + }, + "title": "Controller Unfinished jobs sec [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_min": "0", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.workqueue.unfinished.sec", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "rainbow", + "split_mode": "terms", + "stacked": "none", + "terms_field": "kubernetes.controllermanager.name", + "terms_order_by": "61ca57f2-469d-11e7-af02-69e470af7417", + "terms_size": "5" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Controller Unfinished jobs sec [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-3e1e1fd0-9c27-11e9-9dc8-fd27291d427f", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-408fccf0-30d6-11e7-8df8-6d3604a72912.json b/test/packages/kubernetes/kibana/visualization/kubernetes-408fccf0-30d6-11e7-8df8-6d3604a72912.json new file mode 100644 index 0000000000..b66b197e74 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-408fccf0-30d6-11e7-8df8-6d3604a72912.json @@ -0,0 +1,86 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.state_node" + } + } + }, + "title": "Nodes [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "listeners": {}, + "params": { + "axis_formatter": "number", + "axis_position": "left", + "background_color_rules": [ + { + "id": "67ee7da0-30e0-11e7-8df8-6d3604a72912" + } + ], + "bar_color_rules": [ + { + "id": "68cdba10-30e0-11e7-8df8-6d3604a72912" + } + ], + "filter": { + "language": "lucene", + "query": "data_stream.dataset:kubernetes.state_node" + }, + "gauge_color_rules": [ + { + "id": "69765620-30e0-11e7-8df8-6d3604a72912" + } + ], + "gauge_inner_width": 10, + "gauge_style": "half", + "gauge_width": 10, + "id": "4c4690b0-30e0-11e7-8df8-6d3604a72912", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "4c4690b1-30e0-11e7-8df8-6d3604a72912", + "label": "Nodes", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.node.name", + "id": "4c4690b2-30e0-11e7-8df8-6d3604a72912", + "type": "cardinality" + } + ], + "point_size": 1, + "seperate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "terms_field": "kubernetes.deployment.name" + } + ], + "show_legend": 1, + "time_field": "@timestamp", + "type": "metric" + }, + "title": "Nodes [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-408fccf0-30d6-11e7-8df8-6d3604a72912", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-44f12b40-2bf4-11e7-859b-f78b612cde28.json b/test/packages/kubernetes/kibana/visualization/kubernetes-44f12b40-2bf4-11e7-859b-f78b612cde28.json new file mode 100644 index 0000000000..2a9b2fad70 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-44f12b40-2bf4-11e7-859b-f78b612cde28.json @@ -0,0 +1,155 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "title": "CPU usage by node [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "listeners": {}, + "params": { + "axis_formatter": "number", + "axis_position": "left", + "background_color_rules": [ + { + "id": "12c1f2f0-2bf2-11e7-859b-f78b612cde28" + } + ], + "bar_color_rules": [ + { + "id": "1373ddd0-2bf2-11e7-859b-f78b612cde28" + } + ], + "filter": { + "language": "lucene", + "query": "(data_stream.dataset:kubernetes.container OR data_stream.dataset:kubernetes.state_node)" + }, + "gauge_color_rules": [ + { + "id": "140e4910-2bf2-11e7-859b-f78b612cde28" + } + ], + "gauge_inner_width": 10, + "gauge_style": "half", + "gauge_width": 10, + "id": "0d5c6b10-2bf2-11e7-859b-f78b612cde28", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": "0.5", + "formatter": "0.0a", + "id": "0d5c9220-2bf2-11e7-859b-f78b612cde28", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.container.cpu.usage.nanocores", + "id": "0d5c9221-2bf2-11e7-859b-f78b612cde28", + "type": "sum" + }, + { + "field": "metricset.period", + "id": "8b346300-bf95-11ea-a07c-851701f0d645", + "type": "avg" + }, + { + "id": "25ae6580-bf95-11ea-a07c-851701f0d645", + "script": "params.sum_nanocores / (params._interval / params.avg_period)", + "type": "calculation", + "variables": [ + { + "field": "0d5c9221-2bf2-11e7-859b-f78b612cde28", + "id": "39e40aa0-bf95-11ea-a07c-851701f0d645", + "name": "sum_nanocores" + }, + { + "field": "8b346300-bf95-11ea-a07c-851701f0d645", + "id": "85213600-bf95-11ea-a07c-851701f0d645", + "name": "avg_period" + } + ] + } + ], + "override_index_pattern": 0, + "point_size": 1, + "seperate_axis": 0, + "series_interval": "10s", + "series_time_field": "@timestamp", + "split_color_mode": "gradient", + "split_mode": "terms", + "stacked": "none", + "terms_field": "kubernetes.node.name", + "terms_order_by": "0d5c9221-2bf2-11e7-859b-f78b612cde28", + "terms_size": "10000", + "value_template": "{{value}} nanocores" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(211,49,21,1)", + "fill": "0", + "formatter": "0.0a", + "hide_in_legend": 1, + "id": "22f65d40-31a7-11e7-84cc-096d2b38e6e5", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.node.cpu.capacity.cores", + "id": "22f65d41-31a7-11e7-84cc-096d2b38e6e5", + "type": "avg" + }, + { + "id": "4af4c390-34d6-11e7-be88-cb6a123dc1bb", + "script": "params.cores * 1000000000", + "type": "calculation", + "variables": [ + { + "field": "22f65d41-31a7-11e7-84cc-096d2b38e6e5", + "id": "4cd32080-34d6-11e7-be88-cb6a123dc1bb", + "name": "cores" + } + ] + } + ], + "override_index_pattern": 0, + "point_size": "0", + "seperate_axis": 0, + "series_interval": "10s", + "series_time_field": "@timestamp", + "split_color_mode": "gradient", + "split_mode": "terms", + "stacked": "none", + "terms_field": "kubernetes.node.name", + "terms_order_by": "22f65d41-31a7-11e7-84cc-096d2b38e6e5", + "terms_size": "10000", + "value_template": "{{value}} nanocores" + } + ], + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "CPU usage by node [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-44f12b40-2bf4-11e7-859b-f78b612cde28", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-485c8550-9c3a-11e9-9dc8-fd27291d427f.json b/test/packages/kubernetes/kibana/visualization/kubernetes-485c8550-9c3a-11e9-9dc8-fd27291d427f.json new file mode 100644 index 0000000000..45c3e3992d --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-485c8550-9c3a-11e9-9dc8-fd27291d427f.json @@ -0,0 +1,77 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.controllermanager" + } + } + }, + "title": "Controller Workqueue retries [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_min": "0", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.workqueue.retries.count", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + }, + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "b3107560-9c39-11e9-92c1-f7d03186c592", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "rainbow", + "split_mode": "terms", + "stacked": "none", + "terms_direction": "desc", + "terms_field": "kubernetes.controllermanager.name", + "terms_order_by": "61ca57f2-469d-11e7-af02-69e470af7417", + "terms_size": "5" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Controller Workqueue retries [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-485c8550-9c3a-11e9-9dc8-fd27291d427f", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-58e644f0-30d6-11e7-8df8-6d3604a72912.json b/test/packages/kubernetes/kibana/visualization/kubernetes-58e644f0-30d6-11e7-8df8-6d3604a72912.json new file mode 100644 index 0000000000..39a984ada5 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-58e644f0-30d6-11e7-8df8-6d3604a72912.json @@ -0,0 +1,88 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "title": "Top CPU intensive pods [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "listeners": {}, + "params": { + "axis_formatter": "number", + "axis_position": "left", + "bar_color_rules": [ + { + "id": "802104d0-2bfc-11e7-859b-f78b612cde28" + } + ], + "filter": { + "language": "lucene", + "query": "data_stream.dataset:kubernetes.container" + }, + "id": "5d3692a0-2bfc-11e7-859b-f78b612cde28", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "0.0 a", + "id": "5d3692a1-2bfc-11e7-859b-f78b612cde28", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.container.cpu.usage.core.ns", + "id": "5d3692a2-2bfc-11e7-859b-f78b612cde28", + "type": "max" + }, + { + "field": "5d3692a2-2bfc-11e7-859b-f78b612cde28", + "id": "6c905240-2bfc-11e7-859b-f78b612cde28", + "type": "derivative", + "unit": "1s" + }, + { + "field": "6c905240-2bfc-11e7-859b-f78b612cde28", + "id": "9a51f710-359d-11e7-aa4a-8313a0c92a88", + "type": "positive_only", + "unit": "" + } + ], + "offset_time": "", + "override_index_pattern": 0, + "point_size": 1, + "seperate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "terms", + "stacked": "none", + "terms_field": "kubernetes.pod.name", + "terms_order_by": "5d3692a2-2bfc-11e7-859b-f78b612cde28", + "value_template": "{{value}} ns" + } + ], + "show_legend": 1, + "time_field": "@timestamp", + "type": "top_n" + }, + "title": "Top CPU intensive pods [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-58e644f0-30d6-11e7-8df8-6d3604a72912", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-7aac4fd0-30e0-11e7-8df8-6d3604a72912.json b/test/packages/kubernetes/kibana/visualization/kubernetes-7aac4fd0-30e0-11e7-8df8-6d3604a72912.json new file mode 100644 index 0000000000..cb3fb9ecbd --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-7aac4fd0-30e0-11e7-8df8-6d3604a72912.json @@ -0,0 +1,75 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.state_deployment" + } + } + }, + "title": "Unavailable pods per deployment [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "filter": { + "language": "lucene", + "query": "data_stream.dataset:kubernetes.state_deployment" + }, + "id": "117fadf0-30df-11e7-8df8-6d3604a72912", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(254,146,0,1)", + "fill": 0.5, + "formatter": "number", + "id": "64456840-30df-11e7-8df8-6d3604a72912", + "label": "Unavailable pods", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.deployment.replicas.unavailable", + "id": "64456841-30df-11e7-8df8-6d3604a72912", + "type": "avg" + } + ], + "point_size": 1, + "seperate_axis": 0, + "split_color_mode": "gradient", + "split_filters": [ + { + "color": "#68BC00", + "id": "53d35ad0-30df-11e7-8df8-6d3604a72912" + } + ], + "split_mode": "terms", + "stacked": "stacked", + "terms_field": "kubernetes.deployment.name", + "terms_size": "10000" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Unavailable pods per deployment [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-7aac4fd0-30e0-11e7-8df8-6d3604a72912", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-7cbeb750-5794-11e8-afa2-e9067ea62228.json b/test/packages/kubernetes/kibana/visualization/kubernetes-7cbeb750-5794-11e8-afa2-e9067ea62228.json new file mode 100644 index 0000000000..6a355b33ef --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-7cbeb750-5794-11e8-afa2-e9067ea62228.json @@ -0,0 +1,84 @@ +{ + "attributes": { + "description": "Overview of Kubernetes API Server", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.apiserver" + } + } + }, + "title": "API Server Top clients by number of requests [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "bar_color_rules": [ + { + "id": "61a13010-5794-11e8-8bd0-2180975e72dd" + } + ], + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "5m", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(104,188,0,1)", + "fill": 0.5, + "filter": "", + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "Top clients by number of requests (5m)", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.apiserver.request.count", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "max" + }, + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "0e6ff4a0-5792-11e8-8bd0-2180975e72dd", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "seperate_axis": 0, + "split_color_mode": "gradient", + "split_filters": [ + { + "color": "#68BC00", + "id": "9e4b8030-5792-11e8-8bd0-2180975e72dd" + } + ], + "split_mode": "terms", + "stacked": "stacked", + "terms_field": "kubernetes.apiserver.request.client", + "terms_order_by": "61ca57f2-469d-11e7-af02-69e470af7417", + "terms_size": "10" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "top_n" + }, + "title": "API Server Top clients by number of requests [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-7cbeb750-5794-11e8-afa2-e9067ea62228", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-7d80f790-9d96-11e9-b2ae-49acc4cbcea9.json b/test/packages/kubernetes/kibana/visualization/kubernetes-7d80f790-9d96-11e9-b2ae-49acc4cbcea9.json new file mode 100644 index 0000000000..d7ae83ab93 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-7d80f790-9d96-11e9-b2ae-49acc4cbcea9.json @@ -0,0 +1,74 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.controllermanager" + } + } + }, + "title": "Controller CPU [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_min": "0", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "CPU time", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.process.cpu.sec", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + }, + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "47731390-9d96-11e9-9e81-115d18bcfeaa", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "value_template": "{{value}} s" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "timeseries" + }, + "title": "Controller CPU [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-7d80f790-9d96-11e9-b2ae-49acc4cbcea9", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-826d80c0-9c97-11e9-94fd-c91206cd5249.json b/test/packages/kubernetes/kibana/visualization/kubernetes-826d80c0-9c97-11e9-94fd-c91206cd5249.json new file mode 100644 index 0000000000..3d8e19fbae --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-826d80c0-9c97-11e9-94fd-c91206cd5249.json @@ -0,0 +1,156 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.scheduler" + } + } + }, + "title": "Process summary [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "drop_last_bucket": 1, + "filter": { + "language": "lucene", + "query": "_exists_:\"kubernetes.scheduler.process.started.sec\"" + }, + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "30s", + "markdown": "| | |\n|---|---|\n|**Days running**|{{ math.started.last.raw }}|\n|**File descriptors open**|{{ max_of_kubernetes_scheduler_process_fds_open_count.fds.last.raw }}|\n|**Resident Memory**|{{ max_of_kubernetes_scheduler_process_memory_resident_bytes.resident_memory.last.formatted }}|\n|**Virtual Memory**|{{ max_of_kubernetes_scheduler_process_memory_virtual_bytes.virtual_memory.last.formatted }}|\n", + "markdown_css": "#markdown-61ca57f0-469d-11e7-af02-69e470af7417 table,#markdown-61ca57f0-469d-11e7-af02-69e470af7417 tr,#markdown-61ca57f0-469d-11e7-af02-69e470af7417 td,#markdown-61ca57f0-469d-11e7-af02-69e470af7417 th{border:none}", + "markdown_less": "\ntable, tr, td, th {\n border: none;\n}\n\n", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.process.started.sec", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "max" + }, + { + "id": "94b52370-9c95-11e9-87bc-455cc45f013a", + "script": "round( (params._timestamp /1000 - params.started) / 86400, 2)", + "type": "math", + "variables": [ + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "97cc0060-9c95-11e9-87bc-455cc45f013a", + "name": "started" + } + ] + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "var_name": "started" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "a8475ca0-9c95-11e9-87bc-455cc45f013a", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.process.fds.open.count", + "id": "a8475ca1-9c95-11e9-87bc-455cc45f013a", + "type": "max" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "var_name": "fds" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "bytes", + "id": "cdc6d190-9c95-11e9-87bc-455cc45f013a", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.process.memory.resident.bytes", + "id": "cdc6d191-9c95-11e9-87bc-455cc45f013a", + "type": "max" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "var_name": "resident_memory" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "bytes", + "id": "dfa0fa80-9c95-11e9-87bc-455cc45f013a", + "label": "", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.process.memory.virtual.bytes", + "id": "dfa0fa81-9c95-11e9-87bc-455cc45f013a", + "type": "max" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "var_name": "virtual_memory" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "markdown" + }, + "title": "Process summary [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-826d80c0-9c97-11e9-94fd-c91206cd5249", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-84d9b200-9d98-11e9-b2ae-49acc4cbcea9.json b/test/packages/kubernetes/kibana/visualization/kubernetes-84d9b200-9d98-11e9-b2ae-49acc4cbcea9.json new file mode 100644 index 0000000000..939a6a0581 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-84d9b200-9d98-11e9-b2ae-49acc4cbcea9.json @@ -0,0 +1,88 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.scheduler" + } + } + }, + "title": "Scheduler Memory [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(251,158,0,1)", + "fill": 0.5, + "formatter": "bytes", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "Resident", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.process.memory.resident.bytes", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "bytes", + "id": "a14defa0-9d98-11e9-9e81-115d18bcfeaa", + "label": "Virtual", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.process.memory.virtual.bytes", + "id": "a14defa1-9d98-11e9-9e81-115d18bcfeaa", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "timeseries" + }, + "title": "Scheduler Memory [Metricset Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-84d9b200-9d98-11e9-b2ae-49acc4cbcea9", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-8a95de50-9c38-11e9-9dc8-fd27291d427f.json b/test/packages/kubernetes/kibana/visualization/kubernetes-8a95de50-9c38-11e9-9dc8-fd27291d427f.json new file mode 100644 index 0000000000..3a2e67eda6 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-8a95de50-9c38-11e9-9dc8-fd27291d427f.json @@ -0,0 +1,74 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.controllermanager" + } + } + }, + "title": "Controller Workqueue adds [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.workqueue.adds.count", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + }, + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "9b5d85d0-9c38-11e9-92c1-f7d03186c592", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "rainbow", + "split_mode": "terms", + "stacked": "none", + "terms_field": "kubernetes.controllermanager.name", + "terms_order_by": "61ca57f2-469d-11e7-af02-69e470af7417", + "terms_size": "5" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Controller Workqueue adds [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-8a95de50-9c38-11e9-9dc8-fd27291d427f", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-8c6c2690-9bd8-11e9-9dc8-fd27291d427f.json b/test/packages/kubernetes/kibana/visualization/kubernetes-8c6c2690-9bd8-11e9-9dc8-fd27291d427f.json new file mode 100644 index 0000000000..0edb589986 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-8c6c2690-9bd8-11e9-9dc8-fd27291d427f.json @@ -0,0 +1,109 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.proxy" + } + } + }, + "title": "Proxy HTTP request duration [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(165,228,85,1)", + "fill": 0.5, + "formatter": "us,ms,2", + "id": "05a03f20-9bd8-11e9-871d-d3c7d4c337ef", + "label": "P99", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.http.request.duration.us.percentile.99", + "id": "05a06630-9bd8-11e9-871d-d3c7d4c337ef", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "us,ms,2", + "id": "dc0faec0-9bd7-11e9-871d-d3c7d4c337ef", + "label": "P90", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.http.request.duration.us.percentile.90", + "id": "dc0faec1-9bd7-11e9-871d-d3c7d4c337ef", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(65,117,0,1)", + "fill": 0.5, + "formatter": "us,ms,2", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "P50", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.http.request.duration.us.percentile.50", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Proxy HTTP request duration [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-8c6c2690-9bd8-11e9-9dc8-fd27291d427f", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-95595810-9ca8-11e9-94fd-c91206cd5249.json b/test/packages/kubernetes/kibana/visualization/kubernetes-95595810-9ca8-11e9-94fd-c91206cd5249.json new file mode 100644 index 0000000000..de3b1adc2e --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-95595810-9ca8-11e9-94fd-c91206cd5249.json @@ -0,0 +1,75 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.scheduler" + } + } + }, + "title": "Scheduler scheduling attempts [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_min": "0", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "Attempts", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.scheduling.pod.attempts.count", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + }, + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "18aae5d0-9ca9-11e9-b8ef-3ddf5d748ddb", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "rainbow", + "split_mode": "terms", + "stacked": "none", + "terms_field": "kubernetes.scheduler.result", + "terms_order_by": "61ca57f2-469d-11e7-af02-69e470af7417" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "timeseries" + }, + "title": "Scheduler scheduling attempts [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-95595810-9ca8-11e9-94fd-c91206cd5249", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-95a7f110-57a2-11e8-afa2-e9067ea62228.json b/test/packages/kubernetes/kibana/visualization/kubernetes-95a7f110-57a2-11e8-afa2-e9067ea62228.json new file mode 100644 index 0000000000..6c98692cca --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-95a7f110-57a2-11e8-afa2-e9067ea62228.json @@ -0,0 +1,85 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.apiserver" + } + } + }, + "title": "API Server Top clients by resource [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "bar_color_rules": [ + { + "id": "61a13010-5794-11e8-8bd0-2180975e72dd" + } + ], + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "5m", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(104,188,0,1)", + "fill": 0.5, + "filter": "", + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "Top clients by number of requests (5m)", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.apiserver.request.count", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "max" + }, + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "0e6ff4a0-5792-11e8-8bd0-2180975e72dd", + "type": "derivative", + "unit": "" + } + ], + "override_index_pattern": 0, + "point_size": 1, + "seperate_axis": 0, + "series_drop_last_bucket": 1, + "split_color_mode": "gradient", + "split_filters": [ + { + "color": "#68BC00", + "id": "9e4b8030-5792-11e8-8bd0-2180975e72dd" + } + ], + "split_mode": "terms", + "stacked": "stacked", + "terms_field": "kubernetes.apiserver.request.resource", + "terms_order_by": "61ca57f2-469d-11e7-af02-69e470af7417", + "terms_size": "10" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "top_n" + }, + "title": "API Server Top clients by resource [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-95a7f110-57a2-11e8-afa2-e9067ea62228", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-a4c9d360-30df-11e7-8df8-6d3604a72912.json b/test/packages/kubernetes/kibana/visualization/kubernetes-a4c9d360-30df-11e7-8df8-6d3604a72912.json new file mode 100644 index 0000000000..931814b3a8 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-a4c9d360-30df-11e7-8df8-6d3604a72912.json @@ -0,0 +1,88 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.container" + } + } + }, + "title": "Top memory intensive pods [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "bar_color_rules": [ + { + "id": "802104d0-2bfc-11e7-859b-f78b612cde28" + } + ], + "filter": { + "language": "lucene", + "query": "data_stream.dataset:kubernetes.container" + }, + "id": "5d3692a0-2bfc-11e7-859b-f78b612cde28", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "bytes", + "id": "5d3692a1-2bfc-11e7-859b-f78b612cde28", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.container.memory.usage.bytes", + "id": "5d3692a2-2bfc-11e7-859b-f78b612cde28", + "type": "sum" + }, + { + "field": "5d3692a2-2bfc-11e7-859b-f78b612cde28", + "id": "3972e9f0-256f-11e8-84e6-87221f87ae3b", + "type": "cumulative_sum" + }, + { + "field": "3972e9f0-256f-11e8-84e6-87221f87ae3b", + "id": "3e9fd5a0-256f-11e8-84e6-87221f87ae3b", + "type": "derivative", + "unit": "10s" + } + ], + "offset_time": "", + "override_index_pattern": 0, + "point_size": 1, + "seperate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "terms", + "stacked": "none", + "terms_field": "kubernetes.pod.name", + "terms_order_by": "5d3692a2-2bfc-11e7-859b-f78b612cde28", + "terms_size": "10", + "value_template": "" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "top_n" + }, + "title": "Top memory intensive pods [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-a4c9d360-30df-11e7-8df8-6d3604a72912", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-b8a24790-9bf0-11e9-9dc8-fd27291d427f.json b/test/packages/kubernetes/kibana/visualization/kubernetes-b8a24790-9bf0-11e9-9dc8-fd27291d427f.json new file mode 100644 index 0000000000..573d1c5c19 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-b8a24790-9bf0-11e9-9dc8-fd27291d427f.json @@ -0,0 +1,262 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.proxy" + } + } + }, + "title": "Proxy network programming [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_min": "0", + "axis_position": "left", + "axis_scale": "normal", + "background_color_rules": [ + { + "id": "4fb3cbb0-9be5-11e9-871d-d3c7d4c337ef" + } + ], + "bar_color_rules": [ + { + "id": "4b459c20-9be5-11e9-871d-d3c7d4c337ef" + } + ], + "default_index_pattern": "metrics-*", + "gauge_color_rules": [ + { + "id": "4de1cad0-9be5-11e9-871d-d3c7d4c337ef" + } + ], + "gauge_inner_width": 10, + "gauge_style": "half", + "gauge_width": 10, + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(63,112,2,1)", + "fill": 0.5, + "formatter": "number", + "hidden": false, + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "Under 16ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.networkprogramming.duration.us.bucket.16000", + "id": "4dbb3a90-9be6-11e9-871d-d3c7d4c337ef", + "type": "avg" + }, + { + "field": "4dbb3a90-9be6-11e9-871d-d3c7d4c337ef", + "id": "189688a0-9be7-11e9-871d-d3c7d4c337ef", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(138,223,24,1)", + "fill": "0.4", + "formatter": "number", + "hidden": false, + "id": "a60dbe60-9be7-11e9-871d-d3c7d4c337ef", + "label": "Under 512ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.networkprogramming.duration.us.bucket.512000", + "id": "a60dbe61-9be7-11e9-871d-d3c7d4c337ef", + "type": "avg" + }, + { + "field": "a60dbe61-9be7-11e9-871d-d3c7d4c337ef", + "id": "b55a1080-9be7-11e9-871d-d3c7d4c337ef", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(174,161,255,1)", + "fill": "0.4", + "formatter": "number", + "hidden": false, + "id": "c5cc4f90-9be8-11e9-871d-d3c7d4c337ef", + "label": "Under 1024 ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.networkprogramming.duration.us.bucket.1024000", + "id": "c5cc4f91-9be8-11e9-871d-d3c7d4c337ef", + "type": "avg" + }, + { + "field": "c5cc4f91-9be8-11e9-871d-d3c7d4c337ef", + "id": "f17cb2b0-9be8-11e9-871d-d3c7d4c337ef", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(115,216,255,1)", + "fill": "0.2", + "formatter": "number", + "hidden": false, + "id": "e0901380-9be7-11e9-871d-d3c7d4c337ef", + "label": "Under 4096 ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.networkprogramming.duration.us.bucket.4096000", + "id": "e0901381-9be7-11e9-871d-d3c7d4c337ef", + "type": "avg" + }, + { + "field": "e0901381-9be7-11e9-871d-d3c7d4c337ef", + "id": "f2b6fce0-9be7-11e9-871d-d3c7d4c337ef", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(252,196,0,1)", + "fill": "0.2", + "formatter": "number", + "id": "882fa5a0-9be9-11e9-871d-d3c7d4c337ef", + "label": "Under 8192 ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.networkprogramming.duration.us.bucket.8192000", + "id": "882fccb0-9be9-11e9-871d-d3c7d4c337ef", + "type": "avg" + }, + { + "field": "882fccb0-9be9-11e9-871d-d3c7d4c337ef", + "id": "9adb9d80-9be9-11e9-871d-d3c7d4c337ef", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(209,119,103,1)", + "fill": "0", + "formatter": "number", + "id": "86549550-9bef-11e9-871d-d3c7d4c337ef", + "label": "Under 16384 ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.networkprogramming.duration.us.bucket.16384000", + "id": "86549551-9bef-11e9-871d-d3c7d4c337ef", + "type": "avg" + }, + { + "field": "86549551-9bef-11e9-871d-d3c7d4c337ef", + "id": "86549552-9bef-11e9-871d-d3c7d4c337ef", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": "0", + "formatter": "number", + "id": "c66bf580-9be9-11e9-871d-d3c7d4c337ef", + "label": "All", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.networkprogramming.duration.us.bucket.+Inf", + "id": "c66bf581-9be9-11e9-871d-d3c7d4c337ef", + "type": "avg" + }, + { + "field": "c66bf581-9be9-11e9-871d-d3c7d4c337ef", + "id": "d48b3950-9be9-11e9-871d-d3c7d4c337ef", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Proxy network programming [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-b8a24790-9bf0-11e9-9dc8-fd27291d427f", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-ba7bf750-9bf5-11e9-9dc8-fd27291d427f.json b/test/packages/kubernetes/kibana/visualization/kubernetes-ba7bf750-9bf5-11e9-9dc8-fd27291d427f.json new file mode 100644 index 0000000000..c6d14c0128 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-ba7bf750-9bf5-11e9-9dc8-fd27291d427f.json @@ -0,0 +1,245 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.proxy" + } + } + }, + "title": "Proxy sync rules [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(63,112,2,1)", + "fill": 0.5, + "formatter": "number", + "hidden": false, + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "Under 16 ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.rules.duration.us.bucket.16000", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + }, + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "038a94d0-9bf4-11e9-9f03-d58417b2a60d", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(138,223,24,1)", + "fill": "0.4", + "formatter": "number", + "hidden": false, + "id": "250daed0-9bf4-11e9-9f03-d58417b2a60d", + "label": "Under 512 ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.rules.duration.us.bucket.512000", + "id": "250dd5e0-9bf4-11e9-9f03-d58417b2a60d", + "type": "avg" + }, + { + "field": "250dd5e0-9bf4-11e9-9f03-d58417b2a60d", + "id": "250dd5e1-9bf4-11e9-9f03-d58417b2a60d", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(174,161,255,1)", + "fill": "0.3", + "formatter": "number", + "hidden": false, + "id": "744141b0-9bf4-11e9-9f03-d58417b2a60d", + "label": "Under 1024 ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.rules.duration.us.bucket.1024000", + "id": "744141b1-9bf4-11e9-9f03-d58417b2a60d", + "type": "avg" + }, + { + "field": "744141b1-9bf4-11e9-9f03-d58417b2a60d", + "id": "744141b2-9bf4-11e9-9f03-d58417b2a60d", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(115,216,255,1)", + "fill": "0.2", + "formatter": "number", + "hidden": false, + "id": "e4454060-9bf4-11e9-9f03-d58417b2a60d", + "label": "Under 4096 ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.rules.duration.us.bucket.4096000", + "id": "e4454061-9bf4-11e9-9f03-d58417b2a60d", + "type": "avg" + }, + { + "field": "e4454061-9bf4-11e9-9f03-d58417b2a60d", + "id": "e4456770-9bf4-11e9-9f03-d58417b2a60d", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(252,196,0,1)", + "fill": "0.1", + "formatter": "number", + "hidden": false, + "id": "1ac14f80-9bf5-11e9-9f03-d58417b2a60d", + "label": "Under 8192 ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.rules.duration.us.bucket.8192000", + "id": "1ac17690-9bf5-11e9-9f03-d58417b2a60d", + "type": "avg" + }, + { + "field": "1ac17690-9bf5-11e9-9f03-d58417b2a60d", + "id": "1ac17691-9bf5-11e9-9f03-d58417b2a60d", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(209,119,103,1)", + "fill": "0", + "formatter": "number", + "hidden": false, + "id": "3cbf17c0-9bf5-11e9-9f03-d58417b2a60d", + "label": "Under 16384 ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.rules.duration.us.bucket.16384000", + "id": "3cbf17c1-9bf5-11e9-9f03-d58417b2a60d", + "type": "avg" + }, + { + "field": "3cbf17c1-9bf5-11e9-9f03-d58417b2a60d", + "id": "3cbf17c2-9bf5-11e9-9f03-d58417b2a60d", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(104,188,0,1)", + "fill": "0", + "formatter": "number", + "id": "58fb7c80-9bf5-11e9-9f03-d58417b2a60d", + "label": "All", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.sync.rules.duration.us.bucket.+Inf", + "id": "58fb7c81-9bf5-11e9-9f03-d58417b2a60d", + "type": "avg" + }, + { + "field": "58fb7c81-9bf5-11e9-9f03-d58417b2a60d", + "id": "58fb7c82-9bf5-11e9-9f03-d58417b2a60d", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Proxy sync rules [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-ba7bf750-9bf5-11e9-9dc8-fd27291d427f", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-bcb194a0-9bf8-11e9-9dc8-fd27291d427f.json b/test/packages/kubernetes/kibana/visualization/kubernetes-bcb194a0-9bf8-11e9-9dc8-fd27291d427f.json new file mode 100644 index 0000000000..1b5121e504 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-bcb194a0-9bf8-11e9-9dc8-fd27291d427f.json @@ -0,0 +1,76 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "title": "Host selector [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "controls": [ + { + "fieldName": "host.hostname", + "id": "1561982488150", + "indexPatternRefName": "control_0_index_pattern", + "label": "Hostname", + "options": { + "dynamicOptions": true, + "multiselect": true, + "order": "desc", + "size": 5, + "type": "terms" + }, + "parent": "", + "type": "list" + }, + { + "fieldName": "service.address", + "id": "1561982723711", + "indexPatternRefName": "control_1_index_pattern", + "label": "Service address", + "options": { + "dynamicOptions": true, + "multiselect": false, + "order": "desc", + "size": 5, + "type": "terms" + }, + "parent": "", + "type": "list" + } + ], + "pinFilters": false, + "updateFiltersOnChange": false, + "useTimeFilter": false + }, + "title": "Host selector [Metrics Kubernetes]", + "type": "input_control_vis" + } + }, + "id": "kubernetes-bcb194a0-9bf8-11e9-9dc8-fd27291d427f", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [ + { + "id": "metrics-*", + "name": "control_0_index_pattern", + "type": "index-pattern" + }, + { + "id": "metrics-*", + "name": "control_1_index_pattern", + "type": "index-pattern" + } + ], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-cac89fb0-9906-11e9-ba57-b7ab4e2d4b58.json b/test/packages/kubernetes/kibana/visualization/kubernetes-cac89fb0-9906-11e9-ba57-b7ab4e2d4b58.json new file mode 100644 index 0000000000..b4169611ab --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-cac89fb0-9906-11e9-ba57-b7ab4e2d4b58.json @@ -0,0 +1,87 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.proxy" + } + } + }, + "title": "Proxy memory [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(251,158,0,1)", + "fill": 0.5, + "formatter": "bytes", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "Resident", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.process.memory.resident.bytes", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(164,221,0,1)", + "fill": 0.5, + "formatter": "bytes", + "id": "1ea6cee0-9907-11e9-b4e1-6dc893538542", + "label": "Virtual", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.process.memory.virtual.bytes", + "id": "1ea6cee1-9907-11e9-b4e1-6dc893538542", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Proxy memory [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-cac89fb0-9906-11e9-ba57-b7ab4e2d4b58", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-cd059410-2bfb-11e7-859b-f78b612cde28.json b/test/packages/kubernetes/kibana/visualization/kubernetes-cd059410-2bfb-11e7-859b-f78b612cde28.json new file mode 100644 index 0000000000..637e7df037 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-cd059410-2bfb-11e7-859b-f78b612cde28.json @@ -0,0 +1,86 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.state_deployment" + } + } + }, + "title": "Deployments [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "listeners": {}, + "params": { + "axis_formatter": "number", + "axis_position": "left", + "background_color_rules": [ + { + "id": "67ee7da0-30e0-11e7-8df8-6d3604a72912" + } + ], + "bar_color_rules": [ + { + "id": "68cdba10-30e0-11e7-8df8-6d3604a72912" + } + ], + "filter": { + "language": "lucene", + "query": "data_stream.dataset:kubernetes.state_deployment" + }, + "gauge_color_rules": [ + { + "id": "69765620-30e0-11e7-8df8-6d3604a72912" + } + ], + "gauge_inner_width": 10, + "gauge_style": "half", + "gauge_width": 10, + "id": "4c4690b0-30e0-11e7-8df8-6d3604a72912", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "4c4690b1-30e0-11e7-8df8-6d3604a72912", + "label": "Deployments", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.deployment.name", + "id": "4c4690b2-30e0-11e7-8df8-6d3604a72912", + "type": "cardinality" + } + ], + "point_size": 1, + "seperate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "terms_field": "kubernetes.deployment.name" + } + ], + "show_legend": 1, + "time_field": "@timestamp", + "type": "metric" + }, + "title": "Deployments [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-cd059410-2bfb-11e7-859b-f78b612cde28", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-d6564360-2bfc-11e7-859b-f78b612cde28.json b/test/packages/kubernetes/kibana/visualization/kubernetes-d6564360-2bfc-11e7-859b-f78b612cde28.json new file mode 100644 index 0000000000..e848f5227d --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-d6564360-2bfc-11e7-859b-f78b612cde28.json @@ -0,0 +1,135 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "title": "Memory usage by node [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "background_color_rules": [ + { + "id": "12c1f2f0-2bf2-11e7-859b-f78b612cde28" + } + ], + "bar_color_rules": [ + { + "id": "1373ddd0-2bf2-11e7-859b-f78b612cde28" + } + ], + "filter": { + "language": "lucene", + "query": "(data_stream.dataset:kubernetes.container OR data_stream.dataset:kubernetes.state_node)" + }, + "gauge_color_rules": [ + { + "id": "140e4910-2bf2-11e7-859b-f78b612cde28" + } + ], + "gauge_inner_width": 10, + "gauge_style": "half", + "gauge_width": 10, + "id": "0d5c6b10-2bf2-11e7-859b-f78b612cde28", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "bytes", + "id": "0d5c9220-2bf2-11e7-859b-f78b612cde28", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.container.memory.usage.bytes", + "id": "0d5c9221-2bf2-11e7-859b-f78b612cde28", + "type": "sum" + }, + { + "field": "0d5c9221-2bf2-11e7-859b-f78b612cde28", + "id": "9f0cf900-1ffb-11e8-81f2-43be86397500", + "type": "cumulative_sum" + }, + { + "field": "9f0cf900-1ffb-11e8-81f2-43be86397500", + "id": "a926e130-1ffb-11e8-81f2-43be86397500", + "type": "derivative", + "unit": "10s" + } + ], + "point_size": 1, + "seperate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "terms", + "stacked": "none", + "terms_field": "kubernetes.node.name", + "terms_order_by": "0d5c9221-2bf2-11e7-859b-f78b612cde28", + "terms_size": "10000" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(211,49,21,1)", + "fill": "0", + "formatter": "bytes", + "hide_in_legend": 1, + "id": "8ba3b270-31a7-11e7-84cc-096d2b38e6e5", + "label": "Node capacity", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.node.memory.capacity.bytes", + "id": "8ba3b271-31a7-11e7-84cc-096d2b38e6e5", + "type": "sum" + }, + { + "field": "8ba3b271-31a7-11e7-84cc-096d2b38e6e5", + "id": "d1fb2670-1ffb-11e8-81f2-43be86397500", + "type": "cumulative_sum" + }, + { + "field": "d1fb2670-1ffb-11e8-81f2-43be86397500", + "id": "dc8b01f0-1ffb-11e8-81f2-43be86397500", + "type": "derivative", + "unit": "10s" + } + ], + "point_size": "0", + "seperate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "terms", + "stacked": "none", + "terms_field": "kubernetes.node.name", + "terms_order_by": "8ba3b271-31a7-11e7-84cc-096d2b38e6e5", + "terms_size": "10000" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Memory usage by node [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-d6564360-2bfc-11e7-859b-f78b612cde28", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-d86b2da0-9c20-11e9-9dc8-fd27291d427f.json b/test/packages/kubernetes/kibana/visualization/kubernetes-d86b2da0-9c20-11e9-9dc8-fd27291d427f.json new file mode 100644 index 0000000000..08ec8662aa --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-d86b2da0-9c20-11e9-9dc8-fd27291d427f.json @@ -0,0 +1,109 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.controllermanager" + } + } + }, + "title": "Controller HTTP request duration [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(165,228,85,1)", + "fill": 0.5, + "formatter": "us,ms,2", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "P99", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.http.request.duration.us.percentile.99", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "us,ms,2", + "id": "24fb4960-9c22-11e9-b29f-d55be9348723", + "label": "P90", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.http.request.duration.us.percentile.90", + "id": "24fb4961-9c22-11e9-b29f-d55be9348723", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(65,117,0,1)", + "fill": 0.5, + "formatter": "us,ms,2", + "id": "2e397790-9c22-11e9-b29f-d55be9348723", + "label": "P50", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.controllermanager.http.request.duration.us.percentile.50", + "id": "2e397791-9c22-11e9-b29f-d55be9348723", + "type": "avg" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Controller HTTP request duration [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-d86b2da0-9c20-11e9-9dc8-fd27291d427f", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-d9fc1b80-9c9c-11e9-94fd-c91206cd5249.json b/test/packages/kubernetes/kibana/visualization/kubernetes-d9fc1b80-9c9c-11e9-94fd-c91206cd5249.json new file mode 100644 index 0000000000..009d830ae7 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-d9fc1b80-9c9c-11e9-94fd-c91206cd5249.json @@ -0,0 +1,248 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.scheduler" + } + } + }, + "title": "Scheduler end to end scheduling duration [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_min": "0", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(63,112,2,1)", + "fill": 0.5, + "formatter": "number", + "hidden": false, + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "Under 16ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.scheduling.e2e.duration.us.bucket.16000", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + }, + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "2a21d690-9c9d-11e9-8174-4589c2c40897", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(138,223,24,1)", + "fill": "0.4", + "formatter": "number", + "hidden": false, + "id": "a0344a80-9ca1-11e9-9b2f-4dae0cc2bdf1", + "label": "Under 512ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.scheduling.e2e.duration.us.bucket.512000", + "id": "a0347190-9ca1-11e9-9b2f-4dae0cc2bdf1", + "type": "avg" + }, + { + "field": "a0347190-9ca1-11e9-9b2f-4dae0cc2bdf1", + "id": "9a865d10-9ca3-11e9-9b2f-4dae0cc2bdf1", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(174,161,255,1)", + "fill": "0.3", + "formatter": "number", + "hidden": false, + "id": "c456c780-9ca1-11e9-9b2f-4dae0cc2bdf1", + "label": "Under 1024ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.scheduling.e2e.duration.us.bucket.1024000", + "id": "c456c781-9ca1-11e9-9b2f-4dae0cc2bdf1", + "type": "avg" + }, + { + "field": "c456c781-9ca1-11e9-9b2f-4dae0cc2bdf1", + "id": "a77cd850-9ca3-11e9-9b2f-4dae0cc2bdf1", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(115,216,255,1)", + "fill": "0.2", + "formatter": "number", + "hidden": false, + "id": "127a9810-9ca2-11e9-9b2f-4dae0cc2bdf1", + "label": "Under 4096ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.scheduling.e2e.duration.us.bucket.4096000", + "id": "127a9811-9ca2-11e9-9b2f-4dae0cc2bdf1", + "type": "avg" + }, + { + "field": "127a9811-9ca2-11e9-9b2f-4dae0cc2bdf1", + "id": "b48a0fe0-9ca3-11e9-9b2f-4dae0cc2bdf1", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(252,196,0,1)", + "fill": "0.1", + "formatter": "number", + "hidden": false, + "id": "6a2ba130-9ca2-11e9-9b2f-4dae0cc2bdf1", + "label": "Under 8192ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.scheduling.e2e.duration.us.bucket.8192000", + "id": "6a2ba131-9ca2-11e9-9b2f-4dae0cc2bdf1", + "type": "avg" + }, + { + "field": "6a2ba131-9ca2-11e9-9b2f-4dae0cc2bdf1", + "id": "c727fe00-9ca3-11e9-9b2f-4dae0cc2bdf1", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(209,119,103,1)", + "fill": "0", + "formatter": "number", + "hidden": false, + "id": "f93a66e0-9ca2-11e9-9b2f-4dae0cc2bdf1", + "label": "Under 16384ms", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.scheduling.e2e.duration.us.bucket.16384000", + "id": "f93a66e1-9ca2-11e9-9b2f-4dae0cc2bdf1", + "type": "avg" + }, + { + "field": "f93a66e1-9ca2-11e9-9b2f-4dae0cc2bdf1", + "id": "ce42d700-9ca3-11e9-9b2f-4dae0cc2bdf1", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(104,188,0,1)", + "fill": "0", + "formatter": "number", + "hidden": false, + "id": "310452c0-9ca3-11e9-9b2f-4dae0cc2bdf1", + "label": "All", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.scheduler.scheduling.e2e.duration.us.bucket.+Inf", + "id": "310452c1-9ca3-11e9-9b2f-4dae0cc2bdf1", + "type": "avg" + }, + { + "field": "310452c1-9ca3-11e9-9b2f-4dae0cc2bdf1", + "id": "e0692420-9ca3-11e9-9b2f-4dae0cc2bdf1", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "timeseries" + }, + "title": "Scheduler end to end scheduling duration [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-d9fc1b80-9c9c-11e9-94fd-c91206cd5249", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-da1ff7c0-30ed-11e7-b9e5-2b5b07213ab3.json b/test/packages/kubernetes/kibana/visualization/kubernetes-da1ff7c0-30ed-11e7-b9e5-2b5b07213ab3.json new file mode 100644 index 0000000000..b4a0a9fb89 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-da1ff7c0-30ed-11e7-b9e5-2b5b07213ab3.json @@ -0,0 +1,90 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.state_deployment" + } + } + }, + "title": "Available pods [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "listeners": {}, + "params": { + "axis_formatter": "number", + "axis_position": "left", + "background_color_rules": [ + { + "id": "508ffb30-30d5-11e7-8df8-6d3604a72912" + } + ], + "bar_color_rules": [ + { + "id": "674d83b0-30d5-11e7-8df8-6d3604a72912" + } + ], + "filter": { + "language": "lucene", + "query": "data_stream.dataset:kubernetes.state_deployment" + }, + "gauge_color_rules": [ + { + "id": "50f9b980-30d5-11e7-8df8-6d3604a72912" + } + ], + "gauge_inner_width": "10", + "gauge_max": "5", + "gauge_style": "half", + "gauge_width": "10", + "id": "2fe9d3b0-30d5-11e7-8df8-6d3604a72912", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "2fe9d3b1-30d5-11e7-8df8-6d3604a72912", + "label": "Available Pods", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.deployment.replicas.available", + "id": "54cf79a0-30d5-11e7-8df8-6d3604a72912", + "type": "sum" + } + ], + "override_index_pattern": 1, + "point_size": 1, + "seperate_axis": 0, + "series_index_pattern": "*", + "series_interval": "10s", + "series_time_field": "@timestamp", + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_legend": 1, + "time_field": "@timestamp", + "type": "metric" + }, + "title": "Available pods [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-da1ff7c0-30ed-11e7-b9e5-2b5b07213ab3", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-e0ddd3e0-98fe-11e9-ba57-b7ab4e2d4b58.json b/test/packages/kubernetes/kibana/visualization/kubernetes-e0ddd3e0-98fe-11e9-ba57-b7ab4e2d4b58.json new file mode 100644 index 0000000000..1c78b00a1b --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-e0ddd3e0-98fe-11e9-ba57-b7ab4e2d4b58.json @@ -0,0 +1,74 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.proxy" + } + } + }, + "title": "Proxy CPU [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_min": "0", + "axis_position": "left", + "axis_scale": "normal", + "default_index_pattern": "metrics-*", + "default_timefield": "@timestamp", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "CPU time", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.proxy.process.cpu.sec", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + }, + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "35da60d0-98fe-11e9-b4e1-6dc893538542", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "value_template": "{{value}} s" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "Proxy CPU [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-e0ddd3e0-98fe-11e9-ba57-b7ab4e2d4b58", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-e1018b90-2bfb-11e7-859b-f78b612cde28.json b/test/packages/kubernetes/kibana/visualization/kubernetes-e1018b90-2bfb-11e7-859b-f78b612cde28.json new file mode 100644 index 0000000000..2c3b130132 --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-e1018b90-2bfb-11e7-859b-f78b612cde28.json @@ -0,0 +1,89 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "data_stream.dataset: kubernetes.state_deployment" + } + } + }, + "title": "Desired pods [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "listeners": {}, + "params": { + "axis_formatter": "number", + "axis_position": "left", + "background_color_rules": [ + { + "id": "508ffb30-30d5-11e7-8df8-6d3604a72912" + } + ], + "bar_color_rules": [ + { + "id": "674d83b0-30d5-11e7-8df8-6d3604a72912" + } + ], + "filter": { + "language": "lucene", + "query": "data_stream.dataset:kubernetes.state_deployment" + }, + "gauge_color_rules": [ + { + "id": "50f9b980-30d5-11e7-8df8-6d3604a72912" + } + ], + "gauge_inner_width": "10", + "gauge_max": "5", + "gauge_style": "half", + "gauge_width": "10", + "id": "2fe9d3b0-30d5-11e7-8df8-6d3604a72912", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": 0.5, + "formatter": "number", + "id": "2fe9d3b1-30d5-11e7-8df8-6d3604a72912", + "label": "Desired Pods", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.deployment.replicas.desired", + "id": "54cf79a0-30d5-11e7-8df8-6d3604a72912", + "type": "sum" + } + ], + "override_index_pattern": 1, + "point_size": 1, + "seperate_axis": 0, + "series_interval": "10s", + "series_time_field": "@timestamp", + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_legend": 1, + "time_field": "@timestamp", + "type": "metric" + }, + "title": "Desired pods [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-e1018b90-2bfb-11e7-859b-f78b612cde28", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/kibana/visualization/kubernetes-ec360ff0-57a0-11e8-afa2-e9067ea62228.json b/test/packages/kubernetes/kibana/visualization/kubernetes-ec360ff0-57a0-11e8-afa2-e9067ea62228.json new file mode 100644 index 0000000000..45914f754e --- /dev/null +++ b/test/packages/kubernetes/kibana/visualization/kubernetes-ec360ff0-57a0-11e8-afa2-e9067ea62228.json @@ -0,0 +1,112 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": {} + }, + "title": "API Server Requests [Metrics Kubernetes]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "auto", + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(159,5,0,1)", + "fill": "0", + "filter": { + "language": "lucene", + "query": "NOT (kubernetes.apiserver.request.verb: WATCH or kubernetes.apiserver.request.verb: CONNECT)" + }, + "formatter": "us,ms,2", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "Avg response time", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.apiserver.request.latency.sum", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "sum" + }, + { + "field": "kubernetes.apiserver.request.count", + "id": "a2185e50-57a0-11e8-af57-a1d645d2b569", + "type": "max" + }, + { + "id": "b09133d0-57a0-11e8-af57-a1d645d2b569", + "script": "params.sum / params.count", + "type": "calculation", + "variables": [ + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "b27c8910-57a0-11e8-af57-a1d645d2b569", + "name": "sum" + }, + { + "field": "a2185e50-57a0-11e8-af57-a1d645d2b569", + "id": "b5fc8810-57a0-11e8-af57-a1d645d2b569", + "name": "count" + } + ] + } + ], + "point_size": 1, + "seperate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "value_template": "{{value}} ms" + }, + { + "axis_position": "right", + "chart_type": "line", + "color": "rgba(22,165,165,1)", + "fill": 0.5, + "formatter": "number", + "id": "c0019340-57a1-11e8-a049-ff54cef064a2", + "label": "Requests rate", + "line_width": 1, + "metrics": [ + { + "field": "kubernetes.apiserver.request.count", + "id": "c001ba50-57a1-11e8-a049-ff54cef064a2", + "type": "max" + }, + { + "field": "c001ba50-57a1-11e8-a049-ff54cef064a2", + "id": "dc83b390-57a1-11e8-a049-ff54cef064a2", + "type": "derivative", + "unit": "" + } + ], + "point_size": 1, + "seperate_axis": 1, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "type": "timeseries" + }, + "title": "API Server Requests [Metrics Kubernetes]", + "type": "metrics" + } + }, + "id": "kubernetes-ec360ff0-57a0-11e8-afa2-e9067ea62228", + "migrationVersion": { + "visualization": "7.8.0" + }, + "references": [], + "type": "visualization" +} \ No newline at end of file diff --git a/test/packages/kubernetes/manifest.yml b/test/packages/kubernetes/manifest.yml index 536aff2af0..64325fdadd 100644 --- a/test/packages/kubernetes/manifest.yml +++ b/test/packages/kubernetes/manifest.yml @@ -3,21 +3,181 @@ name: kubernetes title: Kubernetes version: 999.999.999 license: basic -description: Kubernetes Integration +description: This Elastic integration collects metrics from Kubernetes clusters type: integration categories: - containers - kubernetes release: experimental conditions: - kibana.version: '^7.9.0' + kibana.version: "^7.15.0" +screenshots: + - src: /img/metricbeat_kubernetes_overview.png + title: Metricbeat Kubernetes Overview + size: 1896x961 + type: image/png +icons: + - src: /img/logo_kubernetes.svg + title: Logo Kubernetes + size: 32x32 + type: image/svg+xml policy_templates: - - name: kubernetes - title: Kubernetes metrics - description: Collect metrics from Kubernetes + - name: kubelet + title: Kubelet + description: Collect metrics from Kubernetes Kubelet API + data_streams: + - container + - node + - pod + - system + - volume inputs: - type: kubernetes/metrics - title: Collect Kubernetes metrics - description: Collecting API Server, Controller Manager, Scheduler, Proxy and Event metrics as well as Node, Pod, Container, Volume and System metrics from Kubelet and metrics from kube_state_metrics (container, cronjob, deployment, daemonset, node, persistentvolume, persistentvolumeclaim, pod, replicaset, resourcequota, service, statefulset, storageclass). + title: Collect Kubernetes metrics from Kubelet API + description: Collecting Node, Pod, Container, Volume and System metrics from Kubelet + input_group: metrics + icons: + - src: /img/logo_kubernetes.svg + title: Logo Kubernetes + size: 32x32 + type: image/svg+xml + screenshots: + - src: /img/metricbeat_kubernetes_overview.png + title: Metricbeat Kubernetes Overview + size: 1896x961 + type: image/png + - name: kube-state-metrics + title: kube-state-metrics + description: Collect metrics from kube-state-metrics + data_streams: + - state_container + - state_cronjob + - state_daemonset + - state_deployment + - state_job + - state_node + - state_persistentvolume + - state_persistentvolumeclaim + - state_pod + - state_replicaset + - state_resourcequota + - state_service + - state_statefulset + - state_storageclass + inputs: + - type: kubernetes/metrics + title: Collect Kubernetes metrics from kube-state-metrics + description: Collecting metrics from kube-state-metrics (container, cronjob, deployment, daemonset, node, persistentvolume, persistentvolumeclaim, pod, replicaset, resourcequota, service, statefulset, storageclass) + input_group: metrics + icons: + - src: /img/logo_kubernetes.svg + title: Logo Kubernetes + size: 32x32 + type: image/svg+xml + screenshots: + - src: /img/metricbeat_kubernetes_overview.png + title: Metricbeat Kubernetes Overview + size: 1896x961 + type: image/png + - name: kube-apiserver + title: kube-apiserver + description: Collect metrics from Kubernetes API Server + data_streams: + - apiserver + inputs: + - type: kubernetes/metrics + title: Collect Kubernetes metrics from Kubernetes API Server + description: Collecting metrics from Kubernetes API Server + input_group: metrics + icons: + - src: /img/logo_kubernetes.svg + title: Logo Kubernetes + size: 32x32 + type: image/svg+xml + screenshots: + - src: /img/metricbeat_kubernetes_overview.png + title: Metricbeat Kubernetes Overview + size: 1896x961 + type: image/png + - name: kube-proxy + title: kube-proxy + description: Collect metrics from Kubernetes Proxy + data_streams: + - proxy + inputs: + - type: kubernetes/metrics + title: Collect Kubernetes metrics from Kubernetes Proxy + description: Collecting metrics from Kubernetes Proxy + input_group: metrics + icons: + - src: /img/logo_kubernetes.svg + title: Logo Kubernetes + size: 32x32 + type: image/svg+xml + screenshots: + - src: /img/metricbeat-kubernetes-proxy.png + title: Metricbeat Kubernetes Proxy + size: 1854x920 + type: image/png + - name: kube-scheduler + title: kube-scheduler + description: Collect metrics from Kubernetes Scheduler + data_streams: + - scheduler + inputs: + - type: kubernetes/metrics + title: Collect Kubernetes metrics from Kubernetes Scheduler + description: Collecting metrics from Kubernetes Scheduler + input_group: metrics + icons: + - src: /img/logo_kubernetes.svg + title: Logo Kubernetes + size: 32x32 + type: image/svg+xml + screenshots: + - src: /img/metricbeat_kubernetes_scheduler.png + title: Metricbeat Kubernetes Scheduler + size: 1856x897 + type: image/png + - name: kube-controller-manager + title: kube-controller-manager + description: Collect metrics from Kubernetes controller-manager + data_streams: + - controllermanager + inputs: + - type: kubernetes/metrics + title: Collect Kubernetes metrics from Kubernetes controller-manager + description: Collecting metrics from Kubernetes controller-manager + input_group: metrics + icons: + - src: /img/logo_kubernetes.svg + title: Logo Kubernetes + size: 32x32 + type: image/svg+xml + screenshots: + - src: /img/metricbeat-kubernetes-controllermanager.png + title: Metricbeat Kubernetes Controller Manager + size: 1896x961 + type: image/png + - name: events + title: Events + description: Collect events from Kubernetes API server + data_streams: + - event + inputs: + - type: kubernetes/metrics + title: Collect Kubernetes events from Kubernetes API Server + description: Collecting events from Kubernetes API Server + input_group: metrics + icons: + - src: /img/logo_kubernetes.svg + title: Logo Kubernetes + size: 32x32 + type: image/svg+xml + screenshots: + - src: /img/metricbeat_kubernetes_overview.png + title: Metricbeat Kubernetes Overview + size: 1896x961 + type: image/png owner: - github: elastic/integrations-platforms + github: elastic/integrations