Add k8s cluster identifier

[ChrsMark]

What does this PR do?

This PR add cluster identifier fields (defined in ECS) as part of k8s metadata in:

• event's enrichment with autodiscovery
• event's enrichment in kubernetes module (where already happens)
• event's enrichment in add_kubernetes_metadata processor

Note: [MetaGenerators' refactoring ] The identifiers are stored under orchestrator.cluster.url/name and because of this the metadata generators are refactored a little bit so as to cover the addition of such fields that are out of kubernetes.* namespace. The change is transparent and kubernetes.* metadata are still reported in the same way. The refactoring is about making it easier to handle in the future ECS fields populated by k8s metadata generators. The logic is covered in interfaces' docs.

The transparency of the refactoring is ensured by Event's testing in tests below:

1. https://github.com/elastic/beats/pull/26056/files#diff-40f67a7afe61b20e5336ebd9ad20c40c717794f0821ba9234301a7e4544315efR114
2. https://github.com/elastic/beats/pull/26056/files#diff-9f24ba274864b933a726842c2b9fb8ee487a13545579938f9e373d0463611142R335
3. https://github.com/elastic/beats/pull/26056/files#diff-4abca4d17de2c5a0ed69171309dc21636be3ca778d0d34943e7747f2a7c6dbc1R235

The fields are populated following the flow bellow:

1. Try to get this info from kube_config if provided
2. Else (when inCluster mode) try to get the info from kubeadm-config configMap (if available). Only for clusters setup with kubeadm.
3. Else try to get the info from cloud’s meta api (only on GKE)
4. Else these fields will not be populated (see [Metricbeat] Add cluster identifier to Kubernetes metadata #17467 (comment))

Why is it important?

To add cluster identifier ECS fields as part of k8s metadata.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• update docs in metadata generators
• update cluster roles
• manual testing

How to test this PR locally

A. Verify that events from state_* metricsets are enriched properly

Enable kubernetes module with the following datasets:

- module: kubernetes
  metricsets:
    - state_node
    - state_deployment
    - state_pod
    - state_container
    - state_service
  period: 10s
  hosts: ["0.0.0.0:8081"]
  add_metadata: true
  kube_config: /Users/chrismark/.kube/config

Note: In the example above I run kube-state-metrics on local cluster using kind and I expose it to my host machine using kubectl -n kube-system port-forward svc/kube-state-metrics 8081:8080. In this case I need to define add_metadata as true and also provide the proper kube_config so as to reach the k8s API. You can try kubectl config view -o jsonpath='{"Cluster name\tServer\n"}{range .clusters[*]}{.name}{"\t"}{.cluster.server}{"\n"}{end}' to verify the values.
2. Ensure that orchestrator.cluster.name, orchestrator.cluster.name, kubernetes.namespace and kubernetes.node.name are being populated properly.
3. Perform same test while running with inCluster mode, running metricbeat as Pod in the cluster (Note that the k8s cluster should be create with kubeadm since values for cluster info are retrieved from kubeadm-config configmap, you can try kubectl -n kube-system get configmap kubeadm-config -o yaml to verify it)

B. Verify that events from add_kuberentes_metadata are enriched properly

0. Use updated manifests from https://github.com/elastic/beats/tree/master/deploy/kubernetes
1. Deploy Filebeat on kubernetes (cluster should be created with kubadm [ie a kind cluster] ) and configure log collection like this:

    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

2. Ensure that orchestrator.cluster.name, orchestrator.cluster.name, kubernetes.namespace and kubernetes.node.name are being populated properly.

C. Verify that events from autodiscover provider are enriched properly

0. Use updated manifests from https://github.com/elastic/beats/tree/master/deploy/kubernetes
1. Deploy Filebeat on kubernetes (cluster should be created with kubadm [ie a kind cluster] ) and configure log collection like this:

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

2. Ensure that orchestrator.cluster.name, orchestrator.cluster.name, kubernetes.namespace and kubernetes.node.name are being populated properly.

D. Perform one of the above scenarios with Metricbeat running as Pod on GKE.

Related issues

• Closes [Metricbeat] Add cluster identifier to Kubernetes metadata #17467

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #26056 updated

• Start Time: 2021-06-16T11:40:22.265+0000

• Duration: 140 min 3 sec

• Commit: e720e74

Test stats 🧪

Test	Results
Failed	0
Passed	46977
Skipped	5132
Total	52109

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	46977
Skipped	5132
Total	52109

[jsoriano]

The .url will be set with 172.18.0.2:6443. Note that when we have multiple endpoints in HA cluster we use the first one of the list.

@ChrsMark take into account that this value (or the cluster IP itself) doesn't need to be unique in an organization with multiple clusters. An organization will probably use the same CIDRs for the internal networks of their clusters (at least there is no reason to use different ones). Also take into account that the Endpoints are the addresses of the pods that are providing the API server service, so they can change for the same cluster if there are multiple of them and/or if they are replaced or upgraded.

As discussed in #17467 (comment), I think the values for the cluster URL should be unique. Look at this more as the external address that clients use to connect.

Kubernetes doesn't have these concepts of "url" or "name" natively, some strategies discussed in the original issue to collect this data were to rely on cluster-info or the cloud provider metadata. In some cases this information may not be available, and I think this is ok, I would prefer to leave this empty in these cases, and users may chose to use fields to manually fill this info.

[ChrsMark]

👍🏼 Thanks Jaime for the comments. Indeed using endpoints will not be representative.
In this I changed the approach as described in the updated PR's descirption.

To cut it short here is the logic:

1. use kube_config if provide (equivalent to kubectl config view -o jsonpath='{"Cluster name\tServer\n"}{range .clusters[*]}{.name}{"\t"}{.cluster.server}{"\n"}{end}')
2. use kubeadm-info condifgmap if cluster is built with kubeadm
3. use GKE meta API.

For 2 we use values from clusterName and controlPlaneEndpoint. A sample configmap looks like the one bellow:

apiVersion: v1
data:
  ClusterConfiguration: |
    apiServer:
      extraArgs:
        authorization-mode: Node,RBAC
        cloud-provider: aws
        service-node-port-range: 80-50000
      timeoutForControlPlane: 4m0s
    apiVersion: kubeadm.k8s.io/v1beta2
    certificatesDir: /etc/kubernetes/pki
    clusterName: kubernetes
    controlPlaneEndpoint: 34.255.80.223:6443

For 3 we use /kubeconfig metadata endpoint to get kube_config and then from it to extract the server url:

apiVersion: v1
kind: Config
clusters:
- cluster:
    server: https://35.223.150.134
    certificate-authority: '/etc/srv/kubernetes/pki/ca-certificates.crt'
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    namespace: default
    user: exec-plugin-auth
  name: default-context
current-context: default-context
users:
- name: exec-plugin-auth
  user:
    exec:
      apiVersion: "client.authentication.k8s.io/v1alpha1"
      command: '/home/kubernetes/bin/gke-exec-auth-plugin'
      args: ["--cache-dir", '/var/lib/kubelet/pki/']

I think this is a best effort approach to get these non-native k8s metadata using actually external resources. Let me know what you think @jsoriano .

Note: Once we verify the approach I will complete the leftover changes and open it for complete review.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[jsoriano]

Ok, from my side this can be merged, but please continue afterwards with the refactor of add_cloud_metadata, or create an issue so we do it soon.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b add_k8s_cluster_identifier upstream/add_k8s_cluster_identifier
git merge upstream/master
git push upstream add_k8s_cluster_identifier