Fixes conditions of kubernetes data_streams that are based on k8s labels & pipelines #2041

ChrsMark · 2021-10-27T13:21:20Z

What does this PR do?

Fixes conditions of k8s data_streans that are based on k8s labels

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

elasticmachine · 2021-10-27T13:21:38Z

Pinging @elastic/integrations (Team:Integrations)

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

mtojek

nit: please rename the PR to something more meaningful :)

elasticmachine · 2021-10-27T14:13:26Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-11-04T12:57:03.113+0000
Duration: 40 min 28 sec
Commit: 167ab72

Test stats 🧪

Test	Results
Failed	0
Passed	114
Skipped	0
Total	114

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

ChrsMark · 2021-10-28T21:39:47Z

/test

ChrsMark · 2021-11-01T08:02:45Z

/test

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark · 2021-11-01T12:45:57Z

I wonder why the condition does not work...I'm able to see the new condition working with the following:

providers.kubernetes:
  scope: node
  kube_config: /Users/chrismark/.kube/config
  node: "kind-control-plane"
  resources:
    pod:
      enabled: true

inputs:
  - name: redis
    type: redis/metrics
    use_output: default
    meta:
      package:
        name: redis
        version: 0.3.6
    data_stream:
      namespace: default
    streams:
      - data_stream:
          dataset: kubernetes.scheduler
          type: metrics
        metricsets:
          - scheduler
        hosts:
          - '${kubernetes.pod.ip}:10251'
        period: 10s
        condition: ${kubernetes.labels.component} == 'kube-scheduler'

and ./elastic-agent -c ./elastic-agent.yml inspect output -o default producing the proper output.

I will try with running the tests locally.

ChrsMark · 2021-11-02T10:09:09Z

@mtojek with what version of stack the tests run? This patch would require > 7.16, so maybe this is the reason for the CI failure.

mtojek · 2021-11-02T10:31:21Z

It runs with 8.0.0-SNAPSHOT.

There are no hits in:

You can also check internal filebeat and metricbeat logs (in the GCP bucket).

ChrsMark · 2021-11-02T11:46:31Z

@mtojek I think that the issue is that elastic-agent Pod is not deployed using the proper version, and even if the stack is 7.16 or 8.0 it will use 7.15.1 because at https://github.com/elastic/elastic-package/blob/2c331472bb62cc53a12c9ba3ec695dba28e1cc61/internal/testrunner/runners/system/servicedeployer/kubernetes.go#L170 it seems to use the DefaultStackVersion which is 7.15.1.

Changing this on my local elastic-package source code to use 7.16.0-SNAPSHOT and it works. Wondering what is the proper way to handle it though. Shouldn't this runner honor the value of --version flag that is set when bringing up the stack?

mtojek · 2021-11-02T15:03:58Z

Oh, this is very unfortunate. This is because the system test runner isn't aware of the stack version and uses a default one. I will open an issue in elastic-package and will try to address it tomorrow.

mtojek · 2021-11-03T11:42:36Z

New release of elastic-package is ready. I triggered the Dependabot job to pull this update into Integrations.

ChrsMark · 2021-11-03T11:44:48Z

Thank you @mtojek . Feel free to kick the tests here too when update is available.

ChrsMark · 2021-11-04T08:55:45Z

Now we hit another error related to kubernetes.namespace.name field -> https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-2041/11/tests/

MichaelKatsoulis · 2021-11-04T09:20:41Z

Now we hit another error related to kubernetes.namespace.name field -> https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-2041/11/tests/

I think this is because in those failing data_streams in sample_event.json there is still kubernetes.namespace and because of that the pipeline tries to run in order to rename them to kubernetes.namespace.name. But as elastic package now uses agent 8.0 the field is already there. We need to update the sample_event.json files.

ChrsMark · 2021-11-04T11:03:26Z

I can reproduce it locally:

2021/11/04 11:02:47 DEBUG deleting data in data stream...
--- Test results for package: kubernetes - START ---
FAILURE DETAILS:
kubernetes/state_pod default:
[0] found error.message in event: field [kubernetes.namespace.name] already exists

ChrsMark · 2021-11-04T11:36:56Z

Events are stored properly in ES but there is an error attached:

{
  "_index": ".ds-metrics-kubernetes.pod-ep-2021.11.04-000001",
  "_id": "3J-76nwBxrC4rvPvLDkA",
  "_version": 1,
  "_score": 1,
  "_source": {
    "kubernetes": {
      "node": {
        "name": "kind-control-plane"
      },
      "pod": {
        "start_time": "2021-11-04T08:31:24Z",
        "uid": "c052b569-c772-43e8-89cc-149ed6f5c69a",
        "memory": {
          "rss": {
            "bytes": 19312640
          },
          "major_page_faults": 341,
          "usage": {
            "bytes": 61587456
          },
          "available": {
            "bytes": 0
          },
          "page_faults": 10919,
          "working_set": {
            "bytes": 20094976
          }
        },
        "ip": "172.18.0.2",
        "name": "kube-scheduler-kind-control-plane",
        "cpu": {
          "usage": {
            "nanocores": 4316814
          }
        },
        "network": {
          "tx": {
            "bytes": 0,
            "errors": 0
          },
          "rx": {
            "bytes": 0,
            "errors": 0
          }
        }
      },
      "namespace": {
        "name": "kube-system"
      },
      "labels": {
        "component": "kube-scheduler",
        "tier": "control-plane"
      }
    },
    "agent": {
      "name": "kind-control-plane",
      "id": "a71f0ee1-21c4-4023-9f79-259253a5e952",
      "type": "metricbeat",
      "ephemeral_id": "3e620d64-ad21-475e-a33b-2362427bcdfe",
      "version": "8.0.0"
    },
    "elastic_agent": {
      "id": "a71f0ee1-21c4-4023-9f79-259253a5e952",
      "version": "8.0.0",
      "snapshot": true
    },
    "error": {
      "message": "field [kubernetes.namespace.name] already exists"
    },
    "orchestrator": {
      "cluster": {
        "name": "kind",
        "url": "kind-control-plane:6443"
      }
    },
    "@timestamp": "2021-11-04T11:35:05.941Z",
    "ecs": {
      "version": "1.12.0"
    },
    "service": {
      "address": "https://kind-control-plane:10250/stats/summary",
      "type": "kubernetes"
    },
    "data_stream": {
      "namespace": "ep",
      "type": "metrics",
      "dataset": "kubernetes.pod"
    },
    "host": {
      "hostname": "kind-control-plane",
      "os": {
        "kernel": "4.9.184-linuxkit",
        "codename": "Core",
        "name": "CentOS Linux",
        "family": "redhat",
        "type": "linux",
        "version": "7 (Core)",
        "platform": "centos"
      },
      "containerized": true,
      "ip": [
        "10.244.0.1",
        "10.244.0.1",
        "10.244.0.1",
        "10.244.0.1",
        "10.244.0.1",
        "172.18.0.2",
        "fc00:f853:ccd:e793::2",
        "fe80::42:acff:fe12:2",
        "172.21.0.2",
        "10.244.0.1",
        "10.244.0.1"
      ],
      "name": "kind-control-plane",
      "id": "0869aff1cd47d866040557d7253b4a69",
      "mac": [
        "a2:25:eb:67:3f:08",
        "4a:66:fe:df:9f:17",
        "fe:30:fe:a5:2a:3a",
        "da:69:6d:a2:ef:1a",
        "66:c7:8b:4c:e3:d0",
        "02:42:ac:12:00:02",
        "02:42:ac:15:00:02",
        "52:02:21:75:a8:fa",
        "1a:17:9c:6b:22:ca"
      ],
      "architecture": "x86_64"
    },
    "metricset": {
      "period": 5000,
      "name": "pod"
    },
    "event": {
      "duration": 327042327,
      "agent_id_status": "verified",
      "ingested": "2021-11-04T11:35:07Z",
      "module": "kubernetes",
      "dataset": "kubernetes.pod"
    }
  },
  "fields": {
    "orchestrator.cluster.name": [
      "kind"
    ],
    "elastic_agent.version": [
      "8.0.0"
    ],
    "host.os.name.text": [
      "CentOS Linux"
    ],
    "host.hostname": [
      "kind-control-plane"
    ],
    "host.mac": [
      "a2:25:eb:67:3f:08",
      "4a:66:fe:df:9f:17",
      "fe:30:fe:a5:2a:3a",
      "da:69:6d:a2:ef:1a",
      "66:c7:8b:4c:e3:d0",
      "02:42:ac:12:00:02",
      "02:42:ac:15:00:02",
      "52:02:21:75:a8:fa",
      "1a:17:9c:6b:22:ca"
    ],
    "service.type": [
      "kubernetes"
    ],
    "host.os.version": [
      "7 (Core)"
    ],
    "kubernetes.pod.network.tx.bytes": [
      0
    ],
    "kubernetes.pod.network.rx.bytes": [
      0
    ],
    "host.os.name": [
      "CentOS Linux"
    ],
    "agent.name": [
      "kind-control-plane"
    ],
    "host.name": [
      "kind-control-plane"
    ],
    "event.agent_id_status": [
      "verified"
    ],
    "kubernetes.pod.memory.rss.bytes": [
      19312640
    ],
    "kubernetes.namespace.name": [
      "kube-system"
    ],
    "host.os.type": [
      "linux"
    ],
    "kubernetes.pod.memory.page_faults": [
      10919
    ],
    "data_stream.type": [
      "metrics"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "agent.id": [
      "a71f0ee1-21c4-4023-9f79-259253a5e952"
    ],
    "host.containerized": [
      true
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "service.address": [
      "https://kind-control-plane:10250/stats/summary"
    ],
    "agent.version": [
      "8.0.0"
    ],
    "host.os.family": [
      "redhat"
    ],
    "kubernetes.pod.network.rx.errors": [
      0
    ],
    "kubernetes.node.name": [
      "kind-control-plane"
    ],
    "kubernetes.pod.network.tx.errors": [
      0
    ],
    "kubernetes.pod.uid": [
      "c052b569-c772-43e8-89cc-149ed6f5c69a"
    ],
    "kubernetes.pod.cpu.usage.nanocores": [
      4316814
    ],
    "host.ip": [
      "10.244.0.1",
      "10.244.0.1",
      "10.244.0.1",
      "10.244.0.1",
      "10.244.0.1",
      "172.18.0.2",
      "fc00:f853:ccd:e793::2",
      "fe80::42:acff:fe12:2",
      "172.21.0.2",
      "10.244.0.1",
      "10.244.0.1"
    ],
    "agent.type": [
      "metricbeat"
    ],
    "kubernetes.pod.start_time": [
      "2021-11-04T08:31:24.000Z"
    ],
    "orchestrator.cluster.url": [
      "kind-control-plane:6443"
    ],
    "kubernetes.pod.memory.major_page_faults": [
      341
    ],
    "event.module": [
      "kubernetes"
    ],
    "host.os.kernel": [
      "4.9.184-linuxkit"
    ],
    "kubernetes.pod.name": [
      "kube-scheduler-kind-control-plane"
    ],
    "kubernetes.pod.memory.available.bytes": [
      0
    ],
    "elastic_agent.snapshot": [
      true
    ],
    "host.id": [
      "0869aff1cd47d866040557d7253b4a69"
    ],
    "kubernetes.pod.ip": [
      "172.18.0.2"
    ],
    "kubernetes.pod.memory.working_set.bytes": [
      20094976
    ],
    "elastic_agent.id": [
      "a71f0ee1-21c4-4023-9f79-259253a5e952"
    ],
    "data_stream.namespace": [
      "ep"
    ],
    "metricset.period": [
      5000
    ],
    "host.os.codename": [
      "Core"
    ],
    "kubernetes.labels.tier": [
      "control-plane"
    ],
    "event.duration": [
      327042327
    ],
    "kubernetes.labels.component": [
      "kube-scheduler"
    ],
    "metricset.name": [
      "pod"
    ],
    "event.ingested": [
      "2021-11-04T11:35:07.000Z"
    ],
    "@timestamp": [
      "2021-11-04T11:35:05.941Z"
    ],
    "host.os.platform": [
      "centos"
    ],
    "error.message": [
      "field [kubernetes.namespace.name] already exists"
    ],
    "data_stream.dataset": [
      "kubernetes.pod"
    ],
    "agent.ephemeral_id": [
      "3e620d64-ad21-475e-a33b-2362427bcdfe"
    ],
    "kubernetes.pod.memory.usage.bytes": [
      61587456
    ],
    "event.dataset": [
      "kubernetes.pod"
    ]
  }
}

@MichaelKatsoulis @mtojek any ideas here?

MichaelKatsoulis · 2021-11-04T11:49:57Z

There seems to be a problem with the pipeline. I am looking into it

ChrsMark · 2021-11-04T12:51:00Z

@MichaelKatsoulis since you found the fix for the pipeline already (discussed offline) I think I will merge this one and open a follow-up with your solution.

ChrsMark · 2021-11-04T12:51:59Z

I cannot merge this one without a suceessfull CI, I will include the changes in the pipelines here too.

mtojek · 2021-11-04T12:53:41Z

I can help you with merge if you like (admin rights).

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark · 2021-11-04T12:58:34Z

I would say since the PR is already a bug fix we include this fix too.

…els (elastic#2041)

ChrsMark added 3 commits October 27, 2021 14:08

Fix conditions of k8s data_streans that are based on k8s labels

e716d0c

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

Merge remote-tracking branch 'upstream/master' into fix_k8s_package

b85313d

Add changelog and fix version

d4d7f44

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark self-assigned this Oct 27, 2021

ChrsMark requested review from MichaelKatsoulis and mtojek October 27, 2021 13:21

ChrsMark added the Team:Integrations Label for the Integrations team label Oct 27, 2021

Fix changelog's pr number

3b991c2

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

mtojek approved these changes Oct 27, 2021

View reviewed changes

ChrsMark changed the title ~~Fix k8s package~~ Fixes conditions of kubernetes data_streans that are based on k8s labels Oct 27, 2021

MichaelKatsoulis approved these changes Oct 27, 2021

View reviewed changes

ChrsMark mentioned this pull request Oct 27, 2021

[snapshot] Update kubernetes package to version 1.1.2 elastic/package-storage#2573

Merged

Update changelog.yml

1b6e3f3

ChrsMark changed the title ~~Fixes conditions of kubernetes data_streans that are based on k8s labels~~ Fixes conditions of kubernetes data_streams that are based on k8s labels Oct 27, 2021

Merge remote-tracking branch 'upstream/master' into fix_k8s_package

9febd6e

ChrsMark force-pushed the fix_k8s_package branch from 68b060d to 9febd6e Compare October 29, 2021 08:47

Merge remote-tracking branch 'upstream/master' into fix_k8s_package

f36effb

fix version

0f9927f

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

mtojek mentioned this pull request Nov 2, 2021

Kubernetes service deployer: use correct stack version elastic/elastic-package#567

Closed

mtojek mentioned this pull request Nov 3, 2021

Update dependency on elastic-package v0.28.0 #2117

Merged

ChrsMark added 2 commits November 3, 2021 15:12

Merge remote-tracking branch 'upstream/master' into fix_k8s_package

1929f0f

Merge remote-tracking branch 'upstream/master' into fix_k8s_package

c358a88

fix pipelines

167ab72

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>

ChrsMark merged commit ea03a82 into elastic:master Nov 4, 2021

ChrsMark changed the title ~~Fixes conditions of kubernetes data_streams that are based on k8s labels~~ Fixes conditions of kubernetes data_streams that are based on k8s labels & pipelines Nov 4, 2021

ChrsMark mentioned this pull request Nov 4, 2021

[snapshot] Add kubernetes package version 1.1.2 elastic/package-storage#2707

Merged

eyalkraft pushed a commit to build-security/integrations that referenced this pull request Mar 30, 2022

Fixes conditions of kubernetes data_streams that are based on k8s lab…

5b46a80

…els (elastic#2041)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes conditions of kubernetes data_streams that are based on k8s labels & pipelines #2041

Fixes conditions of kubernetes data_streams that are based on k8s labels & pipelines #2041

ChrsMark commented Oct 27, 2021

elasticmachine commented Oct 27, 2021

mtojek left a comment

elasticmachine commented Oct 27, 2021 •

edited

Loading

Build stats

Test stats 🧪

ChrsMark commented Oct 28, 2021

ChrsMark commented Nov 1, 2021

ChrsMark commented Nov 1, 2021

ChrsMark commented Nov 2, 2021

mtojek commented Nov 2, 2021 •

edited

Loading

ChrsMark commented Nov 2, 2021

mtojek commented Nov 2, 2021

mtojek commented Nov 3, 2021

ChrsMark commented Nov 3, 2021

ChrsMark commented Nov 4, 2021

MichaelKatsoulis commented Nov 4, 2021

ChrsMark commented Nov 4, 2021

ChrsMark commented Nov 4, 2021

MichaelKatsoulis commented Nov 4, 2021

ChrsMark commented Nov 4, 2021

ChrsMark commented Nov 4, 2021

mtojek commented Nov 4, 2021

ChrsMark commented Nov 4, 2021

Fixes conditions of kubernetes data_streams that are based on k8s labels & pipelines #2041

Fixes conditions of kubernetes data_streams that are based on k8s labels & pipelines #2041

Conversation

ChrsMark commented Oct 27, 2021

What does this PR do?

elasticmachine commented Oct 27, 2021

mtojek left a comment

Choose a reason for hiding this comment

elasticmachine commented Oct 27, 2021 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

ChrsMark commented Oct 28, 2021

ChrsMark commented Nov 1, 2021

ChrsMark commented Nov 1, 2021

ChrsMark commented Nov 2, 2021

mtojek commented Nov 2, 2021 • edited Loading

ChrsMark commented Nov 2, 2021

mtojek commented Nov 2, 2021

mtojek commented Nov 3, 2021

ChrsMark commented Nov 3, 2021

ChrsMark commented Nov 4, 2021

MichaelKatsoulis commented Nov 4, 2021

ChrsMark commented Nov 4, 2021

ChrsMark commented Nov 4, 2021

MichaelKatsoulis commented Nov 4, 2021

ChrsMark commented Nov 4, 2021

ChrsMark commented Nov 4, 2021

mtojek commented Nov 4, 2021

ChrsMark commented Nov 4, 2021

elasticmachine commented Oct 27, 2021 •

edited

Loading

mtojek commented Nov 2, 2021 •

edited

Loading