[metricbeat] Add state_job metricset #26479

jarpy · 2021-06-24T14:28:43Z

Add a Metricbeat metricset for Kubernetes Job resources.

What does this PR do?

Adds a state_job metricset to the Metricbeat Kubernetes module. Follows the established pattern of consuming Kube State Metrics output.

Why is it important?

Enables monitoring and alerting for Kubernetes Jobs.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Run unit tests

$ MODULE=kubernetes mage -v test

Deploy Metricbeat and enable this metricset. Verify that metrics about Jobs are being collected as expected along with k8s metadata.

Add a Metricbeat metricset for Kubernetes Job resources.

jarpy · 2021-06-24T14:32:19Z

metricbeat/module/kubernetes/state_job/state_job.go

+			// Add owner information provided by the "kube_job_owner" InfoMetric.
+			"owner_kind":          p.Label("owner.kind"),
+			"owner_name":          p.Label("owner.name"),
+			"owner_is_controller": p.Label("owner.is_controller"),


This could, in theory, be a proper boolean, but I wasn't sure how to cast it down here in the Labels section, where BooleanMetric can't be used (as far as I can tell).

Any tips? OK to leave it as a "boolean string", maybe?

Since we don't just have true/false but it can also be <none> I think we are OK with a "boolean string".

Cool, thanks. Would you normally store the exact "<none>" string from KSM, or should I translate it to "none" or similar?

We could "cook" it a little bit and convert it to "none" but I think keeping as is work too. I'm more into leaving it as what we get from the original source.

So need to change :).

elasticmachine · 2021-06-24T14:53:45Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Started by user Toby McLaughlin
Start Time: 2021-06-30T23:20:40.199+0000
Duration: 144 min 32 sec
Commit: 81d1a33

Test stats 🧪

Test	Results
Failed	0
Passed	47307
Skipped	5257
Total	52564

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	47307
Skipped	5257
Total	52564

jarpy · 2021-06-25T03:54:26Z

Fixed a couple of errors detected by CI.

I haven't been able to get the integtests running locally, sorry about that,

deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml

metricbeat/docs/modules/kubernetes.asciidoc

elasticmachine · 2021-06-28T13:32:38Z

Pinging @elastic/integrations (Team:Integrations)

ChrsMark

Thank you for this PR @jarpy, looks great already! I left some comments, let me know what you think!

deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml

deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-daemonset-configmap.yaml

ChrsMark · 2021-06-28T14:29:13Z

metricbeat/module/kubernetes/state_job/_meta/fields.yml

+        - name: is_controller
+          type: keyword
+          description: >
+            Owner is controller ("true" or "false")


It can be also <none>, or this value is just omitted?

You're quite right. <none> is a possibility. I hadn't seen it because I had been testing with CronJobs.

Fixed in 618705f.

ChrsMark · 2021-06-28T14:30:08Z

metricbeat/module/kubernetes/state_job/state_job.go

+			// Add owner information provided by the "kube_job_owner" InfoMetric.
+			"owner_kind":          p.Label("owner.kind"),
+			"owner_name":          p.Label("owner.name"),
+			"owner_is_controller": p.Label("owner.is_controller"),


Since we don't just have true/false but it can also be <none> I think we are OK with a "boolean string".

metricbeat/docs/modules/kubernetes.asciidoc

Relates: elastic#26479 (comment)

jarpy · 2021-06-30T11:39:20Z

Thanks for checking this out @ChrsMark, and for the keen observations.

ChrsMark

lgtm! Thanks for adding this!

Let's wfg now.

MichaelKatsoulis

lgtm. Thanks @jarpy !

jarpy · 2021-06-30T23:04:46Z

Thank you both! We have big things planned for our fleet monitoring after this is in. :D

Let's wfg now.

I don't know what that is, but I'm going to assume it's a good thing. ;)

As a guest here, I'm not sure if I should hit "merge", or you would prefer to. I also don't know your backport workflow. So I must ask for a just a little more of your time and help, cheers.

Jenkins, test this please, becuase it looks like the last build stalled on Auditbeat at:

Allocating a worker with the labels 'immutable && ubuntu-18'

jarpy · 2021-06-30T23:09:59Z

Jenkins, retest this please (?)

jarpy · 2021-06-30T23:21:05Z

Triggered rerun from the Jenkins UI.

ChrsMark · 2021-07-01T07:50:43Z

Hey @jarpy sorry for stalling cause of the CI, wfg (== waitForGreen) took longer...

This will hit 7.15 and hence the backport-v7.15.0 label should be enough to open the backport PR to the proper branch (7.x here).

I will merge it now and take take care of merging the backport PR too.

Thanks again and feel free to reach out to us while you are working on your monitoring solution cause dogfooding is always valuable!

(cherry picked from commit 24c07b2)

(cherry picked from commit 24c07b2) # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/kubernetes/fields.go

…stage-failed-within-same-build * upstream/master: (36 commits) Revert "[CI] fight the flakiness with some retry option in the CI only for the Pull Requests (elastic#26617)" (elastic#26704) Packaging: linux/armv7 is not supported (elastic#26706) Cyberarkpas: Link to official docs on how to setup TLS (elastic#26614) Make network_direction, registered_domain and convert processors compatible with ES older than 7.13.0 (elastic#26676) Disable armv7 packaging (elastic#26679) [Heartbeat] use --params flag for synthetics (elastic#26674) Update dependent package to avoid downloading a suspicious file (elastic#26406) [mergify] set title and allow bp in any direction (elastic#26648) Fix memory leak in SQL helper when database is not available (elastic#26607) [CI] fight the flakiness with some retry option in the CI only for the Pull Requests (elastic#26617) [mergify] automate PRs that change the backport rules (elastic#26641) [Metricbeat] Add Airflow module in xpack (elastic#26220) chore: add-backport-next (elastic#26620) [metricbeat] Add state_job metricset (elastic#26479) CI: jenkins labels are less time consuming now (elastic#26613) [MetricBeat] [AWS] Fix aws metric tags with resourcegroupstaggingapi paginator (elastic#26385) (elastic#26443) Move openmetrics module to oss (elastic#26561) Skip flaky test TestFilestreamMetadataUpdatedOnRename (elastic#26609) [filebeat][fortinet] Use default add_locale for fortinet.firewall (elastic#26524) Enroll proxy settings (elastic#26514) ...

[metricbeat] Add state_job metricset

97c04a8

Add a Metricbeat metricset for Kubernetes Job resources.

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 24, 2021

jarpy commented Jun 24, 2021

View reviewed changes

jarpy requested a review from ChrsMark June 24, 2021 14:33

jarpy added the Team:Observability label Jun 24, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 24, 2021

[metricbeat] Run "mage check" to fix state_job

1b43db4

jarpy added 2 commits June 25, 2021 00:37

[metricbeat] Fix state_job typo in agent config

d15ce92

[metricbeat] Fix package in state_job integ test

32effd9

jarpy commented Jun 25, 2021

View reviewed changes

deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml Outdated Show resolved Hide resolved

jarpy commented Jun 25, 2021

View reviewed changes

metricbeat/docs/modules/kubernetes.asciidoc Show resolved Hide resolved

ChrsMark added the Team:Integrations Label for the Integrations team label Jun 28, 2021

ChrsMark reviewed Jun 28, 2021

View reviewed changes

ChrsMark added the test-plan Add this PR to be manual test plan label Jun 28, 2021

ChrsMark requested a review from MichaelKatsoulis June 28, 2021 14:47

ChrsMark mentioned this pull request Jun 28, 2021

Add state_job data_stream for kubernetes package elastic/integrations#1227

Closed

jarpy added 2 commits June 30, 2021 16:34

[metric] Fix state_job ordering in agent manifests

95d96e1

Relates: elastic#26479 (comment)

[metricbeat] Update field docs for state_job

618705f

Add batch/jobs authz in k8s manifests

81d1a33

ChrsMark approved these changes Jun 30, 2021

View reviewed changes

MichaelKatsoulis approved these changes Jun 30, 2021

View reviewed changes

ChrsMark added the backport-v7.15.0 Automated backport with mergify label Jul 1, 2021

ChrsMark merged commit 24c07b2 into elastic:master Jul 1, 2021

ChrsMark assigned ChrsMark and MichaelKatsoulis Jul 1, 2021

ChrsMark mentioned this pull request Jul 1, 2021

Add agent fleet enrolment k8s manifest #26566

Merged

6 tasks

jarpy deleted the state_job branch July 1, 2021 14:12

ChrsMark pushed a commit to ChrsMark/beats that referenced this pull request Jul 1, 2021

[metricbeat] Add state_job metricset (elastic#26479)

ddc6c5f

(cherry picked from commit 24c07b2)

ChrsMark mentioned this pull request Jul 1, 2021

Cherry-pick #26479 to 7.x: [metricbeat] Add state_job metricset #26670

Merged

6 tasks

ChrsMark added the v7.15.0 label Jul 1, 2021

mergify bot pushed a commit that referenced this pull request Jul 1, 2021

[metricbeat] Add state_job metricset (#26479)

0955e61

(cherry picked from commit 24c07b2) # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/module/kubernetes/fields.go

mergify bot mentioned this pull request Jul 1, 2021

[7.x](backport #26479) [metricbeat] Add state_job metricset #26671

Closed

ChrsMark added a commit that referenced this pull request Jul 5, 2021

[metricbeat] Add state_job metricset (#26479) (#26670)

61c89f8

MichaelKatsoulis mentioned this pull request Jul 6, 2021

Add state_job data_stream in kubernetes package elastic/integrations#1290

Merged

4 tasks

andresrc added the test-plan-added This PR has been added to the test plan label Aug 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[metricbeat] Add state_job metricset #26479

[metricbeat] Add state_job metricset #26479

jarpy commented Jun 24, 2021 •

edited by ChrsMark

jarpy Jun 24, 2021

ChrsMark Jun 28, 2021

jarpy Jun 30, 2021

ChrsMark Jun 30, 2021

elasticmachine commented Jun 24, 2021 •

edited by jenkins-beats-ci bot

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

jarpy commented Jun 25, 2021

elasticmachine commented Jun 28, 2021

ChrsMark left a comment

ChrsMark Jun 28, 2021

jarpy Jun 30, 2021

ChrsMark Jun 28, 2021

jarpy commented Jun 30, 2021

ChrsMark left a comment

MichaelKatsoulis left a comment

jarpy commented Jun 30, 2021

jarpy commented Jun 30, 2021

jarpy commented Jun 30, 2021

ChrsMark commented Jul 1, 2021 •

edited

[metricbeat] Add state_job metricset #26479

[metricbeat] Add state_job metricset #26479

Conversation

jarpy commented Jun 24, 2021 • edited by ChrsMark

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

jarpy Jun 24, 2021

Choose a reason for hiding this comment

ChrsMark Jun 28, 2021

Choose a reason for hiding this comment

jarpy Jun 30, 2021

Choose a reason for hiding this comment

ChrsMark Jun 30, 2021

Choose a reason for hiding this comment

elasticmachine commented Jun 24, 2021 • edited by jenkins-beats-ci bot

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

jarpy commented Jun 25, 2021

elasticmachine commented Jun 28, 2021

ChrsMark left a comment

Choose a reason for hiding this comment

ChrsMark Jun 28, 2021

Choose a reason for hiding this comment

jarpy Jun 30, 2021

Choose a reason for hiding this comment

ChrsMark Jun 28, 2021

Choose a reason for hiding this comment

jarpy commented Jun 30, 2021

ChrsMark left a comment

Choose a reason for hiding this comment

MichaelKatsoulis left a comment

Choose a reason for hiding this comment

jarpy commented Jun 30, 2021

jarpy commented Jun 30, 2021

jarpy commented Jun 30, 2021

ChrsMark commented Jul 1, 2021 • edited

jarpy commented Jun 24, 2021 •

edited by ChrsMark

elasticmachine commented Jun 24, 2021 •

edited by jenkins-beats-ci bot

ChrsMark commented Jul 1, 2021 •

edited