packetbeat/{beater,protos/tcp,protos/udp,sniffer}: add initial metrics for TCP/UDP traffic #33833

efd6 · 2022-11-28T03:13:25Z

What does this PR do?

This adds metrics for TCP/UDP packet count and total bytes, and histograms
for time required to process TCP/UDP packets prior to acking from a
publication and time between TCP/UDP packet arrivals.

Why is it important?

This allows us to help users configure their systems to match the requirements that they have.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine · 2022-11-28T03:44:32Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-06T05:33:10.930+0000
Duration: 47 min 17 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1721
Skipped	19
Total	1740

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2022-11-28T06:08:32Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

mergify · 2022-12-01T03:43:31Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b st-5524 upstream/st-5524
git merge upstream/main
git push upstream st-5524

…raffic This adds metrics for UDP packet count and total bytes, and histograms for time required to process UDP packets prior to acking from a publication and time between UDP packet arrivals.

This adds metrics for TCP packet count and total bytes, and histograms for time required to process TCP packets prior to acking from a publication and time between TCP packet arrivals.

device retained in registration ID to prevent collision: panic: name 68DD99D548B50928 already used goroutine 157 [running]: github.com/elastic/elastic-agent-libs/monitoring.panicErr(...) /var/lib/jenkins/workspace/PR-33833-10-4f1d82ee-c0da-4414-bc63-8fa79cc6cbf5/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.13/monitoring/registry.go:257 github.com/elastic/elastic-agent-libs/monitoring.(*Registry).Add(0xc000569fc0, {0xc00071ae50, 0x10654fd}, {0x35b8160, 0xc0001f8440}, 0x78) /var/lib/jenkins/workspace/PR-33833-10-4f1d82ee-c0da-4414-bc63-8fa79cc6cbf5/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.13/monitoring/registry.go:155 +0xda github.com/elastic/elastic-agent-libs/monitoring.(*Registry).NewRegistry(0xc000569fc0, {0xc00071ae50, 0x10}, {0x0?, 0x35adc60?, 0x1?}) /var/lib/jenkins/workspace/PR-33833-10-4f1d82ee-c0da-4414-bc63-8fa79cc6cbf5/pkg/mod/github.com/elastic/elastic-agent-libs@v0.2.13/monitoring/registry.go:94 +0x158 github.com/elastic/beats/v7/libbeat/monitoring/inputmon.NewInputRegistry({0x3107792, 0x3}, {0xc00071ae50, 0x10}, 0xc00070ff50?) /var/lib/jenkins/workspace/PR-33833-10-4f1d82ee-c0da-4414-bc63-8fa79cc6cbf5/src/github.com/elastic/beats/libbeat/monitoring/inputmon/input.go:43 +0x99 github.com/elastic/beats/v7/packetbeat/protos/udp.newInputMetrics({0xc00071ae50?, 0x0?}, {0x0, 0x0}) /var/lib/jenkins/workspace/PR-33833-10-4f1d82ee-c0da-4414-bc63-8fa79cc6cbf5/src/github.com/elastic/beats/packetbeat/protos/udp/udp.go:155 +0x7c github.com/elastic/beats/v7/packetbeat/protos/udp.NewUDP({0x35d9388, 0xc000314420}, {0xc00071ae50, 0x10}, {0x0, 0x0}) /var/lib/jenkins/workspace/PR-33833-10-4f1d82ee-c0da-4414-bc63-8fa79cc6cbf5/src/github.com/elastic/beats/packetbeat/protos/udp/udp.go:57 +0x9d github.com/elastic/beats/v7/packetbeat/sniffer.DecodersFor.func1(0x0?, {0x0, 0x0}) /var/lib/jenkins/workspace/PR-33833-10-4f1d82ee-c0da-4414-bc63-8fa79cc6cbf5/src/github.com/elastic/beats/packetbeat/sniffer/decoders.go:69 +0x22f github.com/elastic/beats/v7/packetbeat/sniffer.(*sniffer).sniffStatic(0xc0002e64b0, {0x35d6f68, 0xc0005695c0}, {0x0, 0x0}) /var/lib/jenkins/workspace/PR-33833-10-4f1d82ee-c0da-4414-bc63-8fa79cc6cbf5/src/github.com/elastic/beats/packetbeat/sniffer/sniffer.go:282 +0x159 github.com/elastic/beats/v7/packetbeat/sniffer.(*Sniffer).Run.func1() /var/lib/jenkins/workspace/PR-33833-10-4f1d82ee-c0da-4414-bc63-8fa79cc6cbf5/src/github.com/elastic/beats/packetbeat/sniffer/sniffer.go:202 +0x1be golang.org/x/sync/errgroup.(*Group).Go.func1() /var/lib/jenkins/workspace/PR-33833-10-4f1d82ee-c0da-4414-bc63-8fa79cc6cbf5/pkg/mod/golang.org/x/sync@v0.0.0-20220819030929-7fc1605a5dde/errgroup/errgroup.go:75 +0x64 created by golang.org/x/sync/errgroup.(*Group).Go /var/lib/jenkins/workspace/PR-33833-10-4f1d82ee-c0da-4414-bc63-8fa79cc6cbf5/pkg/mod/golang.org/x/sync@v0.0.0-20220819030929-7fc1605a5dde/errgroup/errgroup.go:72 +0xa5

When reading pcap files the device is empty, so we can end up with a metric ID collision. We don't need to record metrics when reading from pcap files, so don't return a metric collector when this is the case.

sonarcloud · 2022-12-02T08:41:28Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
No Duplication information

andrewkroh

I'm assuming that you can't access /inputs/ in Packetbeat to retrieve these metrics yet. You'll need to register the handler to make them available like in

beats/filebeat/beater/filebeat.go

Lines 121 to 125 in 9249566

    
           if b.API != nil { 
        
           	if err = inputmon.AttachHandler(b.API.Router()); err != nil { 
        
           		return nil, fmt.Errorf("failed attach inputs api to monitoring endpoint server: %w", err) 
        
           	} 
        
           }

.

…s for TCP/UDP traffic (#33833) This adds metrics for TCP/UDP packet count and total bytes, and histograms for time required to process TCP/UDP packets prior to acking from a publication and time between TCP/UDP packet arrivals.

efd6 added enhancement Packetbeat Team:Security-External Integrations backport-skip Skip notification from the automated backport with mergify 8.7-candidate labels Nov 28, 2022

efd6 self-assigned this Nov 28, 2022

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Nov 28, 2022

efd6 force-pushed the st-5524 branch 2 times, most recently from e999e2a to aa0b2d0 Compare November 28, 2022 03:37

efd6 marked this pull request as ready for review November 28, 2022 06:08

efd6 requested a review from a team as a code owner November 28, 2022 06:08

efd6 force-pushed the st-5524 branch 2 times, most recently from 6e516a8 to a09db19 Compare November 29, 2022 04:26

efd6 changed the title ~~packetbeat/{beater,protos/udp,sniffer}: add initial metrics for UDP traffic~~ packetbeat/{beater,protos/tcp,protos/udp,sniffer}: add initial metrics for UDP traffic Nov 29, 2022

efd6 force-pushed the st-5524 branch 3 times, most recently from 4c73cd2 to 5f42b79 Compare November 30, 2022 00:37

efd6 added 2 commits December 1, 2022 14:14

packetbeat/protos/udp: clean up source order and error lint

0790458

packetbeat/{beater,protos/udp,sniffer}: add initial metrics for UDP t…

a91d8f5

…raffic This adds metrics for UDP packet count and total bytes, and histograms for time required to process UDP packets prior to acking from a publication and time between UDP packet arrivals.

efd6 force-pushed the st-5524 branch 2 times, most recently from 46640ad to 2fa2494 Compare December 1, 2022 04:57

efd6 changed the title ~~packetbeat/{beater,protos/tcp,protos/udp,sniffer}: add initial metrics for UDP traffic~~ packetbeat/{beater,protos/tcp,protos/udp,sniffer}: add initial metrics for TCP/UDP traffic Dec 1, 2022

efd6 force-pushed the st-5524 branch from 74e959f to eb24ffb Compare December 1, 2022 05:52

efd6 added 3 commits December 2, 2022 10:11

use monitoring/inputmon helper type

2255155

packetbeat/protos/tcp: clean up source order and error lint

2d86e6a

packetbeat/{protos/tcp,sniffer}: add initial metrics for TCP traffic

adbd718

This adds metrics for TCP packet count and total bytes, and histograms for time required to process TCP packets prior to acking from a publication and time between TCP packet arrivals.

efd6 force-pushed the st-5524 branch 2 times, most recently from 2cea13b to e928577 Compare December 2, 2022 04:24

don't panic when reading pcap files

a07435c

When reading pcap files the device is empty, so we can end up with a metric ID collision. We don't need to record metrics when reading from pcap files, so don't return a metric collector when this is the case.

efd6 force-pushed the st-5524 branch from e928577 to a07435c Compare December 2, 2022 08:35

andrewkroh reviewed Jan 4, 2023

View reviewed changes

efd6 added 2 commits January 6, 2023 14:49

Merge branch 'master' into st-5524

eeb0b9b

address pr comment

072996c

efd6 requested a review from andrewkroh January 6, 2023 05:28

fix changelog

2021c89

andrewkroh approved these changes Jan 9, 2023

View reviewed changes

efd6 merged commit a5a375b into elastic:main Jan 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

packetbeat/{beater,protos/tcp,protos/udp,sniffer}: add initial metrics for TCP/UDP traffic #33833

packetbeat/{beater,protos/tcp,protos/udp,sniffer}: add initial metrics for TCP/UDP traffic #33833

efd6 commented Nov 28, 2022 •

edited

elasticmachine commented Nov 28, 2022 •

edited by jenkins-beats-ci bot

Build stats

Test stats 🧪

elasticmachine commented Nov 28, 2022

mergify bot commented Dec 1, 2022

sonarcloud bot commented Dec 2, 2022

andrewkroh left a comment

	if b.API != nil {
	if err = inputmon.AttachHandler(b.API.Router()); err != nil {
	return nil, fmt.Errorf("failed attach inputs api to monitoring endpoint server: %w", err)
	}
	}

packetbeat/{beater,protos/tcp,protos/udp,sniffer}: add initial metrics for TCP/UDP traffic #33833

packetbeat/{beater,protos/tcp,protos/udp,sniffer}: add initial metrics for TCP/UDP traffic #33833

Conversation

efd6 commented Nov 28, 2022 • edited

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Nov 28, 2022 • edited by jenkins-beats-ci bot

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Nov 28, 2022

mergify bot commented Dec 1, 2022

sonarcloud bot commented Dec 2, 2022

andrewkroh left a comment

Choose a reason for hiding this comment

efd6 commented Nov 28, 2022 •

edited

elasticmachine commented Nov 28, 2022 •

edited by jenkins-beats-ci bot