feat(ethtool): Gather statistics from namespaces #11895

zeffron · 2022-09-27T19:51:44Z

Required for all PRs

Updated associated README.md.
Wrote appropriate unit tests.
Pull request title or commits are in conventional commit format

resolves #11754

Added network namespace support to the ethtool plugin. By default, it continues
to operate as it always had, gathering metrics from the initial namespace only.
To gather metrics from additional namespaces, CAP_SYS_ADMIN must be added to
the telegraf process, and at least one of the new namespace filters must be
configured.

Unit tests for the new namespace support also require running with
CAP_SYS_ADMIN. They will be skipped if CAP_SYS_ADMIN is misisng. The
original unit tests have been updated to do minimal namespace support testing
and do not require CAP_SYS_ADMIN.

zeffron · 2022-09-27T21:47:04Z

~~I think the macOS test failures are unrelated to anything I've done. It seems to be failures in the google_cloud_storage and graylog input plugins, which I have not touched.~~

Update: not sure what changed, but PR checks are all passing now.

powersj

Thanks for the PR, some questions and comments below!

Makefile

plugins/inputs/ethtool/README.md

powersj · 2022-09-28T14:48:11Z

plugins/inputs/ethtool/ethtool_linux.go

+	if err := netns.Set(initialNamespace); err != nil {
+		c.Log.Errorf("Could not return to initial namespace: %s", err)
+		return nil, err
+	}


My concern with this is if an existing user has some additional network namespaces created, they will suddenly start getting errors from Telegraf around operation not permitted, right?

As an example, I created one namespace and ran again with no elevated permissions and error out, whereas before I was collecting metrics without issue:

2022-09-28T14:39:04Z W! [inputs.ethtool] Could not switch to namespace vnet0: operation not permitted 2022-09-28T14:39:04Z E! [inputs.ethtool] Could not return to initial namespace: operation not permitted 2022-09-28T14:39:04Z E! [inputs.ethtool] Error in plugin: operation not permitted

Hmm.. I did make some changes that I didn't put through my local testing to satisfy the CI linter. I wonder if that changed things. The intention is that unless at least one of the new namespace filters is configured, behavior should be unchanged from prior to this change.
Let me investigate...

Found and fixed the bug. It should be working as intended now (extra warnings in the logs, but functions as before otherwise).

powersj · 2022-09-29T18:08:31Z

plugins/inputs/ethtool/ethtool_linux.go

+	namespaces, err := os.ReadDir("/var/run/netns")
+	if err != nil {
+		c.Log.Warnf("Could not find namespace directory: %s", err)
+		return allInterfaces, nil
+	}


Should the filtering of the interfaces happen before we hit this point? This way telegraf only reads this directory and attempts to change namespaces, if we have a list of namespaces to actually review.

That's one option.

It would move even more of the logic out of the abstraction layer and into the implementation layer, which also makes the unit tests less useful because they're testing less of the runtime logic. Which is why I didn't go with that approach initially.

But it would be overall more performant to filter the interfaces and namespaces sooner, plus, in the namespace case, would reduce annoying log warning spam.

Further testing revealed that this approach has very noticeable performance impact on my workload. Going to see what can be done to improve performance.

zeffron · 2022-10-07T19:53:43Z

Okay, I think this is ready again. More major refactor, but it now creates an OS thread per-namespace. This saves significant CPU performance at the cost of some extra memory. It also makes it easier to have cleaner logs in the case where default (non-namespaced) behavior is desired.

powersj

Thanks for the updates! I have a couple minor comments.

I do want to chat with others on the team about whether this should be a possible separate plugin. I don't want the additional namespace tooling and library to get in the way in what is a rather simple plugin. But this might be fine as is.

plugins/inputs/ethtool/README.md

plugins/inputs/ethtool/ethtool_linux.go

plugins/inputs/ethtool/namespace_linux.go

zeffron · 2022-10-12T20:45:54Z

I do want to chat with others on the team about whether this should be a possible separate plugin. I don't want the additional namespace tooling and library to get in the way in what is a rather simple plugin. But this might be fine as is.

I think ideally, this should become a more global configuration option with an easy way for plugins to take advantage of the namespace logic to do their thing. Many more plugins than just ethtool have the same limitation/issue. This is just (currently) the only networking plugin used in my project, so it's the only one where we need namespace support.

plugins/inputs/ethtool/ethtool_test.go

plugins/inputs/ethtool/ethtool_linux.go

To support monitoring an entire host with one telegraf process, or any number of other uses of namespaced network interfaces, the ethtool plugin now supports gathering statistics from interfaces in additional namespaces. This functionality must be enabled by adding (at least) one of the new namespace filters to the plugin configuration, and requires adding CAP_SYS_ADMIN to the telegraf process. Resolves influxdata#11754

telegraf-tiger · 2022-11-10T21:26:34Z

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

srebhan

Looks good to me. Thanks for your work @zeffron!

telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Sep 27, 2022

zeffron force-pushed the ethtool_namespaces branch 4 times, most recently from bb4b613 to d64989e Compare September 27, 2022 20:57

powersj reviewed Sep 28, 2022

View reviewed changes

zeffron force-pushed the ethtool_namespaces branch 6 times, most recently from 9861d16 to a1db0ff Compare September 28, 2022 19:08

powersj reviewed Sep 29, 2022

View reviewed changes

powersj added the waiting for response waiting for response from contributor label Oct 3, 2022

zeffron marked this pull request as draft October 3, 2022 20:29

zeffron force-pushed the ethtool_namespaces branch from a1db0ff to a0b6bfb Compare October 7, 2022 19:31

zeffron marked this pull request as ready for review October 7, 2022 19:51

telegraf-tiger bot removed the waiting for response waiting for response from contributor label Oct 7, 2022

powersj reviewed Oct 11, 2022

View reviewed changes

plugins/inputs/ethtool/README.md Outdated Show resolved Hide resolved

plugins/inputs/ethtool/ethtool_linux.go Outdated Show resolved Hide resolved

plugins/inputs/ethtool/namespace_linux.go Outdated Show resolved Hide resolved

reimda suggested changes Oct 17, 2022

View reviewed changes

plugins/inputs/ethtool/ethtool_test.go Outdated Show resolved Hide resolved

plugins/inputs/ethtool/ethtool_linux.go Show resolved Hide resolved

zeffron force-pushed the ethtool_namespaces branch from a0b6bfb to 2acdc38 Compare October 28, 2022 23:39

zeffron requested review from reimda and powersj and removed request for reimda and powersj October 28, 2022 23:53

zeffron force-pushed the ethtool_namespaces branch from 5792e40 to 58d7abd Compare November 10, 2022 19:52

zeffron force-pushed the ethtool_namespaces branch from 3be205e to d75d718 Compare November 10, 2022 19:59

fixup! feat(ethtool): Gather statistics from namespaces

f9587b9

zeffron requested review from powersj and removed request for reimda November 10, 2022 21:30

powersj approved these changes Nov 14, 2022

View reviewed changes

powersj added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Nov 14, 2022

srebhan approved these changes Nov 15, 2022

View reviewed changes

srebhan merged commit c7a1d9e into influxdata:master Nov 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ethtool): Gather statistics from namespaces #11895

feat(ethtool): Gather statistics from namespaces #11895

zeffron commented Sep 27, 2022 •

edited

zeffron commented Sep 27, 2022 •

edited

powersj left a comment

powersj Sep 28, 2022

zeffron Sep 28, 2022

zeffron Sep 28, 2022

powersj Sep 29, 2022

zeffron Sep 29, 2022

zeffron Sep 29, 2022

zeffron commented Oct 7, 2022

powersj left a comment

zeffron commented Oct 12, 2022

telegraf-tiger bot commented Nov 10, 2022

Artifact URLs

srebhan left a comment

feat(ethtool): Gather statistics from namespaces #11895

feat(ethtool): Gather statistics from namespaces #11895

Conversation

zeffron commented Sep 27, 2022 • edited

Required for all PRs

zeffron commented Sep 27, 2022 • edited

powersj left a comment

Choose a reason for hiding this comment

powersj Sep 28, 2022

Choose a reason for hiding this comment

zeffron Sep 28, 2022

Choose a reason for hiding this comment

zeffron Sep 28, 2022

Choose a reason for hiding this comment

powersj Sep 29, 2022

Choose a reason for hiding this comment

zeffron Sep 29, 2022

Choose a reason for hiding this comment

zeffron Sep 29, 2022

Choose a reason for hiding this comment

zeffron commented Oct 7, 2022

powersj left a comment

Choose a reason for hiding this comment

zeffron commented Oct 12, 2022

telegraf-tiger bot commented Nov 10, 2022

Artifact URLs

srebhan left a comment

Choose a reason for hiding this comment

zeffron commented Sep 27, 2022 •

edited

zeffron commented Sep 27, 2022 •

edited