Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

datapath: Filter out bpftool probes emitting dmesg messages #10164

Merged
merged 3 commits into from Feb 13, 2020

Conversation

vadorovsky
Copy link
Member

@vadorovsky vadorovsky commented Feb 12, 2020

bpftool feature probes related to trace, perf and write_user helpers are
emitting dmesg messages with warnings which may be confusing for
operators running Cilium on production environments. After this change,
those probes will be not performed.

Fixes #10048

Signed-off-by: Michal Rostecki mrostecki@opensuse.org

Filter out bpftool probes emitting dmesg messages

This change is Reviewable

@vadorovsky vadorovsky requested a review from a team February 12, 2020 10:50
@maintainer-s-little-helper maintainer-s-little-helper bot added this to In progress in 1.8.0 Feb 12, 2020
@vadorovsky
Copy link
Member Author

cilium/linux#1 needs to be accepted first

@vadorovsky vadorovsky added this to In progress in 1.7.0 via automation Feb 12, 2020
@vadorovsky vadorovsky added sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. needs-backport/1.7 labels Feb 12, 2020
@maintainer-s-little-helper
Copy link

Release note label not set, please set the appropriate release note.

1 similar comment
@maintainer-s-little-helper
Copy link

Release note label not set, please set the appropriate release note.

@coveralls
Copy link

coveralls commented Feb 12, 2020

Coverage Status

Coverage decreased (-0.04%) to 44.518% when pulling 4a476c2 on mrostecki:fix-bpftool-dmesg into f5c42e7 on cilium:master.

@vadorovsky vadorovsky requested a review from a team as a code owner February 12, 2020 12:07
@vadorovsky vadorovsky force-pushed the fix-bpftool-dmesg branch 2 times, most recently from a937ada to c09f3cf Compare February 12, 2020 12:30
Copy link
Member

@qmonnet qmonnet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pending the merge of the Linux PR, we're good to go with this one. Thanks!

@aanm aanm added the release-note/misc This PR makes changes that have no direct user impact. label Feb 12, 2020
@aanm aanm added release-note/bug This PR fixes an issue in a previous release of Cilium. and removed release-note/misc This PR makes changes that have no direct user impact. labels Feb 12, 2020
@vadorovsky
Copy link
Member Author

test-me-please

@vadorovsky vadorovsky requested a review from a team as a code owner February 12, 2020 16:44
bpftool feature probes related to trace, perf and write_user helpers are
emitting dmesg messages with warnings which may be confusing for
operators running Cilium on production environments. After this change,
those probes will be not performed.

Fixes cilium#10048

Signed-off-by: Michal Rostecki <mrostecki@opensuse.org>
Cilium fork of the Linux kernel contains necessary enhancements for
bpftool which are not avalavle upstream yet.

Ref: cilium#10048

Signed-off-by: Michal Rostecki <mrostecki@opensuse.org>
This should be done in packer-ci-build, but to get the fix faster, we
install patched bpftool here as a temporary hack...

Signed-off-by: Michal Rostecki <mrostecki@opensuse.org>
@vadorovsky
Copy link
Member Author

test-me-please

@joestringer joestringer merged commit c4b6095 into cilium:master Feb 13, 2020
1.7.0 automation moved this from In progress to Merged Feb 13, 2020
1.8.0 automation moved this from In progress to Merged Feb 13, 2020
@joestringer joestringer removed this from Needs backport from master in 1.7.0-rc4 Feb 13, 2020
@joestringer joestringer added this to Backport pending to v1.7 in 1.7.0-final Feb 13, 2020
@vadorovsky vadorovsky deleted the fix-bpftool-dmesg branch February 13, 2020 09:57
@maintainer-s-little-helper maintainer-s-little-helper bot added this to Backport pending to v1.7 in 1.7.0-rc4 Feb 13, 2020
pchaigno added a commit that referenced this pull request Feb 14, 2020
Provisioning of the Vagrant development environment fails due to the
missing forked bpftool:

  $ ./contrib/vagrant/start.sh
  [...]
  runtime1: Feb 13 17:26:23 runtime1 cilium-agent[20863]: level=error msg="Command execution failed" cmd="[bpftool -j feature probe filter_out \\(trace\\|write_user\\)]" error="exit status 255" subsys=probes
  runtime1: Feb 13 17:26:23 runtime1 cilium-agent[20863]: level=warning msg="{\"error\":\"expected no more arguments, 'kernel', 'dev', 'macros' or 'prefix', got: 'filter_out'?\"}" subsys=probes
  runtime1: Feb 13 17:26:23 runtime1 cilium-agent[20863]: level=fatal msg="could not run bpftool" error="exit status 255" subsys=probes
  runtime1: Cilium failed to start
  [...]

This provisioning failure was introduced by #10164. Cilium now expects
bpftool to be Cilium's (temporary) forked version, but the VirtualBox VM
has the upstream bpftool. This commit installs the forked bpftool as part
of the Vagrant provisioning.

Signed-off-by: Paul Chaignon <paul@cilium.io>
aanm pushed a commit that referenced this pull request Feb 14, 2020
Provisioning of the Vagrant development environment fails due to the
missing forked bpftool:

  $ ./contrib/vagrant/start.sh
  [...]
  runtime1: Feb 13 17:26:23 runtime1 cilium-agent[20863]: level=error msg="Command execution failed" cmd="[bpftool -j feature probe filter_out \\(trace\\|write_user\\)]" error="exit status 255" subsys=probes
  runtime1: Feb 13 17:26:23 runtime1 cilium-agent[20863]: level=warning msg="{\"error\":\"expected no more arguments, 'kernel', 'dev', 'macros' or 'prefix', got: 'filter_out'?\"}" subsys=probes
  runtime1: Feb 13 17:26:23 runtime1 cilium-agent[20863]: level=fatal msg="could not run bpftool" error="exit status 255" subsys=probes
  runtime1: Cilium failed to start
  [...]

This provisioning failure was introduced by #10164. Cilium now expects
bpftool to be Cilium's (temporary) forked version, but the VirtualBox VM
has the upstream bpftool. This commit installs the forked bpftool as part
of the Vagrant provisioning.

Signed-off-by: Paul Chaignon <paul@cilium.io>
nebril pushed a commit that referenced this pull request Feb 17, 2020
[ upstream commit 1810709 ]

Provisioning of the Vagrant development environment fails due to the
missing forked bpftool:

  $ ./contrib/vagrant/start.sh
  [...]
  runtime1: Feb 13 17:26:23 runtime1 cilium-agent[20863]: level=error msg="Command execution failed" cmd="[bpftool -j feature probe filter_out \\(trace\\|write_user\\)]" error="exit status 255" subsys=probes
  runtime1: Feb 13 17:26:23 runtime1 cilium-agent[20863]: level=warning msg="{\"error\":\"expected no more arguments, 'kernel', 'dev', 'macros' or 'prefix', got: 'filter_out'?\"}" subsys=probes
  runtime1: Feb 13 17:26:23 runtime1 cilium-agent[20863]: level=fatal msg="could not run bpftool" error="exit status 255" subsys=probes
  runtime1: Cilium failed to start
  [...]

This provisioning failure was introduced by #10164. Cilium now expects
bpftool to be Cilium's (temporary) forked version, but the VirtualBox VM
has the upstream bpftool. This commit installs the forked bpftool as part
of the Vagrant provisioning.

Signed-off-by: Paul Chaignon <paul@cilium.io>
Signed-off-by: Maciej Kwiek <maciej@isovalent.com>
borkmann pushed a commit that referenced this pull request Feb 17, 2020
[ upstream commit 1810709 ]

Provisioning of the Vagrant development environment fails due to the
missing forked bpftool:

  $ ./contrib/vagrant/start.sh
  [...]
  runtime1: Feb 13 17:26:23 runtime1 cilium-agent[20863]: level=error msg="Command execution failed" cmd="[bpftool -j feature probe filter_out \\(trace\\|write_user\\)]" error="exit status 255" subsys=probes
  runtime1: Feb 13 17:26:23 runtime1 cilium-agent[20863]: level=warning msg="{\"error\":\"expected no more arguments, 'kernel', 'dev', 'macros' or 'prefix', got: 'filter_out'?\"}" subsys=probes
  runtime1: Feb 13 17:26:23 runtime1 cilium-agent[20863]: level=fatal msg="could not run bpftool" error="exit status 255" subsys=probes
  runtime1: Cilium failed to start
  [...]

This provisioning failure was introduced by #10164. Cilium now expects
bpftool to be Cilium's (temporary) forked version, but the VirtualBox VM
has the upstream bpftool. This commit installs the forked bpftool as part
of the Vagrant provisioning.

Signed-off-by: Paul Chaignon <paul@cilium.io>
Signed-off-by: Maciej Kwiek <maciej@isovalent.com>
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Backport pending to v1.7 to Backport done to v1.7 in 1.7.0-rc4 Feb 18, 2020
@joestringer joestringer moved this from Needs backport from master to Backport done to v1.7 in 1.7.0-final Feb 18, 2020
@joestringer joestringer removed this from Backport done to v1.7 in 1.7.0-rc4 Feb 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note/bug This PR fixes an issue in a previous release of Cilium. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.
Projects
No open projects
1.7.0
  
Merged
1.7.0-final
Backport done to v1.7
1.8.0
  
Merged
Development

Successfully merging this pull request may close these issues.

agent: avoid bpftool emitting dmesg messages on general helper probes
6 participants