Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1956836: templates: Rework node-valid-hostname to fix SELinux denial #2618

Merged
merged 1 commit into from Jun 18, 2021
Merged

Bug 1956836: templates: Rework node-valid-hostname to fix SELinux denial #2618

merged 1 commit into from Jun 18, 2021

Conversation

cgwalters
Copy link
Member

BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1956836

Unfortunately right now, executing bash skips a domain transition
(see fedora-selinux/selinux-policy#778)
so the way we're sourcing the script means we stay in initrc_t
and end up triggering a SELinux policy denial.

(BTW this denial turns out to just delay the successful exit
of the script, which will then end up just delaying kubelet start.
it's otherwise harmless, but we also don't want SELinux policy denials
in our product by default)

Fix this in two ways:

While we're here:

  • Clean the confusing+outdated comment about being a NM dispatcher
  • Drop the logger bit which was only necessary as a NM dispatcher;
    since we're always running under systemd, this makes journalctl -u node-valid-hostname
    actually show the script output.
  • Make it crystal clear that the "truncate hostname" is only run in GCP.
  • Fix various typos
  • Use the more precise term "non-localhost" in various places instead of the more ambiguous
    terms "real"/"valid"

@cgwalters cgwalters changed the title templates: Rework node-valid-hostname to fix SELinux denial BZ 1956836: templates: Rework node-valid-hostname to fix SELinux denial Jun 14, 2021
@cgwalters cgwalters changed the title BZ 1956836: templates: Rework node-valid-hostname to fix SELinux denial Bug 1956836: templates: Rework node-valid-hostname to fix SELinux denial Jun 14, 2021
@openshift-ci openshift-ci bot added the bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. label Jun 14, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 14, 2021

@cgwalters: This pull request references Bugzilla bug 1956836, which is invalid:

  • expected the bug to target the "4.9.0" release, but it targets "4.8.0" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1956836: templates: Rework node-valid-hostname to fix SELinux denial

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jun 14, 2021
@cgwalters
Copy link
Member Author

/bugzilla refresh

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 14, 2021

@cgwalters: This pull request references Bugzilla bug 1956836, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.0) matches configured target release for branch (4.9.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @mike-nguyen

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jun 14, 2021
@openshift-ci openshift-ci bot requested a review from mike-nguyen June 14, 2021 21:55
@cgwalters
Copy link
Member Author

cgwalters commented Jun 14, 2021 •
edited

/hold
while we review the CI results form this PR on AWS and GCP (two platforms which will exercise the relevant codepaths) - we should see success/failure in the node journals.

(I didn't really test this locally)

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 14, 2021
@sinnykumari
Copy link
Contributor

/cc @darkmuggle can you please review this.

@cgwalters
Copy link
Member Author

cgwalters commented Jun 16, 2021 •
edited

OK so the new code still works:
GCP logs look good:
Jun 14 22:16:42.102005 ci-op-k5y0btj8-1354f-8rg5c-master-0.c.openshift-gce-devel-ci.int mco-hostname[1280]: ci-op-k5y0btj8-1354f-8rg5c-master-0.c.openshift-gce-devel-ci.internal is longer than 63 characters, using truncated hostname

AWS logs look good:
Jun 14 23:07:26.927262 ip-10-0-128-41 mco-hostname[1575]: waiting for non-localhost hostname to be assigned Jun 14 23:07:26.928323 ip-10-0-128-41 mco-hostname[1575]: node identified as ip-10-0-128-41

But unfortunately because each iptables invocation ends up in the audit log, and we do a lot of those, any AVC denials very quickly get rotated out and lost. This is really useful data we're just throwing away 😢
So I haven't verified for sure this fixes the AVC denial yet. cluster-bot is failing to launch a cluster from this PR right now, so if that isn't fixed soon I'll do it by hand and verify, but I'm also looking in parallel at how to filter the iptables stuff out of the audit logs.

@cgwalters
templates: Rework node-valid-hostname to fix SELinux denial
783aa2c 
BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1956836

Unfortunately right now, executing `bash` skips a domain transition
(see fedora-selinux/selinux-policy#778)
so the way we're sourcing the script means we stay in `initrc_t`
and end up triggering a SELinux policy denial.

(BTW this denial turns out to just delay the successful exit
 of the script, which will then end up just delaying kubelet start.
 it's otherwise harmless, but we also don't want SELinux policy denials
 in our product by default)

Fix this in two ways:

- First, just move the thing to `/usr/local/bin` to avoid issues
  with labeling of `/usr/local/sbin` that were fixed in openshift/os#551
- Second, rework it to be executed directly

While we're here:

- Clean the confusing+outdated comment about being a NM dispatcher
- Drop the `logger` bit which was only necessary as a NM dispatcher;
  since we're *always* running under systemd, this makes `journalctl -u node-valid-hostname`
  actually show the script output.
- Make it crystal clear that the "truncate hostname" is only run in GCP.
- Fix various typos
- Use the more precise term "non-localhost" in various places instead of the more ambiguous
  terms "real"/"valid"
darkmuggle
darkmuggle approved these changes Jun 16, 2021
View reviewed changes
Copy link
Contributor

@darkmuggle darkmuggle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

The fix-ups are nice -- especially dropping things that are not longer needed after the hack of dispatcher scripts were dropped.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 16, 2021
@cgwalters
Copy link
Member Author

/hold cancel
Verified that there's no more SELinux denial in a fresh GCP cluster launched from this PR.

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 16, 2021
@cgwalters
Copy link
Member Author

@sinnykumari this got a lgtm, mind adding an /approve of the general idea? 🙏

@cgwalters
Copy link
Member Author

xref linux-audit/audit-userspace#205

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 17, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, darkmuggle, sinnykumari

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 17, 2021
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

9 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 18, 2021

@cgwalters: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-aws-disruptive 783aa2c link /test e2e-aws-disruptive

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit a1a825b into openshift:master Jun 18, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 18, 2021

@cgwalters: All pull requests linked via external trackers have merged:

Bugzilla bug 1956836 has been moved to the MODIFIED state.

In response to this:

Bug 1956836: templates: Rework node-valid-hostname to fix SELinux denial

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@cgwalters cgwalters mentioned this pull request Jun 22, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@darkmuggle darkmuggle darkmuggle approved these changes

@sinnykumari sinnykumari sinnykumari approved these changes

@kikisdeliveryservice kikisdeliveryservice Awaiting requested review from kikisdeliveryservice

@yuqi-zhang yuqi-zhang Awaiting requested review from yuqi-zhang

@mike-nguyen mike-nguyen Awaiting requested review from mike-nguyen

Assignees

@darkmuggle darkmuggle

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@cgwalters @sinnykumari @openshift-bot @darkmuggle @openshift-merge-robot