Warn / Thread through steps for permissions fixing (pod security policy / fsGroup)

[jayunit100]

Creating a companion issue now. As of now you cant see what the kubelet is actually doing when its mounting w/ perms. So its a minor bug i guess.

Made kubelet mount / fsgroup and PsP setup logging more explicit.

[jayunit100]

addresses #73925 by making the details around how file owners are set explicit in the kubelet. also makes explicit the decision of wether or not setup for file ownership attempt is being made.

[jayunit100]

/sig node

[chrislovecnm]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chrislovecnm, jayunit100
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: gnufied

If they are not already assigned, you can assign the PR to them by writing /assign @gnufied in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/volume/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[chrislovecnm]

/assign gnufied

[gnufied]

I like this idea of informing the user when we aren't changing permissions, but I am not sure if logging the message is right way to go about it. We already log an event when mount is successful:

  Normal   SuccessfulMountVolume  14s                kubelet, dev-hekumar-cinder2-nrr-1  MountVolume.SetUp succeeded for volume "default-token-456gk"

We could extend this message to include information about permission bits. Another thing is - just modifying CSI plugin isn't a good answer. We should make sure the change affects all volume plugins.

[jayunit100]

@gnufied ... agree that events are useful. Logs are typically sent to storage vendors when there is a problem though, hence they are equally important , albeit for a different use case then the end-user use case....

[jayunit100]

Updated the volume_unsupported call so that theres a warning there as well... that handles aws, azure cinder configmap, csi, downardapi, emptydir diskmgr mounter flocker gce git local portworx secret storageos and vsphere . Would prefer to keep the scope of this PR specifically to logs as that is the key thing for vendor communications, which is really the priority ( for me at least :) ).

Next step... better Events.

That said am happy to plumb this into eventing (events are definetly useful, for example - to developers working on CSI plugins, or apps that havent yet triaged a storage bug)... In any case, if its a hard requirement, preferably lets do the event plumbing in a follow on PR?.

If we have events... do we need logs ?

Absolutely, in my opinion, here's why (let me know if im missing something here - but this ive seen alot of outages, on all cloud platforms, even including GKE and other managed providers, and their not pretty, and typically can have symptoms for 12 or 24 hours before/after the occurrence). The reason IMO that just broadcasting events isn't particularly useful for CSI related issues is that, during outages (we just experienced one) ... they cycle, get cleaned, are stored remotely, arent aleratable via log aggreagators, arent in a timeseries w/ other granular info, and so on), so events - although useful to developers during theoretical testing - are really not super useful for production forensics, or at least, not in the current traditional data center / vendor interaction models.

Example scenarios

• You have a vendor storage outage, and a large portion of your cluster goes down, including possibly etcd itself.
• You have mounts that fail periodically over a period of a week, and want to send all kubelet logs for that time frame to a vendor for SLA etc.
• Note if a storage vendor goes out - you may not even have the etcd event storage available at all (unless events are permanantly put in the kubelet somewhere ? i dont think they are though).

[chrislovecnm]

/lgtm

[ericbannon]

nice!

[k8s-ci-robot]

@jayunit100: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jayunit100]

What are the next steps for this ? Would like to move this forward but if no interest ill close it. Needs a little rebase also, ill do that based on feedback.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.