Plugin watcher getting notified of every Kubelet directory event #69015

vladimirvivien · 2018-09-25T00:53:41Z

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug
This is more of an enhancement.

What happened:
The Plugin Watcher is setup to trigger on file activities when they occur in the subdirectory that it scans. During startup, the Watcher is passed (from within the Kubelet) the Kubelet's root plugin directory as a location to scan for newly added plugins -

kubernetes/pkg/kubelet/kubelet.go

Line 789 in 8346631

klet.pluginWatcher = pluginwatcher.NewWatcher(klet.getPluginsDir())

Because the kubelet's plugins subdirectory is also used by existing in-tree drivers (including csi) as a location to store plugins artifacts, the Plugin Watcher is getting triggered by directory/file activities occurring in that subdirectory that has nothing to do with plugin registration as shown in the output below:

> cat kubelog.txt | grep "plugin_watcher"
E0921 18:41:49.164687   81286 plugin_watcher.go:115] error dial failed at socket /var/lib/kubelet/plugins/csi-hostpath/csi.sock, err: failed to dial socket /var/lib/kubelet/plugins/csi-hostpath/csi.sock, err: context deadline exceeded when handling create event: "/var/lib/kubelet/plugins/csi-hostpath/csi.sock": CREATE
E0921 18:42:12.126656   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/csi-hostpath/csi.sock when handling delete event: "/var/lib/kubelet/plugins/csi-hostpath/csi.sock": REMOVE
E0921 18:42:12.127706   81286 plugin_watcher.go:115] error failed to get plugin info using RPC GetInfo at socket /var/lib/kubelet/plugins/csi-hostpath/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration when handling create event: "/var/lib/kubelet/plugins/csi-hostpath/csi.sock": CREATE
E0921 18:42:20.988913   81286 plugin_watcher.go:115] error error accessing path: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 error: lstat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8: no such file or directory when handling create event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": CREATE
E0921 18:42:20.989211   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": REMOVE
E0921 18:42:21.495810   81286 plugin_watcher.go:115] error failed to watch /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8, err: no such file or directory when handling create event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": CREATE
E0921 18:42:21.495960   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": REMOVE
E0921 18:42:22.512498   81286 plugin_watcher.go:115] error failed to watch /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8, err: no such file or directory when handling create event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": CREATE
E0921 18:42:22.512767   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": REMOVE
E0921 18:42:24.539679   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount": REMOVE
E0921 18:42:24.539725   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": REMOVE
E0921 18:42:24.539743   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/vol_data.json when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/vol_data.json": REMOVE
E0921 18:42:24.539780   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": REMOVE
E0921 18:42:24.539612   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount": REMOVE
E0921 18:42:28.592695   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount": REMOVE
E0921 18:42:28.592728   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount": REMOVE
E0921 18:42:28.592752   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": REMOVE
E0921 18:42:28.592767   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/vol_data.json when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/vol_data.json": REMOVE
E0921 18:42:28.592967   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": REMOVE
E0921 18:42:36.611609   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/vol_data.json when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/vol_data.json": REMOVE
E0921 18:42:36.611485   81286 plugin_watcher.go:115] error error accessing path: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount error: lstat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount: no such file or directory when handling create event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": CREATE
E0921 18:42:36.611575   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": REMOVE
E0921 18:42:36.611699   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": REMOVE
E0921 18:42:52.623501   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": REMOVE
E0921 18:42:52.623535   81286 plugin_watcher.go:115] error stat file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 failed: stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8: no such file or directory when handling create event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": CREATE
E0921 18:43:24.698730   81286 plugin_watcher.go:120] error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8": REMOVE

What you expected to happen:
The Plugin Watcher should not be getting triggered by file/directory activities not related to plugin registration/unregistration. One possibility is to have the Watcher scan one directory below plugins like /var/lib/kubelet/plugins/registration or something similar

How to reproduce it (as minimally and precisely as possible):
Run plugin watcher with a CSI driver.

cc @RenaudWasTaken @vikaschoudhary16

The text was updated successfully, but these errors were encountered:

vladimirvivien · 2018-09-25T00:54:18Z

/sig node

tossmilestone · 2018-09-25T06:37:57Z

@vladimirvivien I think you should only place the plugin socket file in the plugin registration directory but not the plugin data.

tianshapjq · 2018-09-25T09:03:19Z

agree with @tossmilestone, we can only place the registration file in the specified path instead of placing all data into it. On the other hand, you can place data into the /var/lib/kubelet/plugins/registration even if registration created, that's a no-end loop.

vladimirvivien · 2018-09-25T16:11:37Z

@tossmilestone
As I mentioned in issue, existing in-tree volume plugins were already using that directory as a location to store storage operation artifacts (not just csi). Here is a shortened list of driver locations that use /var/lib/kubelet/plugins in their code already:

kubernetes/pkg/volume/fc/fc_test.go

Line 432 in bc6cb90

MountPoints: []mount.MountPoint{
kubernetes/pkg/volume/fc/fc_util.go

Line 134 in bc6cb90

// make a directory like /var/lib/kubelet/plugins/kubernetes.io/fc/target-lun-0
kubernetes/pkg/volume/iscsi/disk_manager.go

Line 41 in bc6cb90

// globalPDPath: global mount path like, /var/lib/kubelet/plugins/kubernetes.io/iscsi/{ifaceName}/{portal-some_iqn-lun-lun_id}
kubernetes/pkg/volume/iscsi/iscsi_util.go

Line 166 in bc6cb90

// make a directory like /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface_name/portal-some_iqn-lun-lun_id
kubernetes/pkg/volume/local/local.go

Line 364 in bc6cb90

// The deviceMountPath is generated to the format like :/var/lib/kubelet/plugins/kubernetes.io/local-volume/mounts/localpv.spec.Name;

And several more locations.

The fix would be to have plugin watcher scan a subdirectory of plugins (say /var/lib/kubelet/plugins/registration) instead of plugins directly as it is doing today.

The watcher functions as intended. Except now it picks up all directory activities whether related or not to plugin registration.

RenaudWasTaken · 2018-09-25T19:39:39Z

Well given the fact that the pluginwatcher is beta this is a bit of an issue. My understanding of the "beta" guarantees is that we want to be backwards compatible for all plugins published with the beta API.

Meaning that even if we move the registration directory to a new location, we'll need to keep scanning the old directory (to pick up v1beta plugins).

From a 10 000 feet point of view it seems easier to me if we move the in-tree volume plugin's data. Though it might be a lot of work.

Another less painful option, but a bit grey is the following: Enforce plugins to end with .sock this suffix is recommended for device plugins (though not enforced) and I believe is a convention on linux.
This is a bit grey because we would still be breaking v1beta users but since we're moving a convention from "recommended" to "enforced" it seems "less" breaking to me (and I haven't seen device plugins not using that extension).

What do you think @vladimirvivien ?

vladimirvivien · 2018-10-08T19:05:04Z

@RenaudWasTaken I like the .sock suffix idea. It can be a recommendation (which csi is already doing).

cpressland · 2018-10-09T06:39:45Z

Hmm, I've just deployed a new k8s cluster on 1.12.0 and noticed this repeating in the logs on a few of our pods. Just for my own sanity, is this something I should be worried about or can I ignore it until a fix is released?

Oct 09 06:37:46 westeurope-worker-09 kubelet[19149]: E1009 06:37:46.003633   19149 plugin_watcher.go:115] error dial failed at socket /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m856950613/diagnostic.data/metrics.interim.temp, err: failed to dial socket /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m856950613/diagnostic.data/metrics.interim.temp, err: context deadline exceeded when handling create event: "/var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m856950613/diagnostic.data/metrics.interim.temp": CREATE
Oct 09 06:37:46 westeurope-worker-09 kubelet[19149]: E1009 06:37:46.003717   19149 plugin_watcher.go:115] error dial failed at socket /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m856950613/diagnostic.data/metrics.interim, err: failed to dial socket /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m856950613/diagnostic.data/metrics.interim, err: context deadline exceeded when handling create event: "/var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m856950613/diagnostic.data/metrics.interim": CREATE

RenaudWasTaken · 2018-10-09T14:48:43Z

@cpressland you can ignore these errors.
@vladimirvivien if it's a recommendation then we'll keep spamming the logs with all the data files.

Another thing to keep in mind, it seems like CSI plugins puts a lot of data (and creates a lot of directories). It might be worth considering limiting the recursion to one level.

I'm concerned we might exhaust the amount of fds by tracking the plugins data.

vladimirvivien · 2018-10-10T14:35:37Z

@RenaudWasTaken yeah CSI keeps track of volume operation metadata in the plugins dir. This probably will not change.

I understand this is beta. However, besides CSI and Device plugin, are there any other implementers out there for this feature ? Would it be possible to propose a new location (i.e. /reg/?

vladimirvivien · 2018-10-18T18:58:51Z

@RenaudWasTaken let me know if you open a PR for this as discussed.

figo · 2018-10-31T23:40:01Z

@vladimirvivien thanks for open the issue with detail logs, i can see two problems by looking at logs and code.

plugin watcher trying to dial non-socket files.
too many logs generated (CREATE and DELETE).

I added a suggestion to @RenaudWasTaken fix of this issue #70494
we can fix those problems by:

check file mode when handle CREATE event, only attempt to dial file when file mode is socket.
increase log verbose level for both 'handleDeleteEvent()' and 'handleCreateEvent()'

plugin watcher still has activity when any file create/delete under the directory, but they become light weighted (CREATE file will only trigger file mode check, DELETE file event will trigger plugin searching)

i am not sure we want to go further to limit directory depth where plugin can live.
it has two drawbacks IMO:

it is hard to enforce, not sure everyone will read code/readme.
it provides less flexibility to plugins: if want to organize many plugin socket files with directory.

RenaudWasTaken · 2018-11-01T20:50:11Z

check file mode when handle CREATE event, only attempt to dial file when file mode is socket.

increase log verbose level for both 'handleDeleteEvent()' and 'handleCreateEvent()'

These are definitely good suggestions!

i am not sure we want to go further to limit directory depth where plugin can live.
it has two drawbacks IMO:

it is hard to enforce, not sure everyone will read code/readme.
it provides less flexibility to plugins: if want to organize many plugin socket files with directory.

It seems like there might be some confusion as to why we are thinking of doing this:

Most plugins use the plugin directory to store data (actually a lot of files / dirs)
Watching all the directories and files created inside the plugin directory puts more burden on the Kubelet in terms of cpu / memory consumption
Watching all the directories and files created inside the plugin directory quickly raises the number of fd consumed

By limiting the search of the plugins to one directory, we would be severely limiting the ability of the module to unintentionally harm the overall Kubelet.

It seems actually pretty easy to enforce, as it's a simple if condition. As for not everyone reading code / readme, we can add the documentation to the gRPC interface if this is a concern.

In terms of flexibility, is there a real need to create a plugin in a lower directory? Is there a plugin that has so many sockets that a single directory underneath the plugin directory isn't enough?

Naively it seems to me that $PLUGIN_DIR/vendor_name.io/socket_name-version.sock is enough for the different plugins we enable.

RenaudWasTaken · 2018-11-05T14:21:50Z

ping @figo on the above :)

figo · 2018-11-05T19:21:01Z

In terms of flexibility, is there a real need to create a plugin in a lower directory? Is there a plugin
that > has so many sockets that a single directory underneath the plugin directory isn't enough?
Naively it seems to me that $PLUGIN_DIR/vendor_name.io/socket_name-version.sock is enough for > the different plugins we enable.

@RenaudWasTaken i can not speak from plugin developer, but both below looks reasonable to me $PLUGIN_DIR/vendor_name.io/plugin_type/socket_name-version.sock, or $PLUGIN_DIR/venderName_pluginType_socketName-version.sock
as long as we can communicate to plugin developers clearly and effectively (gRPC interface documentation is needed), the directory depth restriction is reasonable to me.

vladimirvivien · 2018-11-08T20:05:54Z

@figo @RenaudWasTaken I like $PLUGIN_DIR/vendor_name.io/plugin_type/socket_name-version.sock. However, why not put plugin_type one level up as $PLUGIN_DIR/plugin_type/vendor/socket_name-version.sock. For instance $PLUGIN_DIR/csi/kubernetes/hostpath.sock

saad-ali · 2018-11-08T20:09:19Z

Is the suggestion to hard code the number of levels that the recursion algorithm will search to 3? Or just expect a socket only at a hard coded level (e.g. socket must be nested 3 deep)?

Copying my comment from #70810 (comment) for consistency:

Ideally the directory that the plugin sockets are dropped in should be different from the directory where mounts happens.

Currently the plugin socket directory is: /var/lib/kubelet/plugins/)
The mounts directory is /var/lib/kubelet/pods/[poduid]/volumes/... (so far so good).
Problem is the "global mount" directory is also /var/lib/kubelet/plugins/... :(

So options are:

Drop recursively scanning like @msau42 suggested, and say plugins will can only be directly under /var/lib/kubelet/plugins/

This will break existing plugins, for example, the GCP plugin is at /var/lib/kubelet/plugins/com.google.csi.gcepd/csi.sock (see link).

Keep recursively scanning but change the plugin socket directory to /var/lib/kubelet/plugins/sockets/?

This will also break existing plugins, like the GCP one pointed out above.

Leave the plugin dir the same, and leave recursion in, but put a limit on depth of plugin directory as suggested by @figo.

This is ugly but may be the cleanest solution.

Another option I can think of:

Keep recursively scanning but before scanning a directory check if it is mounted (mounter.IsLikelyNotMountPoint(...)). This will be backwards compatible and will prevent scanning of volume data

Thoughts?

figo · 2018-11-08T20:16:23Z

Another option I can think of:

Keep recursively scanning but before scanning a directory check if it is mounted (mounter.IsLikelyNotMountPoint(...)). This will be backwards compatible and will prevent scanning of volume data

Thoughts?

i like this idea better if it is always the case that: volume data is mounted, we can modify the watcher code to not watch the mounted directory in the first place.

vladimirvivien · 2018-11-08T20:42:05Z

Skipping mounted dir along with restructuring search path to known (shallow) dir sounds like a good plan. Though with the new suggested paths, sockets will be placed at a location where mounts should not exist.

saad-ali · 2018-11-08T20:43:27Z

i like this idea better if it is always the case that: volume data is mounted, we can modify the watcher code to not watch the mounted directory in the first place.

I think that would be a safe assumption. The plugin directory is only used as a target path for global mounts.

msau42 · 2018-11-08T20:52:45Z

Ephemeral volume types may not need to mount, it could just be a symlink or plain directory. Can we assume any ephemeral volume CSI driver won't implement global mount?

saad-ali · 2018-11-08T21:12:53Z

Ephemeral volume types may not need to mount, it could just be a symlink or plain directory. Can we assume any ephemeral volume CSI driver won't implement global mount?

Yes, in-tree ephemeral volumes today don't do a global mount so the mounts directory is /var/lib/kubelet/pods/[poduid]/volumes/... which is not the plugin dir. So no issue there.

For CSI the NodeStageVolume staging_target_path will be the global mount path (this plugin dir).
Emphermal volumes shouldn't implement NodeStageVolume call -- but nothing prevents them from doing so.

Maybe let's start with just checking for mount and ignoring mount directories. If it is still an issue we can consider adding a configurable black list of paths. Thoughts?

vladimirvivien · 2018-11-08T22:12:33Z

That sounds good. Also, putting the socket files in a specific location that does not clash with mounts will also help (i.e. $PLUGIN_DIR/csi/kubernetes/hostpath.sock)

AishSundar · 2018-11-09T17:51:18Z

/milestone v1.13
/priority important-soon

RenaudWasTaken · 2018-11-09T23:46:47Z

I just want to point out that one of the issue we have with recursively watching all directories is that it will significantly increase the amount of fd consumed.

Given that plugins actually store data there and in a dynamic way my fear is that it will exhaust the available fds for no real good reasons.

Limiting the depth sounds like a reasonable solution to that issue.

gree-gorey · 2018-11-13T10:55:02Z

btw, because of this verbosity kubelet in my cluster writes 5G logs every day and filled up my rootfs (which is small). Is there any way to decrease this verbosity?

saad-ali · 2018-11-13T22:08:02Z

Given that plugins actually store data there and in a dynamic way my fear is that it will exhaust the available fds for no real good reasons.

Limiting the depth sounds like a reasonable solution to that issue.

We can do both. Don't search mounted dirs, and max depth of search. Who is working on this?

saad-ali · 2018-11-13T22:08:31Z

@gree-gorey That's pretty bad, can you open another issue to track that.

RenaudWasTaken · 2018-11-14T00:04:02Z

@saad-ali we have a PR in flight discussing this issue (recursive scanning) here: #70494

Thinking more about this, the fact that too much logs is generated is clearly linked to this specific issue and how we process plugins.

In any case I'll be fixing that issues too

AishSundar · 2018-11-16T08:15:01Z

@RenaudWasTaken how confident are we in getting this fixed by Code freeze tomorrow? 5pm PST

RenaudWasTaken · 2018-11-16T17:57:42Z

I expect things to wind up today, changes in the above PR aren't very contentious and don't involve significant changes

Temporal workaround for kubernetes/kubernetes#69015

Temporal workaround for kubernetes/kubernetes#69015 (cherry picked from commit 6d32980)

Temporal workaround for kubernetes/kubernetes#69015

Temporal workaround for kubernetes/kubernetes#69015 (cherry picked from commit b6d1341)

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 25, 2018

k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 25, 2018

vladimirvivien mentioned this issue Oct 18, 2018

Kubelet Device Plugin Registration kubernetes/enhancements#595

Closed

RenaudWasTaken mentioned this issue Oct 23, 2018

[Umbrella issue] Plugin Watcher Work Tracker #65773

Closed

20 tasks

vladimirvivien mentioned this issue Oct 31, 2018

Move Kubelet Plugin Registration Watcher to GA #70484

Closed

6 tasks

This was referenced Oct 31, 2018

Plugin registration mechanism of CSI is broken #70485

Closed

Ignore non socket files in the kubelet plugin watcher #70494

Merged

RenaudWasTaken mentioned this issue Nov 8, 2018

Serial suite is timing out #70810

Closed

k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Nov 9, 2018

k8s-ci-robot added this to the v1.13 milestone Nov 9, 2018

gree-gorey mentioned this issue Nov 14, 2018

Excessive log generation by kubelet v1.12.2 #71031

Closed

k8s-ci-robot closed this as completed in #70494 Nov 17, 2018

liggitt mentioned this issue Nov 21, 2018

Reintroduce CSI 0.3.x support in CSI Volume Plugin #71314

Merged

hectorhuertas added a commit to utilitywarehouse/tf_kube_ignition that referenced this issue Dec 19, 2018

Increase number of watches

6d32980

Temporal workaround for kubernetes/kubernetes#69015

hectorhuertas mentioned this issue Dec 19, 2018

Increase number of watches utilitywarehouse/tf_kube_ignition#62

Merged

hectorhuertas added a commit to utilitywarehouse/tf_kube_ignition that referenced this issue Dec 19, 2018

Increase number of watches

c2ede5d

Temporal workaround for kubernetes/kubernetes#69015 (cherry picked from commit 6d32980)

hectorhuertas added a commit to utilitywarehouse/tf_kube_ignition that referenced this issue Dec 19, 2018

Increase number of watches

b6d1341

Temporal workaround for kubernetes/kubernetes#69015

hectorhuertas added a commit to utilitywarehouse/tf_kube_ignition that referenced this issue Dec 19, 2018

Increase number of watches

749c444

Temporal workaround for kubernetes/kubernetes#69015 (cherry picked from commit b6d1341)

RenaudWasTaken mentioned this issue Dec 20, 2018

Pluginwatcher backport sanity patches on 1.12 #72221

Merged

clkao mentioned this issue Feb 27, 2019

kubelet fails to start: plugin watcher fail to traverse pvc under plugin dir #74669

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plugin watcher getting notified of every Kubelet directory event #69015

Plugin watcher getting notified of every Kubelet directory event #69015

vladimirvivien commented Sep 25, 2018

vladimirvivien commented Sep 25, 2018

tossmilestone commented Sep 25, 2018

tianshapjq commented Sep 25, 2018

vladimirvivien commented Sep 25, 2018

RenaudWasTaken commented Sep 25, 2018 •

edited

Loading

vladimirvivien commented Oct 8, 2018

cpressland commented Oct 9, 2018

RenaudWasTaken commented Oct 9, 2018

vladimirvivien commented Oct 10, 2018

vladimirvivien commented Oct 18, 2018

figo commented Oct 31, 2018

RenaudWasTaken commented Nov 1, 2018

RenaudWasTaken commented Nov 5, 2018

figo commented Nov 5, 2018

vladimirvivien commented Nov 8, 2018

saad-ali commented Nov 8, 2018

figo commented Nov 8, 2018

vladimirvivien commented Nov 8, 2018

saad-ali commented Nov 8, 2018

msau42 commented Nov 8, 2018

saad-ali commented Nov 8, 2018

vladimirvivien commented Nov 8, 2018

AishSundar commented Nov 9, 2018

RenaudWasTaken commented Nov 9, 2018

gree-gorey commented Nov 13, 2018

saad-ali commented Nov 13, 2018

saad-ali commented Nov 13, 2018

RenaudWasTaken commented Nov 14, 2018 •

edited

Loading

AishSundar commented Nov 16, 2018

RenaudWasTaken commented Nov 16, 2018

Plugin watcher getting notified of every Kubelet directory event #69015

Plugin watcher getting notified of every Kubelet directory event #69015

Comments

vladimirvivien commented Sep 25, 2018

vladimirvivien commented Sep 25, 2018

tossmilestone commented Sep 25, 2018

tianshapjq commented Sep 25, 2018

vladimirvivien commented Sep 25, 2018

RenaudWasTaken commented Sep 25, 2018 • edited Loading

vladimirvivien commented Oct 8, 2018

cpressland commented Oct 9, 2018

RenaudWasTaken commented Oct 9, 2018

vladimirvivien commented Oct 10, 2018

vladimirvivien commented Oct 18, 2018

figo commented Oct 31, 2018

RenaudWasTaken commented Nov 1, 2018

RenaudWasTaken commented Nov 5, 2018

figo commented Nov 5, 2018

vladimirvivien commented Nov 8, 2018

saad-ali commented Nov 8, 2018

figo commented Nov 8, 2018

vladimirvivien commented Nov 8, 2018

saad-ali commented Nov 8, 2018

msau42 commented Nov 8, 2018

saad-ali commented Nov 8, 2018

vladimirvivien commented Nov 8, 2018

AishSundar commented Nov 9, 2018

RenaudWasTaken commented Nov 9, 2018

gree-gorey commented Nov 13, 2018

saad-ali commented Nov 13, 2018

saad-ali commented Nov 13, 2018

RenaudWasTaken commented Nov 14, 2018 • edited Loading

AishSundar commented Nov 16, 2018

RenaudWasTaken commented Nov 16, 2018

RenaudWasTaken commented Sep 25, 2018 •

edited

Loading

RenaudWasTaken commented Nov 14, 2018 •

edited

Loading