Detected file system events overflowing

[davidwindell]

Hi,

We keep seeing the below error in our ws logs, any idea of the reason?

[o.e.c.a.v.w.FileWatcherService 293] - Detected file system events overflowing

OS and version:
Che 5.18

[ghost]

@davidwindell even though it's a warning https://github.com/eclipse/che/blob/master/wsagent/che-core-api-project/src/main/java/org/eclipse/che/api/vfs/watcher/FileWatcherService.java#L293 it does look like evens may have been lost and thus filewatchers may malfunction (https://docs.oracle.com/javase/8/docs/api/java/nio/file/StandardWatchEventKinds.html#OVERFLOW)

@dkuleshov can this be causes by a huge number of files in @davidwindell's projects?

[davidwindell]

We do seem to see an awful lot of problems with file watching and as you say our projects have ~20k files

[dkuleshov]

Hi,

That is not an error but a warning, most likely caused by a big amount of file system events. Watch service (java.nio.file.WatchService) JDK implementation or underlying (OS level) services may loose or discard some amount of file system events because of being flood by them.

I'm not sure if we can do much here. In general that should not be considered as an issue unless you need to track all those flooding events but I would recommend to reduce the number of directories that you track with watch service, or don't track the most massive directory, of course if that does not break your business logic. Besides that I can assume that manual tuning of the file system or inotify kernel subsystem inside workspace agent could help. However that's just an assumption and requires further investigation.

In fact I don't think that file watcher service is really designed to effectively track such big amounts of items. If it is not necessary to track file system events in real time I would recommend to use org.eclipse.che.api.vfs.watcher.FileTreeWalker it should be more stable for such cases however with worse response time.

[davidwindell]

We've increased inotify watches here #6417

I don't really understand why the file watcher needs to watch every file in our project? Surely it should only watch the files that are actually open in the IDE. There's only ever a small number < 10 of these.

[dkuleshov]

I'm sorry, I haven't realized that we are dealing with Che version 5.19 or older.

In those versions there is a org.eclipse.che.api.vfs.impl.file.FileTreeWatcher service which is deprecated and is being removed at the moment. This service recursively tracks all project directory tree which leads to a file system event flood in some cases. If a project has massive structure, it is quiet possible that watch service may not be able to handle all events accumulated by inotify subsystem and that indeed may results in OVERFLOW kind of event.

This will be most likely fixed after spi branch is merged to master.

[davidwindell]

@dkuleshov could this be related to #6596?

[dkuleshov]

Seems like yes, however I'm concerned that there were no changes to FileTreeWatcher for many versions as it is deprecated but you spotted the issue only on 5.18.

[davidwindell]

I'm going to go ahead and close this as 5.19 is out. Will re-open if we see any further overflows.