Analyzer hang on Linux due to out of file watchers

[davidmorgan]

On Linux file watchers are a limited resource with some arbitrary (system config dependent!) fixed limit.

If there are no more file watchers left, dart analyze can hang forever.

The problem is that analyzer code in context_manager.dart assumes a watcher failure is temporary and tries again, but it will not succeed until more watchers become available. So, it needs to give up at some point.

The permanent failure can be identified by the error message, "(OS Error: No space left on device, errno = 28)", yes, Linux reports "not enough watchers" as "no space left", which has been confusing people for a very long time :)

To repro, first exhaust your file watchers. This is system dependent, on mine I can do

sudo sysctl -w fs.inotify.max_user_watches=0

to set the max number of watchers to zero.

Then, running

dart analyze lib pubspec.yaml

hangs. (Other invocations don't hang, not sure exactly why, I guess a different number of watchers is created).

The number of watches should reset on reboot, or you can change it back, on my system the old value was about 4000000

sudo sysctl -w fs.inotify.max_user_watches=4000000

[srawlins]

Great find, @davidmorgan ! Can you say briefly, whether there might be anything to do in the watcher package?

You filed the issue here and called out context_manager.dart, so I assume the fix should be here. But just to be explicit: there is nothing to do in the watcher package, short of... perhaps implemening its own retry logic?

[srawlins]

@bwilkerson @keertip WDYT about tracking number of watchers here or there? If it is possible, I'm thinking of reporting it in the Insights pages. And/or reporting it in analytics. On linux we might be able to run a sysctl command (or cat?) to read what the current limit is.

[bwilkerson]

I'm not sure what data knowing the number of watchers will give us, given that they can be configured differently from user to user, but I have no problem adding data to the insights pages. If it proves to not be useful we can always remove it.

What might be more interesting is reporting via analytics (and the insight pages?) the number of times we ran out of watchers when trying to set them up. If enough users are impacted that might suggest that we need a better mechanism for watching the disk (or maybe just a fall-back for when we can't use watchers). If nothing else, it will tell us how often there's a risk of server getting out of sync with reality.

[keertip]

+1 to report when we run out of watchers during setup. Would be good to know how widespread this is and how many of our users are affected.

[sigurdm]

Great find, @davidmorgan ! Can you say briefly, whether there might be anything to do in the watcher package?

I wonder if creating a watcher could fail with a well defined and documented exception, such that error handling could be improved, and error messages would be understandable.

[davidmorgan]

I don't think the watcher package can do much better, but I'll keep this one in mind for bigger improvements / possible rewrite :)

VSCode has notification when you run out of watchers, which is how we discovered this issue: Sigurd saw the VSCode notification at the same time as the hang. A bit of info on VSCode watching here: https://github.com/microsoft/vscode/wiki/File-Watcher-Issues "Specifically on Linux, watching a large folder recursively can result in VSCode consuming too many file handles. If that is the case, you will see a warning notification with instructions how to solve that."

This probably is not very common as it would break other things too and other tools would give warnings. I guess, we should join the club and warn too instead of hang :)

[davidmorgan]

Great find, @davidmorgan ! Can you say briefly, whether there might be anything to do in the watcher package?

I wonder if creating a watcher could fail with a well defined and documented exception, such that error handling could be improved, and error messages would be understandable.

Thanks. I thought of adding an exception type, but what we currently throw is the same as what the SDK throws for the same specific error, which seems better. So I think I'll just document it as such.

[sigurdm]

I thought of adding an exception type, but what we currently throw is the same as what the SDK throws for the same specific error, which seems better.

I don't know if watcher-exhaustion exists on mac-os and windows. But it would be nice to smooth out the differences in the watcher package, such that error handling could be unified...

[davidmorgan]

I thought of adding an exception type, but what we currently throw is the same as what the SDK throws for the same specific error, which seems better.

I don't know if watcher-exhaustion exists on mac-os and windows. But it would be nice to smooth out the differences in the watcher package, such that error handling could be unified...

There's no equivalent problem on Mac or Windows, so this is the only exception in the "permanent startup failure" category :)

[keertip]

@davidmorgan , I am unable to repro the problem on my linux setup. Used the command as mentioned by you to set the available file watchers to 0. Then tried dart analyze, it does not hang. In VS Code, I do see the message Unable to watch for file changes and then a message saying it could not start the language server. So can confirm there are no file watchers available.

I am running a locally built sdk and analyzing open source code.