New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FileSystemWatcher may cause problems in containers - inotify limits and ASP.NET Core #3475

Closed
shanselman opened this Issue Aug 29, 2018 · 4 comments

Comments

Projects
None yet
4 participants
@shanselman

shanselman commented Aug 29, 2018

Looking around the web I'm seeing years of issues with FileSystemWatcher saying "The configured user limit (n) on the number of inotify instances has been reached."

UPDATE: Looks like https://github.com/dotnet/corefx/blob/a10890f4ffe0fadf090c922578ba0e606ebdd16c/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs#L371 will assume when inotify_add_watch fails with an ENOSPEC it must but an issue with inotify instances being out of range. In fact, ENOSPEC can also mean "the kernel failed to allocate a needed resource."

From the Man Page - The user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource

Phrased differently. There's two Error Cases and we throw a message that implies there's just One.

This is becoming more prevalent in container situations in constrained sandboxes. I'm trying to deploy https://github.com/shanselman/superzeit (just clone and "now --public" or run locally with docker) to Zeit.co and I'm hitting this regularly. I don't think I'm hitting a limit. I think Zeit (and others) are blocking the syscall.

I think there are two issues here:

1 We should return a different error message if inotify_add_watch fails, and then circuit break so that FileSystemWatcher doesn't prevent the app from starting. If we CAN startup without a watch successfully, we should.

2 It seems DOTNET_USE_POLLING_FILE_WATCHER=1 is used in dotnet-watch and the aspnet file providers but the base System.IO FileSystemWatcher class doesn't support DOTNET_USE_POLLING_FILE_WATCHER? We should probably be consistent.

- If I change reloadOnChange: false in Program.cs to bypass the first watch that is set on AppSettings.json, I end up hitting it later when Razor/MVC sets up its FileWatchers. 

We need at a minimum, to have DOTNET_USE_POLLING_FILE_WATCHER respected everywhere. Another idea would be for a way to have FileSystemWatcher "fail gracefully." We need to test on systems with

Unhandled Exception: System.IO.IOException: The configured user limit (8192) on the number of inotify instances has been reached.
> [0]    at System.IO.FileSystemWatcher.StartRaisingEvents()
> [0]    at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
> [0]    at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
> [0]    at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
> [0]    at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
> [0]    at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
> [0]    at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__0_0()
> [0]    at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
> [0]    at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
> [0]    at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
> [0]    at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
> [0]    at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
> [0]    at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
> [0]    at superzeit.Program.Main(String[] args) in /app/superzeit/Program.cs:line 17

Related issues?

@stephentoub @natemcmaster @muratg @pranavkm

dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   2.1.401
 Commit:    91b1c13032

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.401\

Host (useful for support):
  Version: 2.1.3-servicing-26724-03
  Commit:  124038c13e

.NET Core SDKs installed:
  2.1.400 [C:\Program Files\dotnet\sdk]
  2.1.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3-servicing-26724-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

@spboyer

This comment has been minimized.

Show comment
Hide comment
@spboyer

spboyer commented Aug 29, 2018

TryEnableFileSystemWatcher() - gets called regardless if DOTNET_USE_POLLING_FILE_WATCHER is set or not.

https://github.com/idefa/AspnetCoreHome/blob/master/FileSystem-rel-2.0.0/src/Microsoft.Extensions.FileProviders.Physical/PhysicalFilesWatcher.cs#L93

@sebastienros

This comment has been minimized.

Show comment
Hide comment
@sebastienros

sebastienros Aug 29, 2018

Member

I think this PR should fix your problem: aspnet/FileSystem#333
And it's solving this: aspnet/Mvc#8173

/cc @pranavkm

Member

sebastienros commented Aug 29, 2018

I think this PR should fix your problem: aspnet/FileSystem#333
And it's solving this: aspnet/Mvc#8173

/cc @pranavkm

@shanselman

This comment has been minimized.

Show comment
Hide comment
@shanselman

shanselman Aug 29, 2018

@sebastienros Thanks. I think that just makes watching these files "more correct" and "more efficient."

I'm digging into this. Our code looks right.

shanselman commented Aug 29, 2018

@sebastienros Thanks. I think that just makes watching these files "more correct" and "more efficient."

I'm digging into this. Our code looks right.

@shanselman shanselman changed the title from FileSystemWatcher causes problems in containers - inotify limits and ASP.NET Core to FileSystemWatcher may cause problems in containers - inotify limits and ASP.NET Core Aug 29, 2018

@shanselman

This comment has been minimized.

Show comment
Hide comment
@shanselman

shanselman commented Aug 29, 2018

@shanselman shanselman closed this Aug 29, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment