-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FileSystemWatcher tests fail intermittently on Unix due to too many open inotify instances #16208
Comments
By default the number of inotify instances allowed is relatively low (128), and we use one to back FileSystemWatcher. If the machine on which the tests are running happens to be doing other things at the same time that creates inotify instances, we could bump up against this. @mmitche, can we up the default limits on the machines used in CI? I believe this can be done by adding a line like the following to the end of the /etc/sysctl.conf file:
|
@stephentoub Yeah....does that need to be updated on all of the *nix machines? |
Thanks, Matt. Only the Linux ones. |
@stephentoub This is being done on the new dynamically allocated image |
Thanks. |
Ran in to this again here:
And here:
|
@mmitche, is it possible that multiple corefx test runs might be running on the same CI machine at the same time? The error message highlights that the larger limit is being used, but we're still hitting it, which implies to me either a) we're not properly disposing some of these, b) multiple jobs are running concurrently on the same machine and some of our tests that push these values to the limit are causing problems for other test processes, or c) something else on the machine is using a ton of inotify instances. cc: @sokket |
@stephentoub Except for OSX, no. |
@ianhays can you take a look since you're already working on FSW related test issues? |
In dotnet/corefx#8231 I added code to guard against failing when a test hits the inotify limit, so the FSW tests should be fine. It's still possible to hit the limit but it's significantly less likely now, especially in an innerloop run. I'm closing this as (presumably) resolved for now. |
It makes sense that we're hitting this limit again on Outerloop runs. If we're hitting it on innerloop then something else is wrong. We're hitting it on Outerloop runs because the CreateManyConcurrent test is creating new watchers up to the inotify limit to verify the exception that we throw in that case. The problem is that we run tests in parallel and if another test tries to make a new FSW during or shortly after the CreateManyConcurrent test then it will unexpectedly fail by no fault of its own. This will be resolved by dotnet/corefx#11721 |
Yes, this was an Outerloop run. |
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/centos7.1_release_tst_prtest/1094/console
The text was updated successfully, but these errors were encountered: