Exclude root directory in file watcher(s)

Hi!

We've got an issue while using the Build runner package. We run on Linux 64bit and when executing an script containing the execution of the build runner "watch" method we get a Operating system error:
OS Error: No space left on device, errno = 28

The output of the watch method is within the same project folder, in the main disk partition, with lots of remaining space. I can add that if we run the script just after the computer is re-started, it works, what makes us think that is a temporary space/files related issue. But if we open some other applications like the IDE, we get the erro. In addition as the error we get is an Operating System error, we think that it's like the system is wrongly believing that there is no space left somewhere.

I give you more details about the issue. This is the script containing the build runner watch method that we run:

And this is the Error we get:

And this is the status of the computer disks (main disk is the one with 240GB of space, at the 33%):

Versions details:

-Dart VM version: 2.0.0-dev.63.0 (Unknown timestamp) on "linux_x64"
-Running on Linux 64 bit
-Distro: Linux quick 4.13.0-45-generic dart-archive/watcher#50~16.04.1-Ubuntu SMP Wed May 30 11:18:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
-Using Chrome (but not relevant)
-build_runner: ^0.8.10. Full list of dependencies:

[jakemac53]

Some of those dependencies definitely do use temp directories - but they should be cleaning them up.

Can you look at your temp directory and see if you have anything obvious there? I would check specifically for directories with build or scratch_space in the name.

Hi @jakemac53 !

Thanks a lot for your quick reply. I didn't find anything related to those terms in the tmp folder:

Regarding the dependencies using the temp directories, provably yes. The thing here is if the build runner watch command uses the tmp. Do you think so? I think it shouldn't, as only compiles the code from one place to the other.

thanks!

[jakemac53]

@rental-ninja if you look in those other directories that are just a random sequence of 10 characters, what do you see in them? What about the ~*.tmp directories?

[natebosch]

I think we're hitting ulimit errors adding watchers. We do have an issue to add a better error message for this but haven't implemented anything yet.

#867

The reason you get these while your IDE is running is because it also uses a bunch of file watchers.

I think it shouldn't, as only compiles the code from one place to the other.

We need to use a temp directory to make best effort at hermetic builds whenever we run a process that will be reading files. These temp directories are most likely not causing issues, the problem is the watcher limits.

Thank you both for your replies, always very helpful! Yesterday I was taking a deep view into both options you suggested, particularly to @natebosch one. I was taking a look at this post. I could not solve it but I will continue investigating. In the meantime, if you have any other idea, like another option to solve the ulimit error, share it!

See you!

[refi64]

I just ran into this issue because of my node_modules folder, which may or may not have...uhh...91422 files...

In the mean time, is there a way to tell build_runner to skip watching certain directories?

[jakemac53]

Unfortunately the watcher api doesn't let you exclude a specific directory, so we currently watch the entire root package and then filter out the file events for things that don't match your globs.

We do need to watch some files at the top level directory... although we could potentially use a regular file watcher for those and a directory watcher for subdirectories. This would be a breaking change to our apis though and would complicate things a bit, although there would be some other benefits (not having to filter out events under .dart_tool for instance).

[matanlurey]

+1 for the breaking change @jakemac53 outlines. I think it could help lowering resource CPUs too.

[natebosch]

I think this has come up in other issues but I don't have them on hand - I want to take note of one of the difficulties we've discussed.

If we have don't have a director watcher at the root and someone adds a new directory we would care about, we won't know about it and won't be able to add the watcher on it.

[jakemac53]

What if we moved the .dart_tool/build/generated directory to some other place and just had a symlink to it sort of like bazel-out? That might solve a lot of issues.