This repository has been archived by the owner. It is now read-only.

fs.watch: node[52551] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21) #5463

Closed
shama opened this Issue May 13, 2013 · 70 comments

Comments

Projects
None yet
@shama

shama commented May 13, 2013

I'm getting an error which appears to be directly from libuv while using fs.watch. I'm on OSX 10.8.3 and using node 0.10.5. My ulimit -n is 10480.

I'm able to duplicate the error with the following example (it is a bit contrived but it simulates what is happening with our gruntjs/grunt-contrib-watch module). It only happens when you're watching a larger number of files and another process tries to watch the same files.

I've setup an example in a repo for duplicating the error:

git clone git://github.com/shama/fs-watch-error && cd fs-watch-error
npm install
npm test

Here is the code that duplicates the issue:

var fs = require('fs');
var glob = require('glob');
var limit = 9999;
glob('**/*', function(err, files) {
  if (files.length > limit) files = files.slice(0, limit);
  console.log('watching ' + files.length + ' files...');
  files.forEach(function(file) {
    fs.watch(file, function() {});
  });
});

Then run node index.js & node index.js on a large number of files and you should get:

2013-05-13 11:39 node[54911] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)

I'm looking into updating our module to avoid this error but I'm reporting it here because I believe it shouldn't error out to the console in that way. Considering that I don't seem to be able to catch that error and handle it more appropriately.

Thanks!

@bnoordhuis

This comment has been minimized.

Member

bnoordhuis commented May 16, 2013

Sorry, can't reproduce. FWIW, it doesn't look like a libuv error to me but rather it's Carbon complaining about failing internal bookkeeping.

/cc @indutny

@shama

This comment has been minimized.

shama commented May 20, 2013

Thanks for trying @bnoordhuis. This is a bit over my head which is why it's an issue not a PR but I'll try looking into it more soon. Good excuse for me to learn. :)

@kevindente

This comment has been minimized.

kevindente commented Jun 26, 2013

I'm seeing this problem as well. Only seems to happen with Node 0.10 - I can't reproduce with 0.8.

@bnoordhuis

This comment has been minimized.

Member

bnoordhuis commented Jun 26, 2013

v0.8 doesn't use FSEvents.

FWIW, I still think this is an environmental issue rather than a node.js bug but I'll leave the report open for now.

@pho3nixf1re

This comment has been minimized.

pho3nixf1re commented Jun 28, 2013

I'm getting this same issue on OSX 10.8 with node 0.10.5 using grunt-watch-contrib to watch a large number of files.

@carlos8f

This comment has been minimized.

carlos8f commented Jul 1, 2013

👍

@jamorton

This comment has been minimized.

jamorton commented Jul 17, 2013

Same issue as @pho3nixf1re.
I also get "Fatal error: EMFILE: Too many opened files." if I keep the watcher open for several hours.

@shama

This comment has been minimized.

shama commented Jul 17, 2013

@Jonanin EMFILE is a whole different issue. The default max file descriptor limit for OSX is low 256. You can increase it with temporarily with ulimit -n 1200.

FWIW, the easiest way to avoid this error is watch less files because it only happens (for me at least) when watching a large amount of files. Tighten up your globbing patterns. If you're doing **/* then you're probably watching files in your node_modules folder which isn't necessary. Either structure your app different or add '!**/node_modules/**' to your patterns.

@medikoo

This comment has been minimized.

medikoo commented Aug 9, 2013

On OSX I see it in WebStorm console (when I run command that uses a lot of watchers), but I don't see it when I run same command in OSX terminal.

@indutny

This comment has been minimized.

Member

indutny commented Aug 11, 2013

Can you please try running disk repair from Disk Utility app? Let me know if this issue is still reproducible after the repair.

@shama

This comment has been minimized.

shama commented Aug 11, 2013

Still reproducible for me on my MBP:
screen shot 2013-08-11 at 12 29 25 pm

Also on my older macbook:
screen shot 2013-08-11 at 12 35 25 pm

Thank you for looking into this @indutny!

@indutny

This comment has been minimized.

Member

indutny commented Aug 11, 2013

I've debugged it up to mach_msg() call which receives some custom-struct which contains that -21 error code.

@indutny

This comment has been minimized.

Member

indutny commented Aug 11, 2013

Apparently, you can watch only 451 files at the same time using FSEvents API. Not sure if its a joke from Apple, or something else... Will try to figure out more soon.

@indutny

This comment has been minimized.

Member

indutny commented Aug 21, 2013

This libuv update should fix it joyent/libuv#894.

@shama

This comment has been minimized.

shama commented Aug 21, 2013

Awesome! Thank you @indutny!

@indutny

This comment has been minimized.

Member

indutny commented Aug 22, 2013

Landed in libuv's master: joyent/libuv@cd2794c .

Backporting it to v0.10 before merging into node v0.10: joyent/libuv#896

n1mmy referenced this issue in joyent/libuv Oct 14, 2013

unix: revert recent FSEvent changes
This commit reverts the following commits:

    983fa68 darwin: fix 10.6 build error in fsevents.c
    684e212 fsevents: use shared FSEventStream
    ea4cb77 fsevents: FSEvents is most likely not thread-safe
    9bae606 darwin: create fsevents thread on demand

Several people have reported stability issues on OS X 10.8 and bus
errors on the 10.9 developer preview.

See also nodejs/node-v0.x-archive#6296 and nodejs/node-v0.x-archive#6251.
@thebigredgeek

This comment has been minimized.

thebigredgeek commented Feb 4, 2014

Is this fixed?

@alexgorbatchev

This comment has been minimized.

alexgorbatchev commented Feb 4, 2014

Seeing this on 0.10.25

2014-02-04 10:21 node[80486] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2014-02-04 10:21 node[80486] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2014-02-04 10:21 node[80486] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)
2014-02-04 10:21 node[80486] (CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)


Darwin alex.local 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64
@ziggythehamster

This comment has been minimized.

ziggythehamster commented Dec 3, 2014

+1 for me:

keith@Keiths-iMac ~ $ uname -a
Darwin Keiths-iMac.local 13.4.0 Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64

keith@Keiths-iMac ~ $ node --version
v0.10.33

keith@Keiths-iMac ~ $ system_profiler SPSoftwareDataType SPDeveloperToolsDataType
Software:

    System Software Overview:

      System Version: OS X 10.9.5 (13F34)
      Kernel Version: Darwin 13.4.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      Computer Name: Keith’s iMac
      User Name: Keith Gable (keith)
      Secure Virtual Memory: Enabled
      Time since boot: 3:54

Developer:

    Developer Information:

      Version: 6.1.1 (6A2008a)
      Location: /Applications/Xcode.app
      Applications:
          Xcode: 6.1.1 (6611)
          Instruments: 6.1 (56160)
      SDKs:
          OS X:
              10.10: (14A382)
              10.9: (13F26)
          iOS:
              8.1: (12B411)
          iOS Simulator:
              8.1: (12B411)
@ziggythehamster

This comment has been minimized.

ziggythehamster commented Dec 3, 2014

This Sublime Text 2 configuration snippet seems to have helped:

"folder_exclude_patterns": [".svn", ".git", ".hg", "CVS", "tmp/class-*", "tmp/es_*", "tmp/jshinter*", "tmp/replace_*", "tmp/static_compiler*", "tmp/template_compiler*", "tmp/tree_merger*", "tmp/coffee_script*", "tmp/concat-tmp*", "tmp/export_tree*", "tmp/sass_compiler*"]
@consideRatio

This comment has been minimized.

consideRatio commented Dec 18, 2014

what @ziggythehamster did with excluding folders in a sublime project helped me as well for my Ember-cli app. When i was running a ember-cli dev-server who as well as sublime also watches files i got this error, but it was solved when i using sublime configurations removed heavy project folders like a tmp folder where massive changes would occur quickly in the file system.

So +1 for using the "folder_exclude_patterns": ["foldername1"] setting in sublime

@kaldrenon

This comment has been minimized.

kaldrenon commented Jan 16, 2015

This does not appear to be fixed in v0.11.14

@Samstiles

This comment has been minimized.

Samstiles commented Jan 30, 2015

Still occuring for me as well

@jclif

This comment has been minimized.

jclif commented Feb 23, 2015

I'm getting this error quite often. I believe it's being caused by gulp-watch. Also, the error message is sometimes inserted into vim.

@rushkeldon

This comment has been minimized.

rushkeldon commented Mar 24, 2015

We have been struggling with these errors on a project for a while. I don't think it is the number of files that are being watched - I think this error comes from trying to register a watch on the same file more than once. So, make sure your blobs don't overlap in subsequent calls to watch.

Also, if you are knowingly calling watch on the same file(s) it is good to know that gulp.watch returns a watcher that will relay the event every time, so you can chain a '.on' block like so :

GOOD :

 gulp.watch( constants.APP_TEMPLATES, [ 'appTemplates' ] )
    .on( 'change', function( event ){ logChangedFile( event ); }  );

as opposed to the wrong way that was generating about 250 of these errors every time I ran Gulp.

BAD :

gulp.watch( constants.APP_TEMPLATES, logChangedFile );
gulp.watch( constants.APP_TEMPLATES, [ 'appTemplates' ] );
@Sup3rDave

This comment has been minimized.

Sup3rDave commented Aug 18, 2015

I still get the same issues in v0.12, pretty frustrating to deal with.

@ziggythehamster

This comment has been minimized.

ziggythehamster commented Aug 18, 2015

I'm not 100% sure the problem is Node's fault. I fixed the problem by adding my Broccoli build directory to Sublime Text's and Visual Studio Code's ignore list. I think OS X just has a ridiculously crappy file system watcher.

@nmschulte

This comment has been minimized.

nmschulte commented Sep 17, 2015

I'm experiencing this issue with recent versions of Node and npm on OS X machinery.

On OS X 10.10.4 (14E46):

$ node --version
v0.10.35
$ npm --version
2.13.5

On OS X 10.10.5:

$ node -v
v0.12.7
$ npm -v
2.14.3

On my machine running GNU/Linux (Debian Sid; Linux 4.x), these issues don't exist:

$ node --version
v0.12.4
$ npm --version
2.14.1

This makes cross-environment/platform development a real pain: webpack's watching implementation doesn't always pick up on changes in Linux (possibly more), and the work-around is to use the "old" watching plugin, which suffers from these issues. It's an un-winnable game!

@Sup3rDave

This comment has been minimized.

Sup3rDave commented Sep 17, 2015

I ultimately found that my issue was Sublime contributing to watching too many files on OSX's file system. If I shut down Sublime I was immediately able to watch with no issue, then I could start Sublime up again and continue. Not the greatest solution admittedly, but it works at least.

@nmschulte

This comment has been minimized.

nmschulte commented Sep 17, 2015

@Sup3rDave, so, is this simply an incompatibility issue with OS X? If so, it's still an issue that ought to be dealt with, even if that simply means training OS X users how their OS works... (e.g. ulimit or w/e resource management issue is getting in the way).

Were you able to track what about OS X is causing this?

@Sup3rDave

This comment has been minimized.

Sup3rDave commented Sep 17, 2015

@nmschulte no, I was never able to figure out specifically what it was about OSX other than I guess some issue with its FSEvent stuff which apparently Gulp/Sublime(and probably other things) lean on for watching. I was mid project when I found the workaround, so I was more concerned with getting past it anyway :)

@nmschulte

This comment has been minimized.

nmschulte commented Sep 17, 2015

@Sup3rDave, thanks for the followup.

Well, it seems that this issue will not ever see the light of day, given the recent change of repositories to nodejs/node, and that this issue is closed here in the archive repository. I'm going to confirm the issue still exists with the latest versions of things on OS X, and if so create a new issue on the new repository with the new information and referencing this old one.

@ziggythehamster

This comment has been minimized.

ziggythehamster commented Sep 17, 2015

I think the problem is that there's a finite number of FSEvent watchers you can have among all apps using CarbonCore, and Sublime and others aren't very polite about not using all of the watchers. I added my Ember CLI temp and build directories to Sublime's ignore list and it solved my issue. I think the problem would be resolved by having Node use Cocoa's file system watcher, but that sounds like a huge pain in the ass.

@nmschulte

This comment has been minimized.

nmschulte commented Sep 17, 2015

Carbon Core is deprecated; sounds like Node.js should switch to use Cocoa, regardless of how much of a pain it is. I have to imagine this effort is already underway, considering I don't imagine Node.js plans to drop support for OS X...

https://developer.apple.com/library/mac/releasenotes/General/CarbonCoreDeprecations/index.html#//apple_ref/doc/uid/TP40012224

That said, I'm not entirely certain node is using the deprecated Carbon Core API, or at least perhaps not with my specific incantation of this issue:

https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/FSEvents_ProgGuide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40005289

(FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)

@ziggythehamster

This comment has been minimized.

ziggythehamster commented Sep 17, 2015

I think they're using FSEvents, and FSEvents internally uses CarbonCore. Apple: Do as I say, not as I do, right? :)

I mean, iTunes was/is Carbon64 (something they never released for anyone else) after all.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.