Rimraf chokes on AFS volumes (ENOTEMPTY throttling)

[msegado]

The current strategy of trying to rmdir first and only using readdir on ENOTEMPTY (see

rimraf/rimraf.js

Line 213 in 096b163

// try to rmdir first, and only readdir on ENOTEMPTY or EEXIST (SunOS)

) chokes badly when used on AFS volumes.

By default, it seems AFS fileservers will throttle single clients that make more than 10 failed RPCs in a row by delaying a full 3 seconds (to protect the server from poorly-written clients). Unsurprisingly, this slows rimraf to a crawl, with fairly wide implications for Node projects on AFS volumes given rimraf's popularity. As an example, here's part of an strace log from npm uninstall on my machine (note the timestamps...):

...
18327 17:00:52 rmdir("/path/to/home/myproject/node_modules/gobble/node_modules/minimatch" <unfinished ...>
18318 17:00:52 read(8, "\1\0\0\0\0\0\0\0", 1024) = 8
18318 17:00:52 epoll_wait(5,  <unfinished ...>
18325 17:00:55 <... rmdir resumed> )    = -1 ENOTEMPTY (Directory not empty)
18325 17:00:55 write(8, "\1\0\0\0\0\0\0\0", 8) = 8
18325 17:00:55 rmdir("/path/to/home/myproject/node_modules/gobble/node_modules/mkdirp" <unfinished ...>
18318 17:00:55 <... epoll_wait resumed> {{EPOLLIN, {u32=8, u64=8}}}, 1024, -1) = 1
18318 17:00:55 read(8, "\1\0\0\0\0\0\0\0", 1024) = 8
18318 17:00:55 epoll_wait(5,  <unfinished ...>
18328 17:00:58 <... rmdir resumed> )    = -1 ENOTEMPTY (Directory not empty)
18328 17:00:58 write(8, "\1\0\0\0\0\0\0\0", 8) = 8
18318 17:00:58 <... epoll_wait resumed> {{EPOLLIN, {u32=8, u64=8}}}, 1024, -1) = 1
18328 17:00:58 rmdir("/path/to/home/myproject/node_modules/gobble/node_modules/promise-map-series" <unfinished ...>
18318 17:00:58 read(8, "\1\0\0\0\0\0\0\0", 1024) = 8
18318 17:00:58 epoll_wait(5,  <unfinished ...>
18326 17:01:01 <... rmdir resumed> )    = -1 ENOTEMPTY (Directory not empty)
18326 17:01:01 write(8, "\1\0\0\0\0\0\0\0", 8) = 8
18318 17:01:01 <... epoll_wait resumed> {{EPOLLIN, {u32=8, u64=8}}}, 1024, -1) = 1
...

Of course, it would be ideal if the OS itself could deal with this and not hit the server for each of those requests as is already done for some commands like mkdir -- see http://milek.blogspot.com/2014/01/mkdir-performance.html -- though I'm not familiar enough with the ins and outs of filesystems to know if it's possible to do that for rmdir.

Thoughts?

[msegado]

Incidentally, what disadvantages are there to spawning an rm -rf or rmdir /s /q child process when possible (@isaacs hesitantly brought up the latter in #72)? I don't know the performance implications myself...

[isaacs]

Can you please retry with the latest version of rimraf? I believe some recent changes have likely affected this. Happy to reopen if explicit throttling is still something to consider.