New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The listen gem breaks my laptop #26158

Open
benaubin opened this Issue Aug 14, 2016 · 36 comments

Comments

Projects
None yet
@benaubin

benaubin commented Aug 14, 2016

Steps to reproduce

  1. Create a new rails app
  2. Run it on a Mac with a bad processor

Expected behavior

My laptop should work fine - processes should stay under the 700-something limit that Apple set.

Actual behavior

When running my Rails app on a Macbook Air with 1.8 ghz of cpu (I know... getting a new computer when they release the new Pro's), I have been getting a "System ran out of resources, can't fork any processes" error in all of my applications (or, they just quietly failed). Finally, I opened Activity Monitor and was able to find more then 500 fsevent_watch processes running (my computer has a maximum of 700 processes total). Doing a Google Search, I stumbled upon the [rb-fsevent gem]. Looking at my Gemfile.lock, I was able to track it down to the listen gem.

I'm unable to tell if it is an issue with the listen gem itself, or @fnx's implementation in Rails (see #22254), but it seems like it's more likely the implementation is broken.

Suggested Fix

I'm unable to tell if it is an issue with the listen gem itself, or @fnx's implementation in Rails (see #22254), but it seems like it's more likely the implementation is broken.

I would make sure that the listen gem is opt-in, especially on systems with low process limits (there's a command that allows you to check this, although I forget what it is).

Temporary Workaround

If your computer is currently showing a "System ran out of resources, can't fork any processes" error, reboot it (logging in and out might work, but I don't feel like testing that). You probably won't be able to save most of your work (if your software forks a process in order to save).

First, removed the listen and spring-watcher-listen gems.
Then, in your environments/development.rb file, comment out this line:

# Use an evented file watcher to asynchronously detect changes in source code,
# routes, locales, etc. This feature depends on the listen gem.
# config.file_watcher = ActiveSupport::EventedFileUpdateChecker

System configuration

Rails version:
Rails 5.0.0
Ruby version:
ruby 2.3.0p0

@vipulnsward

This comment has been minimized.

Show comment
Hide comment
@vipulnsward

vipulnsward Aug 15, 2016

Member

Can you create a sample application replicating this for debugging?

Member

vipulnsward commented Aug 15, 2016

Can you create a sample application replicating this for debugging?

@amree

This comment has been minimized.

Show comment
Hide comment
@amree

amree Aug 21, 2016

Just want to chip in.

I'm probably having the same issue. I noticed my fsevent_watch can go up to more than 500 and I can't start new process after that. I can confirm it happens whenever I close my laptop without turning off rails server. So, when I opened my laptop and refresh the localhost, new fsevent_watch somehow will be started even though they already exists. Killing rails server will only kill the new fsevent_watch, not those from before I close my laptop. So, opening and closing the laptop will gradually increase fsevent_watch processes.

System Configuration

Rails version:
Rails 5.0.0.1
Ruby Version
2.3.1p112

amree commented Aug 21, 2016

Just want to chip in.

I'm probably having the same issue. I noticed my fsevent_watch can go up to more than 500 and I can't start new process after that. I can confirm it happens whenever I close my laptop without turning off rails server. So, when I opened my laptop and refresh the localhost, new fsevent_watch somehow will be started even though they already exists. Killing rails server will only kill the new fsevent_watch, not those from before I close my laptop. So, opening and closing the laptop will gradually increase fsevent_watch processes.

System Configuration

Rails version:
Rails 5.0.0.1
Ruby Version
2.3.1p112

@benaubin

This comment has been minimized.

Show comment
Hide comment
@benaubin

benaubin Aug 21, 2016

@vipulnsward Yes. In fact, there's a command to do so:

rails new thisbreaksmylaptop

benaubin commented Aug 21, 2016

@vipulnsward Yes. In fact, there's a command to do so:

rails new thisbreaksmylaptop

@cbeer cbeer referenced this issue Aug 25, 2016

Closed

Turn listen off? #62

@rvirani1

This comment has been minimized.

Show comment
Hide comment
@rvirani1

rvirani1 Sep 12, 2016

All 3 devs on our team are having this issue as well.

rvirani1 commented Sep 12, 2016

All 3 devs on our team are having this issue as well.

@benaubin benaubin changed the title from The spring-watcher-listen gem breaks my laptop to The listen gem breaks my laptop Sep 14, 2016

@benaubin

This comment has been minimized.

Show comment
Hide comment
@benaubin

benaubin Sep 14, 2016

Ok, I came up with a workaround. Looks like it's not only the spring-watch-listen gem, and while removing it helps slightly (I think it might have just been a placebo), it doesn't fully solve the problem.

First I removed the listen and spring-watcher-listen gems.
Then, in my environments/development.rb file, I commented out this line:

 # Use an evented file watcher to asynchronously detect changes in source code,
 # routes, locales, etc. This feature depends on the listen gem.
 # config.file_watcher = ActiveSupport::EventedFileUpdateChecker

Then, Rails seems to work fine.

I very much recommend making using the listen gem opt-in, not opt-out, as it's very hard to debug. Also, laptop breaking probably turns many developers away from rails (it almost did for me).

The listen gem was added in #22254.

benaubin commented Sep 14, 2016

Ok, I came up with a workaround. Looks like it's not only the spring-watch-listen gem, and while removing it helps slightly (I think it might have just been a placebo), it doesn't fully solve the problem.

First I removed the listen and spring-watcher-listen gems.
Then, in my environments/development.rb file, I commented out this line:

 # Use an evented file watcher to asynchronously detect changes in source code,
 # routes, locales, etc. This feature depends on the listen gem.
 # config.file_watcher = ActiveSupport::EventedFileUpdateChecker

Then, Rails seems to work fine.

I very much recommend making using the listen gem opt-in, not opt-out, as it's very hard to debug. Also, laptop breaking probably turns many developers away from rails (it almost did for me).

The listen gem was added in #22254.

@benaubin

This comment has been minimized.

Show comment
Hide comment
@benaubin

benaubin Sep 14, 2016

I'm tagging @fnx, as it looks the listen implementation in Rails which he wrote is what contributes to this, not just the spring-watcher-listen gem.

benaubin commented Sep 14, 2016

I'm tagging @fnx, as it looks the listen implementation in Rails which he wrote is what contributes to this, not just the spring-watcher-listen gem.

fizz added a commit to fizz/junket that referenced this issue Sep 23, 2016

fizz added a commit to fizz/junket that referenced this issue Sep 25, 2016

@metacortex

This comment has been minimized.

Show comment
Hide comment
@metacortex

metacortex Sep 26, 2016

+1

Now I'm killing manually.
pkill -9 fsevent_wat*

metacortex commented Sep 26, 2016

+1

Now I'm killing manually.
pkill -9 fsevent_wat*

@ahoward

This comment has been minimized.

Show comment
Hide comment
@ahoward

ahoward Sep 28, 2016

Contributor

i'm seeing this too. it is way too easy to orphan processes. the listen gem is a piece of shit and should be removed.

Contributor

ahoward commented Sep 28, 2016

i'm seeing this too. it is way too easy to orphan processes. the listen gem is a piece of shit and should be removed.

@ahoward

This comment has been minimized.

Show comment
Hide comment
@ahoward

This comment has been minimized.

Show comment
Hide comment
@ahoward

ahoward Sep 28, 2016

Contributor

the guys that are working on that gem know next to nothing about process management. a disaster.

Contributor

ahoward commented Sep 28, 2016

the guys that are working on that gem know next to nothing about process management. a disaster.

@tenderlove

This comment has been minimized.

Show comment
Hide comment
@tenderlove

tenderlove Sep 28, 2016

Member

@ahoward it might not be great, but lets try to stay civil. 🙇

Member

tenderlove commented Sep 28, 2016

@ahoward it might not be great, but lets try to stay civil. 🙇

@benaubin

This comment has been minimized.

Show comment
Hide comment
@benaubin

benaubin Sep 29, 2016

@tenderlove Plan on removing listen as a default?

benaubin commented Sep 29, 2016

@tenderlove Plan on removing listen as a default?

@benaubin benaubin closed this Sep 29, 2016

@benaubin benaubin reopened this Sep 29, 2016

@tenderlove

This comment has been minimized.

Show comment
Hide comment
@tenderlove

tenderlove Sep 29, 2016

Member

@penne12 I don't think so. We need to either fix listen, or find an alternative. If we can find a suitable alternative then we'd remove listen as default. This seems like it might have potential

Member

tenderlove commented Sep 29, 2016

@penne12 I don't think so. We need to either fix listen, or find an alternative. If we can find a suitable alternative then we'd remove listen as default. This seems like it might have potential

@benaubin

This comment has been minimized.

Show comment
Hide comment
@benaubin

benaubin Sep 29, 2016

@tenderlove I believe active support has something, and while it's not ideal, it doesn't break people's laptops. Literally, it took me almost a week to be able to figure out how to use Rails. Almost completely changed frameworks, even. Rails works fine (esp in development) without listen.

benaubin commented Sep 29, 2016

@tenderlove I believe active support has something, and while it's not ideal, it doesn't break people's laptops. Literally, it took me almost a week to be able to figure out how to use Rails. Almost completely changed frameworks, even. Rails works fine (esp in development) without listen.

@simi

This comment has been minimized.

Show comment
Hide comment
@e2

This comment has been minimized.

Show comment
Hide comment
@e2

e2 Sep 29, 2016

Contributor

Hi everyone, Listen maintainer here...

TL;DR - skip to end..

First, I haven't used a Mac for over 5 years. I only use Linux, so unless I can reproduce a problem on the Mac - at least on Travis (OSX builds), I can't fix it.

I probably can't even reasonably run OSX in a Virtual Machine (AFAIK), so if I can't reproduce it or write a unit/acceptance test for it, I won't even know how something is broke.

Especially if nothing is ever reported.

Otherwise, I do release bugfixed-versions of Listen VERY quickly.

With that said ...

Listen supports multiple backends. rb-fsevent is the problem child here.

Basically, most of Listen is a giant workaround for OSX-related issues - by design. So surprising issues on OSX are ... not surprising. And they're often OSX version-dependent.

You can blame Apple for a lack of decent file/dir monitoring.

One solution I'd suggest is replacing the current Darwin backend with fs-event listener from https://github.com/burke/zeus.

I'd do that myself, but 1) I don't have time nowadays (but PRs are encouraged!) 2) I wouldn't be able to test it, since I don't own a Mac.

One thing that Listen does that's special for OSX is: it runs each watched directory in a separate thread. (This was mostly to allow mapping a file back to a given watcher). I don't know if that's the issue here, but I'd be glad to fix+release. There are acceptance tests for OSX on Travis, so if you can reproduce it somehow, it can likely be added to the build to be resolved forever.

There are lots of hideous details (many related to OSX), so feel free to ask, open issues, etc.

As for alternatives I once researched and created a list of "watch-and-do-something" tools here: https://github.com/guard/guard/wiki/Guard-alternatives

Meanwhile, there's polling mode, which sounds bad, but actually may perform much better than expected.

One good/bad thing about Listen is that it's "directory based" and not "file based" (like many other file watching implementations). This means Listen has better support of removing/adding whole trees of files and editor support (especially useful for Rails users).

The downside is: it needs to scan whole trees of changes on OSX, because the current backend has no hints about which files were changed. And HFS filesystem has 1-second resolution, which makes things worse. Especially when combined with file upper/lower case OSX-specific issues.

I really considered facebook/watchman, but it didn't solve all the problems that Listen was designed to solve (but it was pretty close). I also considered the listeners from Zeus, but again, I don't have/use/want OSX. (I only use Linux).

TL;DR

OSX file monitoring is (was?) shitty. It also was the most problematic and often the slowest.

I doubt those things can be fixed by "a gem" instead of fixing OSX.

Trying to switch implementations is like trading one can of "known" worms with another can of "unknown" worms.

My best bet would be to transplant this:

https://github.com/burke/zeus/blob/master/go/filemonitor/filemonitor_darwin.go or this: https://github.com/burke/zeus/blob/v0.15.5/ext/fsevents/main.m to replace this: https://github.com/guard/listen/blob/master/lib/listen/adapter/darwin.rb

Feel free to mention me (@e2) if I can help/advise/etc.

Contributor

e2 commented Sep 29, 2016

Hi everyone, Listen maintainer here...

TL;DR - skip to end..

First, I haven't used a Mac for over 5 years. I only use Linux, so unless I can reproduce a problem on the Mac - at least on Travis (OSX builds), I can't fix it.

I probably can't even reasonably run OSX in a Virtual Machine (AFAIK), so if I can't reproduce it or write a unit/acceptance test for it, I won't even know how something is broke.

Especially if nothing is ever reported.

Otherwise, I do release bugfixed-versions of Listen VERY quickly.

With that said ...

Listen supports multiple backends. rb-fsevent is the problem child here.

Basically, most of Listen is a giant workaround for OSX-related issues - by design. So surprising issues on OSX are ... not surprising. And they're often OSX version-dependent.

You can blame Apple for a lack of decent file/dir monitoring.

One solution I'd suggest is replacing the current Darwin backend with fs-event listener from https://github.com/burke/zeus.

I'd do that myself, but 1) I don't have time nowadays (but PRs are encouraged!) 2) I wouldn't be able to test it, since I don't own a Mac.

One thing that Listen does that's special for OSX is: it runs each watched directory in a separate thread. (This was mostly to allow mapping a file back to a given watcher). I don't know if that's the issue here, but I'd be glad to fix+release. There are acceptance tests for OSX on Travis, so if you can reproduce it somehow, it can likely be added to the build to be resolved forever.

There are lots of hideous details (many related to OSX), so feel free to ask, open issues, etc.

As for alternatives I once researched and created a list of "watch-and-do-something" tools here: https://github.com/guard/guard/wiki/Guard-alternatives

Meanwhile, there's polling mode, which sounds bad, but actually may perform much better than expected.

One good/bad thing about Listen is that it's "directory based" and not "file based" (like many other file watching implementations). This means Listen has better support of removing/adding whole trees of files and editor support (especially useful for Rails users).

The downside is: it needs to scan whole trees of changes on OSX, because the current backend has no hints about which files were changed. And HFS filesystem has 1-second resolution, which makes things worse. Especially when combined with file upper/lower case OSX-specific issues.

I really considered facebook/watchman, but it didn't solve all the problems that Listen was designed to solve (but it was pretty close). I also considered the listeners from Zeus, but again, I don't have/use/want OSX. (I only use Linux).

TL;DR

OSX file monitoring is (was?) shitty. It also was the most problematic and often the slowest.

I doubt those things can be fixed by "a gem" instead of fixing OSX.

Trying to switch implementations is like trading one can of "known" worms with another can of "unknown" worms.

My best bet would be to transplant this:

https://github.com/burke/zeus/blob/master/go/filemonitor/filemonitor_darwin.go or this: https://github.com/burke/zeus/blob/v0.15.5/ext/fsevents/main.m to replace this: https://github.com/guard/listen/blob/master/lib/listen/adapter/darwin.rb

Feel free to mention me (@e2) if I can help/advise/etc.

@e2

This comment has been minimized.

Show comment
Hide comment
@e2

e2 Sep 29, 2016

Contributor

One thing: please don't spread groundless FUD about Listen. (At least first make sure Listen really is the culprit).

TL;DR - saying that Listen is shitty because of this issue (without knowing the root cause here) is like saying Rails is shitty because of a Listen issue. It only obscures the problem and discourages maintainers from putting in more personal time to try and improve things.

@ahoward - while personally I'm not offended by your comments, they're just misleading. Listen is just a layer of workarounds that makes file monitoring portable, so if this really was a Listen issue, it would likely be reproduceable on Linux, Windows and BSD. If it isn't, it likely isn't a Listen issue.

Also, Listen works as reliably as the backends let it, e.g. rb-fsevent - which I'm not even a contributor of.

Sure, Listen does need another overhaul (for 4.x) - but that's mostly unrelated to this.

Ironically though, Listen's acceptance tests are the best way to test backends like rb-fsevent. So if there's something messed up, I can maybe just replicate it in Listen and detect bad scenarios and blacklist certain gem version combinations (which I've done before).

(But I'd need some way to turn it into a Travis tests - e.g. with shell code to create the problem and detect orphaned processes).

My point: removing Listen would just mean that some poor contributors would have to waste hours to reinvent the same fixes/workarounds that Listen already has.

So it's not constructive to create a movement of "lets drop Listen". Simply because it's not the root problem here. Listen is more like an API interface, so dropping it wouldn't make sense unless the API was bad.

And just to give a sense:

Both Listen and the backend drivers started as hacks as early as 2010: https://github.com/thibaudgg/rb-fsevent/commits/d803637f19d3d1450105b5e0d4f8ed1060f98acb/lib/rb-fsevent/fsevent.rb - and there's a long history of OSX nightmares that brave contributors have been trying to debug and patch ever since.

And many original contributors have simply moved on with their lives. If anyone wants to make the code "less shitty" - there are probably no other obstacles than being unable to contact the maintainers.

I'm not the maintainer of rb-fsevent, but I'd be glad to add a better OSX backend implementation in Listen. Or even just bumping the Listen dependency on rb-fsevent (if a fix becomes available).

I know it's easy to jump to conclusions when we're frustrated - sometimes it's better to just reach out and ask first. If something works shitty, there may be a bigger reason other than "incompetence".

Sorry for the length - just felt like a wrong I had to fix.

Contributor

e2 commented Sep 29, 2016

One thing: please don't spread groundless FUD about Listen. (At least first make sure Listen really is the culprit).

TL;DR - saying that Listen is shitty because of this issue (without knowing the root cause here) is like saying Rails is shitty because of a Listen issue. It only obscures the problem and discourages maintainers from putting in more personal time to try and improve things.

@ahoward - while personally I'm not offended by your comments, they're just misleading. Listen is just a layer of workarounds that makes file monitoring portable, so if this really was a Listen issue, it would likely be reproduceable on Linux, Windows and BSD. If it isn't, it likely isn't a Listen issue.

Also, Listen works as reliably as the backends let it, e.g. rb-fsevent - which I'm not even a contributor of.

Sure, Listen does need another overhaul (for 4.x) - but that's mostly unrelated to this.

Ironically though, Listen's acceptance tests are the best way to test backends like rb-fsevent. So if there's something messed up, I can maybe just replicate it in Listen and detect bad scenarios and blacklist certain gem version combinations (which I've done before).

(But I'd need some way to turn it into a Travis tests - e.g. with shell code to create the problem and detect orphaned processes).

My point: removing Listen would just mean that some poor contributors would have to waste hours to reinvent the same fixes/workarounds that Listen already has.

So it's not constructive to create a movement of "lets drop Listen". Simply because it's not the root problem here. Listen is more like an API interface, so dropping it wouldn't make sense unless the API was bad.

And just to give a sense:

Both Listen and the backend drivers started as hacks as early as 2010: https://github.com/thibaudgg/rb-fsevent/commits/d803637f19d3d1450105b5e0d4f8ed1060f98acb/lib/rb-fsevent/fsevent.rb - and there's a long history of OSX nightmares that brave contributors have been trying to debug and patch ever since.

And many original contributors have simply moved on with their lives. If anyone wants to make the code "less shitty" - there are probably no other obstacles than being unable to contact the maintainers.

I'm not the maintainer of rb-fsevent, but I'd be glad to add a better OSX backend implementation in Listen. Or even just bumping the Listen dependency on rb-fsevent (if a fix becomes available).

I know it's easy to jump to conclusions when we're frustrated - sometimes it's better to just reach out and ask first. If something works shitty, there may be a bigger reason other than "incompetence".

Sorry for the length - just felt like a wrong I had to fix.

@rails rails locked and limited conversation to collaborators Sep 29, 2016

@matthewd

This comment has been minimized.

Show comment
Hide comment
@matthewd

matthewd Sep 29, 2016

Member

@tenderlove already addressed the tone of the previous feedback; I don't think it's productive to continue discussing same.

saying that Listen is shitty because of this issue (without knowing the root cause here) is like saying Rails is shitty because of a Listen issue

For the record: Rails is shitty because of this issue, regardless of which gem is the root cause -- and we will consequently explore any reasonable option to make the problem go away.

Member

matthewd commented Sep 29, 2016

@tenderlove already addressed the tone of the previous feedback; I don't think it's productive to continue discussing same.

saying that Listen is shitty because of this issue (without knowing the root cause here) is like saying Rails is shitty because of a Listen issue

For the record: Rails is shitty because of this issue, regardless of which gem is the root cause -- and we will consequently explore any reasonable option to make the problem go away.

derekprior added a commit to derekprior/dotfiles that referenced this issue Apr 20, 2017

Disable Spring
My current working theory is that spring is worsening the affect of the
[Listen bug] so I'm going to try this for a few days.

[Listen bug]: rails/rails#26158
@derekprior

This comment has been minimized.

Show comment
Hide comment
@derekprior

derekprior May 24, 2017

Contributor

I have disabled spring system wide (DISABLE_SPRING=1 in my profile) and haven't run into this issue since. It's been weeks now. This specifically seems related to the interplay between spring and listen/rb-fsevent/fseventsd

Contributor

derekprior commented May 24, 2017

I have disabled spring system wide (DISABLE_SPRING=1 in my profile) and haven't run into this issue since. It's been weeks now. This specifically seems related to the interplay between spring and listen/rb-fsevent/fseventsd

@benaubin

This comment has been minimized.

Show comment
Hide comment
@benaubin

benaubin May 25, 2017

I'm just going to put it out there, removing all references to listen in the Gemfile fixed the problem on my computer. I recently got a new laptop with an i7, so I've haven't noticed any more problems. I think this leads me to believe there's a problem with how one of the gems kills off stale processes.

benaubin commented May 25, 2017

I'm just going to put it out there, removing all references to listen in the Gemfile fixed the problem on my computer. I recently got a new laptop with an i7, so I've haven't noticed any more problems. I think this leads me to believe there's a problem with how one of the gems kills off stale processes.

@benaubin

This comment has been minimized.

Show comment
Hide comment
@benaubin

benaubin May 25, 2017

To respond to @tenderlove's comment about finding a different solution, I found that how rails did file watching (or maybe skipped it completely) in previous versions was perfect & worked with a small config change. I don't know much about process management, but it seems the killing of processes that fb-fsevent does isn't great (see @ahoward's link to the code).

benaubin commented May 25, 2017

To respond to @tenderlove's comment about finding a different solution, I found that how rails did file watching (or maybe skipped it completely) in previous versions was perfect & worked with a small config change. I don't know much about process management, but it seems the killing of processes that fb-fsevent does isn't great (see @ahoward's link to the code).

@benaubin

This comment has been minimized.

Show comment
Hide comment
@benaubin

benaubin May 25, 2017

Sorry I haven't replied in so long - the issue's been locked so it's been hard to elaborate on the problems I saw.

benaubin commented May 25, 2017

Sorry I haven't replied in so long - the issue's been locked so it's been hard to elaborate on the problems I saw.

shikendon added a commit to denpaio/denpaio that referenced this issue Jun 5, 2017

@thibaudgg

This comment has been minimized.

Show comment
Hide comment
@thibaudgg

thibaudgg Jun 12, 2017

Contributor

Hi @benaubin, we're discussing a complete rewrite of rb-fsevent that could potentially fix your issue.

I would also recommend upgrading to the latest macOS release (Sierra) as Apple fix some scary bugs around fsevents.

Contributor

thibaudgg commented Jun 12, 2017

Hi @benaubin, we're discussing a complete rewrite of rb-fsevent that could potentially fix your issue.

I would also recommend upgrading to the latest macOS release (Sierra) as Apple fix some scary bugs around fsevents.

@ttilley

This comment has been minimized.

Show comment
Hide comment
@ttilley

ttilley Jun 13, 2017

@benaubin above and beyond the rewrite, I understand that the current version of rb-fsevent isn't going away for quite some time and have made quite a few large changes in the latest version on master branch. It's not released as a gem yet mostly so I can add a post install message mentioning the new gem... which doesn't exist yet. I should probably just release and not wait. One of the fixes included makes sure the fsevent_watch doesn't stick around longer than its parent process in the rare case it does... thus wasting resources, especially for people who don't shut down their laptops and simply put them to sleep. EDIT: This change was not mine, and made it into the code, but doesn't appear to have made it into a pre-compiled binary release... and thus has potentially never been used. Ouch.

Like... the changes are technically too large to just force on people as-is possibly... It's 100% backwards compatible but the communication protocol is completely different and the callback now accepts 2 parameters instead of just 1, with the second including some seriously detailed metadata about each and every change (like telling you ahead of time, without having to check yourself, whether the change refers to a directory, file, hard link... and whether the change was simply metadata, which can be safely ignored by most use cases of something like Listen... the API provides this data for free since 10.10 and we've just been ignoring it until now).

If you can upgrade to sierra, please do. There's a massive filesystem metadata corruption bug that completely breaks fsevents and had to be worked around in the most brittle and broken way possible. The code implementing the workaround legitimately scares me. We have to replace the system implementation of realpath() at runtime and it's seriously bad juju in every way.

ttilley commented Jun 13, 2017

@benaubin above and beyond the rewrite, I understand that the current version of rb-fsevent isn't going away for quite some time and have made quite a few large changes in the latest version on master branch. It's not released as a gem yet mostly so I can add a post install message mentioning the new gem... which doesn't exist yet. I should probably just release and not wait. One of the fixes included makes sure the fsevent_watch doesn't stick around longer than its parent process in the rare case it does... thus wasting resources, especially for people who don't shut down their laptops and simply put them to sleep. EDIT: This change was not mine, and made it into the code, but doesn't appear to have made it into a pre-compiled binary release... and thus has potentially never been used. Ouch.

Like... the changes are technically too large to just force on people as-is possibly... It's 100% backwards compatible but the communication protocol is completely different and the callback now accepts 2 parameters instead of just 1, with the second including some seriously detailed metadata about each and every change (like telling you ahead of time, without having to check yourself, whether the change refers to a directory, file, hard link... and whether the change was simply metadata, which can be safely ignored by most use cases of something like Listen... the API provides this data for free since 10.10 and we've just been ignoring it until now).

If you can upgrade to sierra, please do. There's a massive filesystem metadata corruption bug that completely breaks fsevents and had to be worked around in the most brittle and broken way possible. The code implementing the workaround legitimately scares me. We have to replace the system implementation of realpath() at runtime and it's seriously bad juju in every way.

@ttilley

This comment has been minimized.

Show comment
Hide comment
@ttilley

ttilley Jun 13, 2017

I would also agree that monitoring file events on MacOS has been miserable, at least until 10.12's MASSIVE HFS+ metadata corruption fix, and that for end users some of that frustration is actually my fault with some unrelated bugs in rb-fsevent. Most of those issues should be resolved now, with the last being fully resolved in the rewrite (such as the ability to ignore filesystem events originating from your own process, which will be very useful).

ttilley commented Jun 13, 2017

I would also agree that monitoring file events on MacOS has been miserable, at least until 10.12's MASSIVE HFS+ metadata corruption fix, and that for end users some of that frustration is actually my fault with some unrelated bugs in rb-fsevent. Most of those issues should be resolved now, with the last being fully resolved in the rewrite (such as the ability to ignore filesystem events originating from your own process, which will be very useful).

@cben

This comment has been minimized.

Show comment
Hide comment
@cben

cben Jun 14, 2017

Contributor

Some mitigation for people using the polling FileUpdateChecker, if your slowness is mostly due to tons of asset requests on page reload:

Assets have fingerprinted urls and cachable headers so most requests are just "revalidation"s getting a 304. However, rails scans FS for code changes before returning the 304.
But why revalidate when max-age is far in future? Turns out browsers tended to do that on user-
initiated Reload.

Using fresh Chromium on a project with ~250 asserts, I saw page reloads become up to 20% faster than I ever got with EventedFileUpdateChecker before.

P.S. With a browser that did make 250 requests, for me simply rate-limiting the FS scans (assume it remains fresh for say 100ms after last scan) helped a lot, page reload within 20% of EventedFileUpdateChecker.
https://gist.github.com/cben/006e86d4788f98bb6c70b91e4e909af8
Should I make a proper PR out of that? Can someone try it out and confirm it helps?

Contributor

cben commented Jun 14, 2017

Some mitigation for people using the polling FileUpdateChecker, if your slowness is mostly due to tons of asset requests on page reload:

Assets have fingerprinted urls and cachable headers so most requests are just "revalidation"s getting a 304. However, rails scans FS for code changes before returning the 304.
But why revalidate when max-age is far in future? Turns out browsers tended to do that on user-
initiated Reload.

Using fresh Chromium on a project with ~250 asserts, I saw page reloads become up to 20% faster than I ever got with EventedFileUpdateChecker before.

P.S. With a browser that did make 250 requests, for me simply rate-limiting the FS scans (assume it remains fresh for say 100ms after last scan) helped a lot, page reload within 20% of EventedFileUpdateChecker.
https://gist.github.com/cben/006e86d4788f98bb6c70b91e4e909af8
Should I make a proper PR out of that? Can someone try it out and confirm it helps?

mrwillihog pushed a commit to birdrapp/birdr-api that referenced this issue Jun 21, 2017

Matthew Williams
Temporarily disable listen / spring
There is a bug in Mac OS which forks loads of
processes when using these gems. This means
that I have had to disable the auto-reloading
feature until it is fixed. See:

rails/rails#26158

kbaribeau added a commit to kbaribeau/dotfiles that referenced this issue Aug 3, 2017

Disable spring system wide, use rbenv over rvm
I don't even remember what prompted my change to rvm. I think I was
having problems with installing a new version of ruby?

Also, spring is causing problems. See
rails/rails#26158
@benaubin

This comment has been minimized.

Show comment
Hide comment
@benaubin

benaubin Aug 18, 2017

Glad to see progress is being made :happy:!

I hope the new filesystem in High Sierra will help.

Some of these commits we're seeing have some interesting workarounds. I particularly like this one by @denpaio.

@cben not quite sure how your thing relates to this issue? Might be best to create a real issue or PR so it gets its own attention from collaborators.

@thibaudgg what happened with the rewrite? Did you end up deciding to go that route?

benaubin commented Aug 18, 2017

Glad to see progress is being made :happy:!

I hope the new filesystem in High Sierra will help.

Some of these commits we're seeing have some interesting workarounds. I particularly like this one by @denpaio.

@cben not quite sure how your thing relates to this issue? Might be best to create a real issue or PR so it gets its own attention from collaborators.

@thibaudgg what happened with the rewrite? Did you end up deciding to go that route?

@vipulnsward

This comment has been minimized.

Show comment
Hide comment
@vipulnsward

vipulnsward Aug 21, 2017

Member

FYI: I myself face this issue now 😕

Member

vipulnsward commented Aug 21, 2017

FYI: I myself face this issue now 😕

@padi

This comment has been minimized.

Show comment
Hide comment
@padi

padi Aug 25, 2017

Contributor

It's been months that I encountered this issue and it's only now that I found out that it's related to my new Rails project. :| I <3 Rails but I reckon this can be a dealbreaker for new people coming into Rails.

With that said, I'm not knowledgeable enough yet to fix it, but here's a workaround that doesn't have to disable listen/rb-fsevent/spring altogether: we can throttle down applications (including Rails-related processes) in OSX using renice:

Fire and forget way:

$ sudo su -
$ `while true; do renice +15 -p `ps ax | grep 'rb-fsevent' | grep -v grep | awk '{print $1}' | tr 'n' ' '`; done`

... or a command line alternative: make an executable renice-rails in your $PATH with the following content:

#!/bin/sh
# filename: renice-rails
renice +15 -p `ps ax | grep 'rb-fsevent' | grep -v grep | awk '{print $1}' | tr 'n' ' '`

I quickly put up a 1 minute blog about it https://padi.github.io/articles/throttle-down-rails-processes/

Contributor

padi commented Aug 25, 2017

It's been months that I encountered this issue and it's only now that I found out that it's related to my new Rails project. :| I <3 Rails but I reckon this can be a dealbreaker for new people coming into Rails.

With that said, I'm not knowledgeable enough yet to fix it, but here's a workaround that doesn't have to disable listen/rb-fsevent/spring altogether: we can throttle down applications (including Rails-related processes) in OSX using renice:

Fire and forget way:

$ sudo su -
$ `while true; do renice +15 -p `ps ax | grep 'rb-fsevent' | grep -v grep | awk '{print $1}' | tr 'n' ' '`; done`

... or a command line alternative: make an executable renice-rails in your $PATH with the following content:

#!/bin/sh
# filename: renice-rails
renice +15 -p `ps ax | grep 'rb-fsevent' | grep -v grep | awk '{print $1}' | tr 'n' ' '`

I quickly put up a 1 minute blog about it https://padi.github.io/articles/throttle-down-rails-processes/

@MatthiasWinkelmann

This comment has been minimized.

Show comment
Hide comment
@MatthiasWinkelmann

MatthiasWinkelmann Oct 5, 2017

Contributor

As a note that may help some experiencing problems: I noticed that the rails file watcher was watching the project root, not just directories with source files below it.

I happened to have a data directory with a few million small image files within a few hundred thousand nested directories. I experienced extreme slowdowns across the board: rails console would take a minute to start, and any model instantiation would take 10 seconds or so.

I tried to dive into the source and find a way to configure evented_file_watcher, but my attempts with Rails.application.config.watched_dirs didn't make a difference. As far as I can tell, Rails adds /app, /lib, etc. separately, but I believe http://www.github.com/rails//rails/activesupport/lib/active_support/evented_file_update_checker.rb#L51 then tries to be smart and combines into their common ancestor.

The listen gem prevents the worst by filtering a set of known prefixes in https://github.com/guard/listen/blob/master/lib/listen/silencer.rb, where you could 'hide' non-source data. Since I didn't feel like having actual data in /tmp, and I couldn't find a way to easily configure this behaviour, I moved the data out of the project tree.

Contributor

MatthiasWinkelmann commented Oct 5, 2017

As a note that may help some experiencing problems: I noticed that the rails file watcher was watching the project root, not just directories with source files below it.

I happened to have a data directory with a few million small image files within a few hundred thousand nested directories. I experienced extreme slowdowns across the board: rails console would take a minute to start, and any model instantiation would take 10 seconds or so.

I tried to dive into the source and find a way to configure evented_file_watcher, but my attempts with Rails.application.config.watched_dirs didn't make a difference. As far as I can tell, Rails adds /app, /lib, etc. separately, but I believe http://www.github.com/rails//rails/activesupport/lib/active_support/evented_file_update_checker.rb#L51 then tries to be smart and combines into their common ancestor.

The listen gem prevents the worst by filtering a set of known prefixes in https://github.com/guard/listen/blob/master/lib/listen/silencer.rb, where you could 'hide' non-source data. Since I didn't feel like having actual data in /tmp, and I couldn't find a way to easily configure this behaviour, I moved the data out of the project tree.

@rails-bot

This comment has been minimized.

Show comment
Hide comment
@rails-bot

rails-bot bot Jan 3, 2018

This issue has been automatically marked as stale because it has not been commented on for at least three months.
The resources of the Rails team are limited, and so we are asking for your help.
If you can still reproduce this error on the 5-1-stable branch or on master, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

rails-bot bot commented Jan 3, 2018

This issue has been automatically marked as stale because it has not been commented on for at least three months.
The resources of the Rails team are limited, and so we are asking for your help.
If you can still reproduce this error on the 5-1-stable branch or on master, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

@benaubin

This comment has been minimized.

Show comment
Hide comment
@benaubin

benaubin Jan 3, 2018

Still a problem!

benaubin commented Jan 3, 2018

Still a problem!

@rails-bot rails-bot bot removed the stale label Jan 3, 2018

@thibaudgg

This comment has been minimized.

Show comment
Hide comment
@thibaudgg

thibaudgg Mar 2, 2018

Contributor

@benaubin could you please give a try to thibaudgg/rb-fsevent#81, that could maybe solve it.

Contributor

thibaudgg commented Mar 2, 2018

@benaubin could you please give a try to thibaudgg/rb-fsevent#81, that could maybe solve it.

@thibaudgg

This comment has been minimized.

Show comment
Hide comment
@thibaudgg

thibaudgg Mar 3, 2018

Contributor

@here rb-fsevent 0.10.3 is out with a potential fix for this issue.

Contributor

thibaudgg commented Mar 3, 2018

@here rb-fsevent 0.10.3 is out with a potential fix for this issue.

@salex

This comment has been minimized.

Show comment
Hide comment
@salex

salex Apr 22, 2018

After spending most of the day trying to figure out why jQuery was not loading on a demo app I found this page. I was running in development mode with nginx proxy_pass and tried a bunch of nginx configs, none of which solved the problem. I then discovered the same code ran fine an a RaspberryPI but failed on a MacMini! Then I remembered the error when I started the Rails server as a daemon, which led me to this page.

developer$ rails s -p 8603 -b 0.0.0.0 -d
=> Booting Puma
=> Rails 5.1.6 application starting in development 
=> Run `rails server -h` for more startup options
me:icash developer$ 
FATAL: prematurely zombied 

I upgraded to 10.12.6 (from .5) and jQuery is now running, but I still got the FATAL error. I'll assume the server startup process got a little further. The rails version is using rb-fsevent 0.10.3 so it was not a fix.

The server command works on my laptop (air) but fails on the MacMini, which has server software installed but not used. Not sure where to go except High Sierra.

salex commented Apr 22, 2018

After spending most of the day trying to figure out why jQuery was not loading on a demo app I found this page. I was running in development mode with nginx proxy_pass and tried a bunch of nginx configs, none of which solved the problem. I then discovered the same code ran fine an a RaspberryPI but failed on a MacMini! Then I remembered the error when I started the Rails server as a daemon, which led me to this page.

developer$ rails s -p 8603 -b 0.0.0.0 -d
=> Booting Puma
=> Rails 5.1.6 application starting in development 
=> Run `rails server -h` for more startup options
me:icash developer$ 
FATAL: prematurely zombied 

I upgraded to 10.12.6 (from .5) and jQuery is now running, but I still got the FATAL error. I'll assume the server startup process got a little further. The rails version is using rb-fsevent 0.10.3 so it was not a fix.

The server command works on my laptop (air) but fails on the MacMini, which has server software installed but not used. Not sure where to go except High Sierra.

@rails-bot

This comment has been minimized.

Show comment
Hide comment
@rails-bot

rails-bot bot Jul 22, 2018

This issue has been automatically marked as stale because it has not been commented on for at least three months.
The resources of the Rails team are limited, and so we are asking for your help.
If you can still reproduce this error on the 5-2-stable branch or on master, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

rails-bot bot commented Jul 22, 2018

This issue has been automatically marked as stale because it has not been commented on for at least three months.
The resources of the Rails team are limited, and so we are asking for your help.
If you can still reproduce this error on the 5-2-stable branch or on master, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

@rails-bot rails-bot bot closed this Jul 29, 2018

@guilleiguaran guilleiguaran reopened this Aug 16, 2018

@rails-bot rails-bot bot removed the stale label Aug 16, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment