-
Notifications
You must be signed in to change notification settings - Fork 246
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Suppress errors on local symlinks #339
Comments
The idea of physically watching the same directory twice seems wrong, no E.g you can move So what's important here is the EXACT scenario, because I'm more than OSX support is the key impediment here - if everyone used only Linux and In short - you can blame Apple, or provide a good case that cannot be I'd really suggest monkey-patching here instead of polluting the The fact that we're having this discussion can save you and others many Especially with spring - it's best to solve the problem (or make a Overall I simply refuse to support warning disabling on OSX itself - it's I'm also pretty sure the spring Listen watcher doesn't need to watch the Just saying I can't proceed here without a justified, concrete use case. |
Thanks for the quick turnaround, I'll follow up with However, I'm not sure I understood other parts of your response, so please bear with me a bit:
|
Yes, OSX is why Listen is so complex, and I don't have OSX myself to even try to mitigate the issues. Basically, Listen can gravely underperform on OSX given complex large projects, and symlinks can reduce the performance exponentially. I've written about this is some of the replies in other issues. There's an open issue (or PR) for optimizing on OSX a bit, but it needs a major rewrite of dome areas of Listen (think filesystem snapshots in memory). |
As for the use case, I need something more specific - because I can't imagine a scenario where listening to a directory with symlinks is impossible to avoid. Maybe there's a better project setup I can suggest in you scenario - or if that's not possible, I'll have to consider the drawbacks of whatever changes are needed in Listen. |
By good IT practices, I meant that the OSX adapter is pretty resource hungry - and symlinks make things worse. I've never encountered a need for symlinks in a project, that weren't actually just a workaround or a symptom of a bigger issue. For me, I wouldn't want to have to care about how you setup your project - it should be up to you. But, Apple's poor design decisions are not something I can change - e.g. having a log file in the root of your rails project can eat your cpu on OSX - and I cant really do anything about that. Also, not every Listen user is a software developer - I prefer to help people avoid problems instead of silencing warnings that people could mindlessly enable, and then complain about high cpu usage without even providing a repo or logs as to give me a hint as to what could be wrong. |
To clarify, the issue here has nothing to do with Listen actually watching the same directory twice via symlink, or anything related to OSX's potential issues in that scenario. I think #273 does well enough by skipping any directory about to be watched a second time. This issue is that the error messages are unnecessary when this behavior is intended. Unless I'm misunderstanding something, the symlinked directory can be silently skipped without any further problems in the case of a (common and harmless) local symlink. At the end of the day, the symlinked directory will be watched only once, change notifications will be sent once for each real directory's path, and OSX CPUs will never start screaming in agony. For my specific case, I finally managed to set up a workaround in spring-watcher-listen#8 by using the workaround you suggested (move
You've never found a good use for symlinks, ever? Well I certainly have, and in my experience I would never leave $HOME without them 😄 Thanks for all your help so far! |
Just to clarify from my side: #273 is about detecting (filesystem loop and locally symlinked watched dirs) - it doesn't skip symlinked directories (that's not implemented). If you're getting warnings, it means the physical directory is already being watched (
Watching the same physical directory twice doesn't make any sense, ever. In fact, Facebook's watchman actually has a client/server model to prevent this (in a more elegant way - and there are more benefits too).
Yes, except that's not implemented - mostly because symlinks can be created/moved/deleted while listening and internally Listen is already too complex without such extensive symlink support. There's a huge cost in implementing, testing and maintaining this - and I can't afford to cover those costs from my own pocket. I'd be happy to accept PRs though. Ideally, symlinks should be treated as files and not followed - but many users intuitively expect stuff inside symlinks to be watched, so it's really unclear which scenario should be supported. There's just a whole universe of edge cases out there when you take symlinks into account (e.g. symlinks to dirs changing to symlinks to files, symlinked dirs being moved, handling broken symlinks, etc.).
In a project, meaning "project-local symlinks" in the context of listening. Either you listen to edited files (so symlinks are not needed, because those files are the project and not something external) ... or ... you symlink files from other projects (and thus, you don't need to listen for them to changes, because changes to them happen through updating manifests and external dependency releases). That's why I asked about your use case. E.g. if your assets are not in 'app/assets' but in 'vendor/assets' and you're symlinking - that's a project configuration problem. What I mean is that the warnings are a lot more beneficial than any exception I can think of. In fact the filesystem loop was a result of a project which had such a filesystem loop - even tools like the Unix If there are any explanations in the Wiki which would've helped - let me know (or add missing hints yourself, because I'm too familiar with Listen internals to explain things clearly to everyone else). |
ahh, I think I understand better now. I assumed that the SymlinkDetector was acting as a filter that was actively preventing directories from being watched twice, and also warning about it. I didn't understand that its purpose was solely to warn about situations where details in the underlying platform-specific OSX implementation, would cause performance issues that can't be easily avoided due to limitations in the native APIs, which always automatically recursively watches all files in the specified directory. Makes more sense now why removing the warning would not be a good idea in this case, why a simpler workaround (such as automatically excluding symlinks from being watched) might not be possible cross-platform and why the OSX discussion is relevant. Sorry about the misunderstanding, thanks for the full explanation! |
I agree with you here, I've never had a need to listen for changes on a project-local symlink file in my project. I have needed to listen for changes on a file in a directory that happens to contain a project-local symlink nested down somewhere randomly down in its tree, which was the situation here. Anyway, it's unfortunate that the OSX watcher makes this situation so complicated! 😢 |
Ignoring that symlink should be another solution – in this case it should It shouldn't be that hard to make Listen ignore warnings for symlinks that I should have better wiki pages to describe OSX issues with large projects. Anyway, spring Listen watcher is quite recent, and there's always a ton of Thanks for the discussion - it helped me gets a better sense of the big
|
I've updated the wiki page to clarify some of the details we covered here, hopefully this will help future users get less tripped up by this issue. |
Thanks for the changes - and for cutting out the whining and complaining. I had an idea about smarter "metrics" within Listen - so it could simply tell users they have a performance problem. For example, on Travis OSX builds, it can take about 1-2 seconds for a filesystem change to propagate to Listen (that's 1-2 seconds after the filesystem operation has finished). I'm also thinking about using the watchman implementation on OSX (https://facebook.github.io/watchman/). The client-server architecture is much better IMHO (they don't follow symlinks at all), and they've poured in massive resources to optimize on OSX and Linux. Again thanks for the updates. |
OS: Linux (Ubuntu 14.10)
Listen version: 3.03
Minimal test case:
My use case is that I would like to use spring-watcher-listen + Listen 3.0 in a Rails project containing several local symlinks which cannot be removed. Since #273, running Listen spams standard error with the above verbose text for what is a benign and extremely common scenario.
For my case, my workaround for now is to silence the SymlinkDetector with this monkey-patch:
I agree with the discussion in #273 that distinguishing between (harmless) local symlinks and (problematic) filesystem loops would be the 'Best solution' for this issue, as I believe that the current implementation throws out the baby with the bathwater, so to speak. As long as the 'Best' implementation remains a 'pointless waste of time', would it at least be possible to add an opt-in mechanism for suppressing this verbose
ERROR
message so that the local symlink warning could be suppressed without requiring this patch?The text was updated successfully, but these errors were encountered: