Suppress errors on local symlinks

[wjordan]

OS: Linux (Ubuntu 14.10)
Listen version: 3.03

Minimal test case:

mkdir dir
ln -s dir dir2
ruby -e "require 'listen'; Listen.to('.').start"

        ** ERROR: directory is already being watched! **

        Directory: [path]/dir

        is already begin watched through: [path]/dir

        MORE INFO: https://github.com/guard/listen/wiki/Duplicate-directory-errors

My use case is that I would like to use spring-watcher-listen + Listen 3.0 in a Rails project containing several local symlinks which cannot be removed. Since #273, running Listen spams standard error with the above verbose text for what is a benign and extremely common scenario.

For my case, my workaround for now is to silence the SymlinkDetector with this monkey-patch:

require 'listen/record/symlink_detector'
module Listen
  class Record
    class SymlinkDetector
      def _fail(_, _)
        fail Error, "Don't watch locally-symlinked file twice"
      end
    end
  end
end

I agree with the discussion in #273 that distinguishing between (harmless) local symlinks and (problematic) filesystem loops would be the 'Best solution' for this issue, as I believe that the current implementation throws out the baby with the bathwater, so to speak. As long as the 'Best' implementation remains a 'pointless waste of time', would it at least be possible to add an opt-in mechanism for suppressing this verbose ERROR message so that the local symlink warning could be suppressed without requiring this patch?

[e2]

The idea of physically watching the same directory twice seems wrong, no
matter what the scenario. It's not that using symlinks is wrong, it's that
watching both the original directory and symlinked directory at the same
time doesn't make sense.

E.g you can move app to foo/app and watch foo, and symlink app ->
foo/app. Listen is the only tool that needs access to the physical
directory - every other tool should work fine with symlinks.

So what's important here is the EXACT scenario, because I'm more than
willing to accept an exception to a good IT practice, than get reports
about complex performance errors on large projects.

OSX support is the key impediment here - if everyone used only Linux and
local Linux filesystems, there wouldn't be any issues except for the
filesystem loop detecting.

In short - you can blame Apple, or provide a good case that cannot be
worked around without getting rid of the warning.

I'd really suggest monkey-patching here instead of polluting the
documentation with more "special options".

The fact that we're having this discussion can save you and others many
potential headaches with Listen.

Especially with spring - it's best to solve the problem (or make a
well-supported, justified exception based on a use case). I need to write a
spec anyway.

Overall I simply refuse to support warning disabling on OSX itself - it's
better to invest time in smarter symlink support, but I don't have that
time (and I don't have OSX myself, either).

I'm also pretty sure the spring Listen watcher doesn't need to watch the
whole project - so maybe it needs to be patched instead.

Just saying I can't proceed here without a justified, concrete use case.

[wjordan]

Thanks for the quick turnaround, I'll follow up with spring-watcher-listen on your suggestion that it might be easier to patch that project before fixing here.

However, I'm not sure I understood other parts of your response, so please bear with me a bit:

1. How or why OSX relevant here? Are you saying that Listen has some unique issue on OSX symlinks that makes that platform a 'key impediment' to relying on the workaround I've provided? If so, I'd love details so I can track (and possibly help debug/fix) the problem. The majority of my development team uses OSX, so I'm quite concerned about any specific OSX-specific issues that will cause problems.
2. Is there something about the use case I described that is not 'justified' or 'concrete' enough for you to proceed? The SymlinkDetector already correctly prevents Listen from watching both the original directory and the symlinked directory at the same time, so the need to suppress the error messages when it is known that there is in fact no error is the concrete use case, which to me seems quite justified. As I mentioned, I expect it is an extremely common case for a user to have local symlinks nested somewhere within a directory they want to watch using Listen. Do you disagree?
3. Sorry but I'm not sure I understood this part at all, could you clarify what you mean by 'good IT practice' and 'complex performance errors' and how exactly it relates to this issue?

I'm more than willing to accept an exception to a good IT practice, than get reports about complex performance errors on large projects.

[e2]

Yes, OSX is why Listen is so complex, and I don't have OSX myself to even try to mitigate the issues. Basically, Listen can gravely underperform on OSX given complex large projects, and symlinks can reduce the performance exponentially. I've written about this is some of the replies in other issues. There's an open issue (or PR) for optimizing on OSX a bit, but it needs a major rewrite of dome areas of Listen (think filesystem snapshots in memory).

[e2]

As for the use case, I need something more specific - because I can't imagine a scenario where listening to a directory with symlinks is impossible to avoid. Maybe there's a better project setup I can suggest in you scenario - or if that's not possible, I'll have to consider the drawbacks of whatever changes are needed in Listen.

[e2]

By good IT practices, I meant that the OSX adapter is pretty resource hungry - and symlinks make things worse. I've never encountered a need for symlinks in a project, that weren't actually just a workaround or a symptom of a bigger issue.

For me, I wouldn't want to have to care about how you setup your project - it should be up to you. But, Apple's poor design decisions are not something I can change - e.g. having a log file in the root of your rails project can eat your cpu on OSX - and I cant really do anything about that.

Also, not every Listen user is a software developer - I prefer to help people avoid problems instead of silencing warnings that people could mindlessly enable, and then complain about high cpu usage without even providing a repo or logs as to give me a hint as to what could be wrong.

[wjordan]

To clarify, the issue here has nothing to do with Listen actually watching the same directory twice via symlink, or anything related to OSX's potential issues in that scenario. I think #273 does well enough by skipping any directory about to be watched a second time.

This issue is that the error messages are unnecessary when this behavior is intended. Unless I'm misunderstanding something, the symlinked directory can be silently skipped without any further problems in the case of a (common and harmless) local symlink. At the end of the day, the symlinked directory will be watched only once, change notifications will be sent once for each real directory's path, and OSX CPUs will never start screaming in agony.

For my specific case, I finally managed to set up a workaround in spring-watcher-listen#8 by using the workaround you suggested (move app to foo/app and watch foo, and symlink app ->
foo/app) to move the Spring-watched Gemfile out of my project root, but I'd still like to see this underlying error-message issue eventually addressed if possible.

I've never encountered a need for symlinks in a project, that weren't actually just a workaround or a symptom of a bigger issue.

You've never found a good use for symlinks, ever? Well I certainly have, and in my experience I would never leave $HOME without them 😄

Thanks for all your help so far!

[e2]

Just to clarify from my side: #273 is about detecting (filesystem loop and locally symlinked watched dirs) - it doesn't skip symlinked directories (that's not implemented).

If you're getting warnings, it means the physical directory is already being watched (realpath is used to check).

This issue is that the error messages are unnecessary when this behavior is intended.

Watching the same physical directory twice doesn't make any sense, ever. In fact, Facebook's watchman actually has a client/server model to prevent this (in a more elegant way - and there are more benefits too).

the symlinked directory can be silently skipped without any further problems

Yes, except that's not implemented - mostly because symlinks can be created/moved/deleted while listening and internally Listen is already too complex without such extensive symlink support. There's a huge cost in implementing, testing and maintaining this - and I can't afford to cover those costs from my own pocket. I'd be happy to accept PRs though.

Ideally, symlinks should be treated as files and not followed - but many users intuitively expect stuff inside symlinks to be watched, so it's really unclear which scenario should be supported.

There's just a whole universe of edge cases out there when you take symlinks into account (e.g. symlinks to dirs changing to symlinks to files, symlinked dirs being moved, handling broken symlinks, etc.).

You've never found a good use for symlinks, ever?

In a project, meaning "project-local symlinks" in the context of listening. Either you listen to edited files (so symlinks are not needed, because those files are the project and not something external) ... or ... you symlink files from other projects (and thus, you don't need to listen for them to changes, because changes to them happen through updating manifests and external dependency releases).

That's why I asked about your use case. E.g. if your assets are not in 'app/assets' but in 'vendor/assets' and you're symlinking - that's a project configuration problem.

What I mean is that the warnings are a lot more beneficial than any exception I can think of.

In fact the filesystem loop was a result of a project which had such a filesystem loop - even tools like the Unix find fail with an error in that case, so not every use of symlinks is even valid.

If there are any explanations in the Wiki which would've helped - let me know (or add missing hints yourself, because I'm too familiar with Listen internals to explain things clearly to everyone else).

[wjordan]

ahh, I think I understand better now.

I assumed that the SymlinkDetector was acting as a filter that was actively preventing directories from being watched twice, and also warning about it. I didn't understand that its purpose was solely to warn about situations where details in the underlying platform-specific OSX implementation, would cause performance issues that can't be easily avoided due to limitations in the native APIs, which always automatically recursively watches all files in the specified directory. Makes more sense now why removing the warning would not be a good idea in this case, why a simpler workaround (such as automatically excluding symlinks from being watched) might not be possible cross-platform and why the OSX discussion is relevant.

Sorry about the misunderstanding, thanks for the full explanation!

[wjordan]

You've never found a good use for symlinks, ever?

In a project, meaning "project-local symlinks" in the context of listening. Either you listen to edited files (so symlinks are not needed, because those files are the project and not something external) ... or ... you symlink files from other projects (and thus, you don't need to listen for them to changes, because changes to them happen through updating manifests and external dependency releases).

I agree with you here, I've never had a need to listen for changes on a project-local symlink file in my project. I have needed to listen for changes on a file in a directory that happens to contain a project-local symlink nested down somewhere randomly down in its tree, which was the situation here. Anyway, it's unfortunate that the OSX watcher makes this situation so complicated! 😢

[e2]

Ignoring that symlink should be another solution – in this case it should
be the spring watcher which should support configurable options for Listen.

It shouldn't be that hard to make Listen ignore warnings for symlinks that
match the ignite rules anyway.

I should have better wiki pages to describe OSX issues with large projects.

Anyway, spring Listen watcher is quite recent, and there's always a ton of
possible improvements everywhere.

Thanks for the discussion - it helped me gets a better sense of the big
picture.
On Aug 21, 2015 1:51 AM, "Will Jordan" notifications@github.com wrote:

You've never found a good use for symlinks, ever?

In a project, meaning "project-local symlinks" in the context of
listening. Either you listen to edited files (so symlinks are not needed,
because those files are the project and not something external) ... or ...
you symlink files from other projects (and thus, you don't need to listen
for them to changes, because changes to them happen through updating
manifests and external dependency releases).

I agree with you here, I've never had a need to listen for changes on a
project-local symlink file in my project. I have needed to listen for
changes on a file in a directory that happens to contain a project-local
symlink nested down somewhere randomly down in its tree, which was the
situation here. Anyway, it's unfortunate that the OSX watcher makes this
situation so complicated! [image: 😢]

—
Reply to this email directly or view it on GitHub
#339 (comment).

[wjordan]

I've updated the wiki page to clarify some of the details we covered here, hopefully this will help future users get less tripped up by this issue.

[e2]

Thanks for the changes - and for cutting out the whining and complaining.

I had an idea about smarter "metrics" within Listen - so it could simply tell users they have a performance problem. For example, on Travis OSX builds, it can take about 1-2 seconds for a filesystem change to propagate to Listen (that's 1-2 seconds after the filesystem operation has finished).

I'm also thinking about using the watchman implementation on OSX (https://facebook.github.io/watchman/).

The client-server architecture is much better IMHO (they don't follow symlinks at all), and they've poured in massive resources to optimize on OSX and Linux.

Again thanks for the updates.