Add --unsafe-ignore flag to allow ignored content to be removed

[slightlyfaulty]

I keep running into errors like the following:

Problems:
    composer/cache: unknown content encountered on disk
    some/path/.gitignore: unknown content encountered on disk
    some/path/too/.github: unknown content encountered on disk

As you might have guessed these are all ignored paths. The error seems to occur when deleting or renaming a parent folder on alpha that contains ignored paths on beta. The parent folder is successfully deleted/renamed on alpha but remains on beta with only the ignored paths as descendants.

I think the correct behaviour should be to delete or move ALL descendants of a synced folder accordingly regardless of ignores.

Running latest Mutagen v0.9.0 with Windows 10 as alpha and Docker Ubuntu as beta.

[lukasluecke]

I have the same issue using the --ignore-vcs flag, and having .git folders somewhere deep inside beta.

[xenoscopic]

This is really more of a safety issue. Mutagen's rule is essentially that it won't touch any files that it isn't managing (i.e. it won't touch any files that are ignored). The idea is to avoid accidentally wiping out something like a .git directory.

What about having an --unsafe-ignore option that would tell Mutagen to ignore a particular path but also feel free to overwrite/remove it if necessary (e.g. if a parent is deleted)? I haven't really thought through the technical implications of this, but would something like that work?

[paales]

@havoc-io Encountering this issue as well, the pub/static folders are generated folders and will be created / removed at any point. (we are ignoring the .DS_Store files, so that makes sense..

In our case it would be perfectly viable to actually remove the DS_Store files if the parent is removed.. So an --unsafe-ignore would work in our case. :)

[xenoscopic]

@paales Thanks for the additional experience report, this sort of input is really valuable. I do plan to work on this feature for the v0.11 or v0.12 release. The implementation for this is (I believe) going to be done at the same time as the fix for #113 and #99, but it requires some big changes to Mutagen's internal algorithms that are going to require a lot of testing for safety, so I don't have an exact timeline (and the design is still sort of theoretical).

In the mean time, what I would suggest is just unignoring .DS_Store for that specific directory (assuming it won't cause problems in the container). So keep your global .DS_Store ignore, and add something like !pub/static/**/.DS_Store (or whatever makes sense). That will cause Mutagen to sync .DS_Store files within pub/static, but it will also give Mutagen permission to delete those files when their parents are deleted.

[paales]

Thanks for the workaround, we're doing that now and that seems to be solving the issue. Thanks :)

[Badg]

I ran into this issue with __pycache__ folders (alpha: host computer, beta: docker container) as well; my workaround was to set the PYTHONPYCACHEPREFIX env var on the docker container, then do one-way-replica as the sync mode with beta permissions set to 640 (files) and 751 (directories) to prevent beta from "failing quietly" into a conflict state (ie, the app itself is running as a separate user in the same group, so it can read, but it would fail if it tried to write, instead of mutagen sync just having a conflict).