preview sets quarantine attribute on all files it opens on an SSHFS volume. #162

Closed
glyph opened this Issue Sep 19, 2014 · 31 comments

Comments

Projects
None yet
10 participants
@glyph

glyph commented Sep 19, 2014

If I ever open the same image twice with Preview from an SSHFS-mounted volume, I get the annoying “is damaged and can’t be opened” dialog.

The images aren't actually damaged. I can read them just fine if I manually remove the attribute.

Other image viewers and editing applications don't have this problem. It appears to be specific to Preview. I'm not sure if it's sshfs-specific or all osxfuse filesystems.

I realize this report is pretty vague. I'm happy to do any additional diagnostics that would be helpful.

@mehaase

This comment has been minimized.

Show comment
Hide comment
@mehaase

mehaase Oct 14, 2014

I have the same issue with Keynote files on encfs. If I open, save, and close, then I can't open it from Finder again. I can open it just fine if I go into Keynote and choose File → Open.

Removing the com.apple.quarantine attribute fixes the problem, as does disabling GateKeeper altogether. I tried passing "-o noautoxattr" (the exact spelling escapes me at this moment) to no avail. I could determine from the verbose output and presence of Apple Double files that the switch does disable native xattr, but that didn't solve the Keynote "damaged" error.

I'm not sure if the issue is in encfs or osxfuse. I assumed the former until I saw this issue. I tried putting the same file into a loopback filesystem and it works fine, so it's not a general osxfuse issue.

mehaase commented Oct 14, 2014

I have the same issue with Keynote files on encfs. If I open, save, and close, then I can't open it from Finder again. I can open it just fine if I go into Keynote and choose File → Open.

Removing the com.apple.quarantine attribute fixes the problem, as does disabling GateKeeper altogether. I tried passing "-o noautoxattr" (the exact spelling escapes me at this moment) to no avail. I could determine from the verbose output and presence of Apple Double files that the switch does disable native xattr, but that didn't solve the Keynote "damaged" error.

I'm not sure if the issue is in encfs or osxfuse. I assumed the former until I saw this issue. I tried putting the same file into a loopback filesystem and it works fine, so it's not a general osxfuse issue.

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Oct 14, 2014

@mehaase Thanks for letting me know I'm not taking crazy pills over here :-).

glyph commented Oct 14, 2014

@mehaase Thanks for letting me know I'm not taking crazy pills over here :-).

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Oct 14, 2014

Also, I should note that I've reproduced this issue on the latest Yosemite beta, so it's not version-specific to Mavericks.

glyph commented Oct 14, 2014

Also, I should note that I've reproduced this issue on the latest Yosemite beta, so it's not version-specific to Mavericks.

@netheril96

This comment has been minimized.

Show comment
Hide comment
@netheril96

netheril96 Oct 18, 2014

I have the same problem with both encfs and sshfs.

I have the same problem with both encfs and sshfs.

@netheril96

This comment has been minimized.

Show comment
Hide comment
@netheril96

netheril96 Oct 18, 2014

@mehaase What is GateKeeper and how do I disable it? It really does not increase security at all for me, but annoying as hell.

@mehaase What is GateKeeper and how do I disable it? It really does not increase security at all for me, but annoying as hell.

@mehaase

This comment has been minimized.

Show comment
Hide comment
@mehaase

mehaase Oct 20, 2014

"GateKeeper" is Apple's branding for signed code. I posted an answer on Stack Exchange that explains how to disable it.

mehaase commented Oct 20, 2014

"GateKeeper" is Apple's branding for signed code. I posted an answer on Stack Exchange that explains how to disable it.

@ni4

This comment has been minimized.

Show comment
Hide comment
@ni4

ni4 Nov 5, 2014

It also happens on my custom Cloud filesystem.
I made sure that com.apple.quarantine attribute has the same value as set and it equals to value in Downloads folder.
So it seems that Finder/GateKeeper/whatever else do some wrong checks for this attribute value - when it is removed all works fine.

ni4 commented Nov 5, 2014

It also happens on my custom Cloud filesystem.
I made sure that com.apple.quarantine attribute has the same value as set and it equals to value in Downloads folder.
So it seems that Finder/GateKeeper/whatever else do some wrong checks for this attribute value - when it is removed all works fine.

@mingchen

This comment has been minimized.

Show comment
Hide comment
@mingchen

mingchen Jan 15, 2015

I run into the same error on Macbook.
Mac OS X: 10.10.1.
enfs: 1.7.5
osxfuse: 2.7.4

Here is a workaround:

$ xattr -r -d com.apple.quarantine /path/to/encfs/mount/point

Refer http://apple.stackexchange.com/questions/129966/files-wrongly-considered-as-damaged-in-encfs-volume

I run into the same error on Macbook.
Mac OS X: 10.10.1.
enfs: 1.7.5
osxfuse: 2.7.4

Here is a workaround:

$ xattr -r -d com.apple.quarantine /path/to/encfs/mount/point

Refer http://apple.stackexchange.com/questions/129966/files-wrongly-considered-as-damaged-in-encfs-volume

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Jan 17, 2015

That workaround only fixes the problem temporarily - next time you open the files it will recur.

Is there a way to disable xattrs entirely so that it can't set the quarantine attribute in the first place? I realize that this has security implications, so I don't think it should be enabled by default, but if I have a volume that I know I don't download things to directly, I should be able to move images and such to it.

glyph commented Jan 17, 2015

That workaround only fixes the problem temporarily - next time you open the files it will recur.

Is there a way to disable xattrs entirely so that it can't set the quarantine attribute in the first place? I realize that this has security implications, so I don't think it should be enabled by default, but if I have a volume that I know I don't download things to directly, I should be able to move images and such to it.

@davek44

This comment has been minimized.

Show comment
Hide comment
@davek44

davek44 Apr 7, 2015

I have this same frustrating issue.

davek44 commented Apr 7, 2015

I have this same frustrating issue.

@Nxt3

This comment has been minimized.

Show comment
Hide comment
@Nxt3

Nxt3 Apr 30, 2015

I have the same issue. Oddly enough, if I try to open a video via QuickTime Player, the file will be damaged. If I open that same file with Preview, it works fine.

Nxt3 commented Apr 30, 2015

I have the same issue. Oddly enough, if I try to open a video via QuickTime Player, the file will be damaged. If I open that same file with Preview, it works fine.

@Nxt3

This comment has been minimized.

Show comment
Hide comment
@Nxt3

Nxt3 Apr 30, 2015

I should add that I use encfs and OSXFUSE

Nxt3 commented Apr 30, 2015

I should add that I use encfs and OSXFUSE

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Apr 30, 2015

At some point between my Jan 16th comment and now, I updated OSXFuse and this issue stopped happening, at least in the few cases I've since tried. Is everyone still having this problem on the most recent version?

glyph commented Apr 30, 2015

At some point between my Jan 16th comment and now, I updated OSXFuse and this issue stopped happening, at least in the few cases I've since tried. Is everyone still having this problem on the most recent version?

@abedwardsw

This comment has been minimized.

Show comment
Hide comment
@abedwardsw

abedwardsw May 1, 2015

This issue drives me nuts. ;-) Good idea on the OSXFuse, i've just updated mine from 2.6.4 to 2.7.6, will let you know if it's resolved for me.

This issue drives me nuts. ;-) Good idea on the OSXFuse, i've just updated mine from 2.6.4 to 2.7.6, will let you know if it's resolved for me.

@Nxt3

This comment has been minimized.

Show comment
Hide comment
@Nxt3

Nxt3 May 1, 2015

@abedwardsw Where did you get the 2.7.6 update? Did you build it from source?

Nxt3 commented May 1, 2015

@abedwardsw Where did you get the 2.7.6 update? Did you build it from source?

@bfleischer

This comment has been minimized.

Show comment
Hide comment
@bfleischer

bfleischer May 1, 2015

Member

@Nxt3 The only difference between 2.7.6 and 2.7.5 is a minor modification to the build script to support building with Xcode 6.3 and 6.4 for package managers like Homebrew. Feature-wise there is no difference at all.

Member

bfleischer commented May 1, 2015

@Nxt3 The only difference between 2.7.6 and 2.7.5 is a minor modification to the build script to support building with Xcode 6.3 and 6.4 for package managers like Homebrew. Feature-wise there is no difference at all.

@abedwardsw

This comment has been minimized.

Show comment
Hide comment
@abedwardsw

abedwardsw May 1, 2015

Sorry, I ended up doing 2.7.5, brew was telling me 2.7.6, but I used the .dmg and went with 2.7.5

Unfortunately I still get the same corruption error until I run xattr -r -d com.apple.quarantine any time I create a new file. I really don't want to completely change the "Allow apps downloaded from: " to "Anywhere" as I like this feature. It seems like the files from encfs are wrongly being distinguished as an app instead of a file...

I am using encfs.

Any ideas? I'll try rebooting, but it does seem the new osxfuse version is loaded properly.

Sorry, I ended up doing 2.7.5, brew was telling me 2.7.6, but I used the .dmg and went with 2.7.5

Unfortunately I still get the same corruption error until I run xattr -r -d com.apple.quarantine any time I create a new file. I really don't want to completely change the "Allow apps downloaded from: " to "Anywhere" as I like this feature. It seems like the files from encfs are wrongly being distinguished as an app instead of a file...

I am using encfs.

Any ideas? I'll try rebooting, but it does seem the new osxfuse version is loaded properly.

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph May 2, 2015

As another data point, I'm using sshfs, not encfs. It's possible that the relevant thing I updated was sshfs and not osxfuse itself. I will report versions and re-test when I get back to the machines with this installed.

glyph commented May 2, 2015

As another data point, I'm using sshfs, not encfs. It's possible that the relevant thing I updated was sshfs and not osxfuse itself. I will report versions and re-test when I get back to the machines with this installed.

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph May 2, 2015

So, I tried again, and discovered that I still have the issue. Versions:

SSHFS version 2.5 (OSXFUSE SSHFS 2.5.0)
OSXFUSE library version: FUSE 2.7.3 / OSXFUSE 2.7.5

The reason I thought it was fixed, is that some software created .DS_Store files. In a directory with an existing .DS_Store file, files can be opened and created with no problem; touch .DS_Store in a directory fixes the issue.

glyph commented May 2, 2015

So, I tried again, and discovered that I still have the issue. Versions:

SSHFS version 2.5 (OSXFUSE SSHFS 2.5.0)
OSXFUSE library version: FUSE 2.7.3 / OSXFUSE 2.7.5

The reason I thought it was fixed, is that some software created .DS_Store files. In a directory with an existing .DS_Store file, files can be opened and created with no problem; touch .DS_Store in a directory fixes the issue.

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Jun 2, 2015

Hi - I guess it's time for my monthly checkin!

I think I have a workaround for this, although I'm not super happy about it, it seems a reasonable compromise. Perhaps this investigation will be useful to the maintainers figuring out another way to handle it.

I thought to myself, OS X uses system users and system daemons for all kinds of jobs, perhaps the kernel is expecting to be able to do some work as another user, or as root, to these files, and marking them as damaged when that doesn't work.

So I marked my sshfs binary as setuid, and I added the -o allow_other mount option to my sshfs command line, and ... I appear to be able to open and edit documents reliably on the mounted volume. I will continue to experiment and follow up if it stops working.

I am of course concerned about a setuid root binary lying around, but it seems better than the option of running a daemon which requires root privileges on the file server side of things to get NFS or SMB. :)

glyph commented Jun 2, 2015

Hi - I guess it's time for my monthly checkin!

I think I have a workaround for this, although I'm not super happy about it, it seems a reasonable compromise. Perhaps this investigation will be useful to the maintainers figuring out another way to handle it.

I thought to myself, OS X uses system users and system daemons for all kinds of jobs, perhaps the kernel is expecting to be able to do some work as another user, or as root, to these files, and marking them as damaged when that doesn't work.

So I marked my sshfs binary as setuid, and I added the -o allow_other mount option to my sshfs command line, and ... I appear to be able to open and edit documents reliably on the mounted volume. I will continue to experiment and follow up if it stops working.

I am of course concerned about a setuid root binary lying around, but it seems better than the option of running a daemon which requires root privileges on the file server side of things to get NFS or SMB. :)

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Jun 2, 2015

I just placed this as an answer on Stack Overflow as well.

glyph commented Jun 2, 2015

I just placed this as an answer on Stack Overflow as well.

@Thinkscape Thinkscape referenced this issue in vgough/encfs Jun 22, 2015

Closed

Problem opening files on OS X #70

@jooray

This comment has been minimized.

Show comment
Hide comment
@jooray

jooray Jun 22, 2015

What about OSX Fuse mount option to clear the attribute when someone wants to set it?

jooray commented Jun 22, 2015

What about OSX Fuse mount option to clear the attribute when someone wants to set it?

@mehaase

This comment has been minimized.

Show comment
Hide comment
@mehaase

mehaase Jun 22, 2015

@glyph Thanks for the update. I tried using the setuid trick with encfs. I'm able to open documents without an error message, but the quarantine attribute is still getting set, and when I try to save, I get a "This file is locked" error. If I look in the Finder, it doesn't have the locked attribute set. xattr -l still shows the quarantine flag being set.

So it's new behavior for encfs, but it's still broken behavior.

Update

I spent more time debugging. By turning on FUSE debug output (e.g. encfs -f -d ...), I can see that it is Pages itself that is setting the Quarantine flag, not Finder. I tried compiling both osxfuse and encfs from scratch on their respective master branches, to no avail. There is a similar issue open on encfs that links back to here. Notably, that issue links to a blog post from BoxCryptor that indicates this is a bug in OS X. At this point, I'm inclined to agree, because Apple's documentation of how the quarantine system works is pitiful. Given that Apple is a black hole when it comes to bug reports, I am officially giving up on trying to fix this "the right way".

Instead, I just patched osxfuse to ignore calls to getxattr when the attribute name is com.apple.quarantine. This is still a bit dangerous – quarantine won't work at all in FUSE volumes – but it's safer (and less hassle) than disabling Gate Keeper or the setuid trick.

diff --git a/lib/fuse.c b/lib/fuse.c
index 50591d5..71f1aee 100644
--- a/lib/fuse.c
+++ b/lib/fuse.c
@@ -2363,7 +2363,9 @@ int fuse_fs_getxattr(struct fuse_fs *fs, const char *path, const char *name,
 #endif
 {
        fuse_get_context()->private_data = fs->user_data;
-       if (fs->op.getxattr) {
+       if (strncmp("com.apple.quarantine", name, 20) == 0) {
+               return -ENODATA;
+       } else if (fs->op.getxattr) {
                if (fs->debug)
 #ifdef __APPLE__
                        fprintf(stderr, "getxattr %s %s %lu %lu\n",

It's ugly, but it works... If you want to try reproducing this, remove your existing osxfuse, disable kext signature verification, then:

$ brew install autoconf automake libtool
$ git clone --recursive -b master git://github.com/osxfuse/osxfuse.git osxfuse
$ cd osxfuse/fuse
$ git apply patch # use the patch from above
$ ./build.sh -t distribution
$ open /tmp/osxfuse/distribution/Packages/Core.pkg

The last step will open an installer to install osxfuse.

Of course, the downside here is turning off kext signing. (I already had it turned off, for other reasons.) The change itself is only in the library, not in the kext, so it's probably possible to install the official, signed kext and then build just the library, but it's getting late and I don't have time to try that right now...

mehaase commented Jun 22, 2015

@glyph Thanks for the update. I tried using the setuid trick with encfs. I'm able to open documents without an error message, but the quarantine attribute is still getting set, and when I try to save, I get a "This file is locked" error. If I look in the Finder, it doesn't have the locked attribute set. xattr -l still shows the quarantine flag being set.

So it's new behavior for encfs, but it's still broken behavior.

Update

I spent more time debugging. By turning on FUSE debug output (e.g. encfs -f -d ...), I can see that it is Pages itself that is setting the Quarantine flag, not Finder. I tried compiling both osxfuse and encfs from scratch on their respective master branches, to no avail. There is a similar issue open on encfs that links back to here. Notably, that issue links to a blog post from BoxCryptor that indicates this is a bug in OS X. At this point, I'm inclined to agree, because Apple's documentation of how the quarantine system works is pitiful. Given that Apple is a black hole when it comes to bug reports, I am officially giving up on trying to fix this "the right way".

Instead, I just patched osxfuse to ignore calls to getxattr when the attribute name is com.apple.quarantine. This is still a bit dangerous – quarantine won't work at all in FUSE volumes – but it's safer (and less hassle) than disabling Gate Keeper or the setuid trick.

diff --git a/lib/fuse.c b/lib/fuse.c
index 50591d5..71f1aee 100644
--- a/lib/fuse.c
+++ b/lib/fuse.c
@@ -2363,7 +2363,9 @@ int fuse_fs_getxattr(struct fuse_fs *fs, const char *path, const char *name,
 #endif
 {
        fuse_get_context()->private_data = fs->user_data;
-       if (fs->op.getxattr) {
+       if (strncmp("com.apple.quarantine", name, 20) == 0) {
+               return -ENODATA;
+       } else if (fs->op.getxattr) {
                if (fs->debug)
 #ifdef __APPLE__
                        fprintf(stderr, "getxattr %s %s %lu %lu\n",

It's ugly, but it works... If you want to try reproducing this, remove your existing osxfuse, disable kext signature verification, then:

$ brew install autoconf automake libtool
$ git clone --recursive -b master git://github.com/osxfuse/osxfuse.git osxfuse
$ cd osxfuse/fuse
$ git apply patch # use the patch from above
$ ./build.sh -t distribution
$ open /tmp/osxfuse/distribution/Packages/Core.pkg

The last step will open an installer to install osxfuse.

Of course, the downside here is turning off kext signing. (I already had it turned off, for other reasons.) The change itself is only in the library, not in the kext, so it's probably possible to install the official, signed kext and then build just the library, but it's getting late and I don't have time to try that right now...

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Jun 23, 2015

Is there a way to just add an additional trust root / sign the locally built kext without disabling kext verification entirely?

glyph commented Jun 23, 2015

Is there a way to just add an additional trust root / sign the locally built kext without disabling kext verification entirely?

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Jun 23, 2015

@mehaase - I'm not quite clear on the division of responsibility between FUSE and SSHFS; particularly, does an equivalent to -o allow_other exist in EncFS or exist at the FUSE layer, or is it an SSHFS-ism?

With -o allow_other and SSHFS, the com.apple.quarantine xattr is indeed still getting set, but I can happily open the file from Finder, from open in the shell, from File→Open, and so on. I am not sure what information the quarantine attribute conveys now, but it looks like it's just saying that a given application touched it.

glyph commented Jun 23, 2015

@mehaase - I'm not quite clear on the division of responsibility between FUSE and SSHFS; particularly, does an equivalent to -o allow_other exist in EncFS or exist at the FUSE layer, or is it an SSHFS-ism?

With -o allow_other and SSHFS, the com.apple.quarantine xattr is indeed still getting set, but I can happily open the file from Finder, from open in the shell, from File→Open, and so on. I am not sure what information the quarantine attribute conveys now, but it looks like it's just saying that a given application touched it.

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Jun 23, 2015

Also: I tried Preview, Keynote, and Pages, just to be sure.

glyph commented Jun 23, 2015

Also: I tried Preview, Keynote, and Pages, just to be sure.

bfleischer added a commit to osxfuse/kext that referenced this issue Oct 17, 2015

Fix opening quarantined files with Preview
Starting with OS X 10.9 syspolicyd (which is running as root) calls
getxattr(2) when opening items in Finder. Blocking these calls results
in Finder displaying an error message. Therefore we are no longer
blocking vnop_getxattr calls by root even if allow_root or allow_other
is not set.

See osxfuse/osxfuse#162 for details.

bfleischer added a commit to osxfuse/kext that referenced this issue Oct 25, 2015

Fix opening quarantined files with Preview
Starting with OS X 10.9 syspolicyd (which is running as root) calls
getxattr(2) when opening items in Finder. Blocking these calls results
in Finder displaying an error message. Therefore we are no longer
blocking vnop_getxattr calls by root even if allow_root or allow_other
is not set.

See osxfuse/osxfuse#162 for details.

@bfleischer bfleischer added this to the 2.8.2 milestone Oct 25, 2015

@bfleischer bfleischer self-assigned this Oct 25, 2015

@bfleischer

This comment has been minimized.

Show comment
Hide comment
@bfleischer

bfleischer Oct 25, 2015

Member

The issue has been fixed in version 2.8.2.

Member

bfleischer commented Oct 25, 2015

The issue has been fixed in version 2.8.2.

@bfleischer bfleischer closed this Oct 25, 2015

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Oct 26, 2015

@bfleischer Do you have any more details on what the resolution was, or why this was happening?

glyph commented Oct 26, 2015

@bfleischer Do you have any more details on what the resolution was, or why this was happening?

@bfleischer

This comment has been minimized.

Show comment
Hide comment
@bfleischer

bfleischer Oct 26, 2015

Member

@glyph By default only the user that mounted the FUSE volume can access it. Starting with OS X 10.9 syspolicyd (which is running as root) calls getxattr(2) when opening items in Finder. Blocking these calls results in Finder displaying the misleading "is damaged and can’t be opened" error message. Therefore we are no longer blocking getxattr(2) calls by root in version 2.8.2.

Member

bfleischer commented Oct 26, 2015

@glyph By default only the user that mounted the FUSE volume can access it. Starting with OS X 10.9 syspolicyd (which is running as root) calls getxattr(2) when opening items in Finder. Blocking these calls results in Finder displaying the misleading "is damaged and can’t be opened" error message. Therefore we are no longer blocking getxattr(2) calls by root in version 2.8.2.

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Oct 26, 2015

@bfleischer this seems like kind of a brittle fix; other users (including root) can read some sensitive data, but not others, by default? It seems like maybe root should just be able to do everything all the time, or at least that a thing like allow_other should not require root access to turn on…

(Nevertheless thank you very much for fixing this bug.)

glyph commented Oct 26, 2015

@bfleischer this seems like kind of a brittle fix; other users (including root) can read some sensitive data, but not others, by default? It seems like maybe root should just be able to do everything all the time, or at least that a thing like allow_other should not require root access to turn on…

(Nevertheless thank you very much for fixing this bug.)

@mehaase

This comment has been minimized.

Show comment
Hide comment
@mehaase

mehaase Oct 26, 2015

@bfleischer This is really awesome, thanks! 2.8.2 doesn't seem to be compatible with the latest encfs, but I tried the 3.0.7 build and it is working swimmingly.

mehaase commented Oct 26, 2015

@bfleischer This is really awesome, thanks! 2.8.2 doesn't seem to be compatible with the latest encfs, but I tried the 3.0.7 build and it is working swimmingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment