New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SystemNative_CopyFile() call to fchmod() introduced in Mono 6.0 is not compatible with the external storage location on some Android OS versions and devices #17133
Comments
/cc @steveisok |
Did you try set ? <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"></uses-permission> |
Also, what is the filesystem on the storage? |
Jonathan Pryor identified an interesting point about this issue. Part of the documented behavior of
The documentation does not specifically mention whether the copied file also retains the permissions, mode, or access control list of the original file, but it seems like a potentially valid design choice for If the team agrees that The recommendation for Android developers would then be to use another option like Possible changes to
|
Cross-reference: The earlier issue #16573 reported for Linux raises essentially the same question about the intended behavior of |
Another interesting detail about this issue is that currently the behavior of In contrast, for the same scenario of overwriting a file owned by another user, the command line commands |
That would be crazy. You can't possibly be considering this. Not only would that make 3rd party libraries that may call File.Copy unusable on android. Developers are gonna release apps that work in their devices and they will likely fail in production, just because of the sheer variety of Android devices. This permission behavior happens on maybe 20% of devices we've encountered so far. Which is a very substantial part of our customer base, but we don't necessarily encounter it during testing. It should not matter how File.Copy was originally defined. This is basically an Android "security feature" preventing you to set file permissions, and it needs to be gracefully handled by Xamarin, not by the user. I have to justify using Xamarin now, because my co-workers say it is always buggy. Which sickens me, because I really like to use C#. But you seriously cannot do breaking changes to basic things like File.Copy after it worked for years. You also cannot mindlessly migrate .NET Core code into mono like that. People actually use your stuff in production, guys!!! |
My feeling is that due to historical reasons we should look at this API as one which tries to copy as much as it can but if it cannot copy/set the permissions it should not fail. We can also propose a new API or a new overload which can be more Linux/Unix oriented. |
What I really don't like about that Also - unrelated to this particular issue - that function should never have been rolled out into production without breaking down those loops. They're not thread-abort / runtime shutdown safe. |
Android doesn't allow setting permissions on that virtual filesystem. It's not needed, and not supported. If mono just didn't run fchmod, everything would work exactly as it should. Also, user ids on android are for distinguishing apps, not users; and group ids are for distinguishing certain kinds of permissions that apps can have. The most responsible way to handle this on android would probably be to put "#ifndef ANDROID" around the fchmod, or to ignore it when it fails. |
I agree with @marek-safar's idea of a Linux/Unix specific overload - for desktop and server apps - and I second @tobiasschulz's sentiment that Android should ignore (although not remove completely) any errors coming from |
We are hitting this issue in production because we made the huge mistake of upgrading our Xamarin developer tools to their latest version. We should have known better. The most unbelievable part of this is that this breaking change was ever allowed to ship. You shipped a change where a File.Copy operation would fail on external storage, and judging by our telemetry on a significant proportion of Android devices. It is unforgivable, and so is the delay in addressing your mistake. It used to be the case that Microsoft shipped higher quality code than this into their stable releases. |
@divil5000 (and everyone else at large) I do apologize for the issue. We are actively working on it and will try to get the fix out to you as soon as we can. |
Is this fixed yet? This problem was originally reported August 1st. |
They don't actually seem to get it. :( All they need to do is ignore the return value of the chmod function. |
We are working to get it fixed in time for the d16-4 preview 3 or 4 Xamarin release. More when I know it. |
So what date is d16-4 going to be ready? |
I can chime in with one bit of information about release timing for Visual Studio 2019 version 16.4. Apologies in advance that I don't have a specific date. I think the best information available at the moment might be that historically the time between Visual Studio 2019 version 16.x release versions has been about 1.5 or 2 months, so if that pattern holds, the Visual Studio 2019 version 16.4 release will be available in November. |
Okay, but it has not been fixed in mono yet, has it? Which mono version is gonna be in 16.4? |
If that question is in reply to my comment, then I can clarify that my comment about the release timeline of Visual Studio 2019 version 16.4 is only relevant in the context of this issue if a fix does indeed get included in the Mono version used by the Xamarin.Android SDK for Visual Studio 2019 version 16.4 Preview 3 or Preview 4. For now, steveisok's comment about the work toward a fix is the more important status information. |
"simply ignoring the return value" is, of course, completely unacceptable and not going to happen. We are going to hard fail on all platforms if the PR will be up shortly, but I don't have any Android devices, so I will need some help testing this. |
…for Android. The `fchmod()` call should happen first to ensure that the target file has the correct permissions prior to writing into it. On Android, we make an exception in case the `fchmod()` fails: if we can `fstat()` both the source and the target file and the target file's permissions are at least as restrictive as the source file's, then we allow the operation to proceed. See mono/mono#17133 for details.
You don't seem to understand what permissions on android mean in that case. It's a fuse file system for external storage. Apps are not supposed to set permissions on external storage. All files are always gonna have the permissions that the virtual file system wants the app to see. |
…for Android. The `fchmod()` call should happen first to ensure that the target file has the correct permissions prior to writing into it. On Android, we make an exception in case the `fchmod()` fails: if we can `fstat()` both the source and the target file and the target file's permissions are at least as restrictive as the source file's, then we allow the operation to proceed. See mono/mono#17133 for details.
…for Android. The `fchmod()` call should happen first to ensure that the target file has the correct permissions prior to writing into it. On Android, we make an exception in case the `fchmod()` fails: if we can `fstat()` both the source and the target file and the target file's permissions are at least as restrictive as the source file's, then we allow the operation to proceed. See mono/mono#17133 for details.
So if you copy a file from internal storage (where you can set you own permissions) to external storage (where you cannot), and the permissions are less restrictive on the external storage because the fuse virtual file system enforces that, the File.Copy call is still gonna fail, it that what you are saying? Because that is what this looks like to me, which means that File.Copy would still be complete unusable. And that is what's completely unacceptable. Example: Source file: Target file: The target file's permissions are less restrictive than the source file's. So this is still gonna fail? Considering that you cannot set the permissions on external storage, it would not be possible to copy a file without group permissions to external storage, where the group can always read/write? |
The correct solution to this would be to revert the code to how it was before you catastrophically broke it, letting down all your customers who were relying on it. |
Thank you so much for this very valuable feedback. I edited my PR accordingly: |
…for Android (mono#364) * SystemNative_CopyFile(): move fchmod() call to top and add exception for Android. The `fchmod()` call should happen first to ensure that the target file has the correct permissions prior to writing into it. On Android, we make an exception in case the `fchmod()` fails: if we can `fstat()` both the source and the target file and the target file's permissions are at least as restrictive as the source file's, then we allow the operation to proceed. See mono/mono#17133 for details. * Update pal_io.c
Changes: mono/mono@8946e49...e1ef774 Context: mono/mono#17133 * mono/mono@e1ef774391d: [2019-08] Bump CoreFX to pickup corefx PR xamarin#367 to fix #17133. (#17622) * mono/mono@6d1f88e0ad2: Bump msbuild to get SDK updates from mono/msbuild#150 * mono/mono@a3f3bfc4c3d: Bump nuget to the latest suggested version * mono/mono@9bd3079f1ca: [2019-08] bump msbuild with more p2 packages * mono/mono@6ac1ff75a27: [dim][regression] Explicit interface override (#17583) (#17627) * mono/mono@a119807a015: [acceptance-tests] Bump mono/roslyn to get build fix * mono/mono@d234d34b700: [2019-08][merp] Install sigterm handler in EnableMicrosoftTelemetry * mono/mono@444a9a3fc48: [2019-08] [merp] Introduce a new 'dump mode' that allows different signal behavior when dumping (#17568)
Changes: mono/mono@8946e49...e1ef774 Context: mono/mono#17133 * mono/mono@e1ef774391d: [2019-08] Bump CoreFX to pickup corefx PR #367 to fix #17133. (#17622) * mono/mono@6d1f88e0ad2: Bump msbuild to get SDK updates from mono/msbuild#150 * mono/mono@a3f3bfc4c3d: Bump nuget to the latest suggested version * mono/mono@9bd3079f1ca: [2019-08] bump msbuild with more p2 packages * mono/mono@6ac1ff75a27: [dim][regression] Explicit interface override (#17583) (#17627) * mono/mono@a119807a015: [acceptance-tests] Bump mono/roslyn to get build fix * mono/mono@d234d34b700: [2019-08][merp] Install sigterm handler in EnableMicrosoftTelemetry * mono/mono@444a9a3fc48: [2019-08] [merp] Introduce a new 'dump mode' that allows different signal behavior when dumping (#17568)
@baulig Other platforms (for example mounted exfat/ntfs/cloud filesystems) don't allow modifying permissions either. So it would likely fail there too and not just android. In mono 6.0 it would fail after the write, now the problem simply got moved to the top of CopyFile and you end up with a 0 length file. The fundamental issue is the assumption that chmod should succeed here. In mono < 6.0, File.Copy didn't even copy permissions. Now it does, and fails if it can't. The workaround (using the managed CopyTo) is pure insanity. Every app would have to write it's own replacement for File.Copy and File.Move, they won't be able to use the faster kernel-level operations that SystemNative_CopyFile implements. Maybe worth noting that on Windows, copying a file in Windows Explorer doesn't copy the file ACLs, neither does a net462 & netcore app. File.Copy should not preserve permissions. File.Move should preserve permissions (even if it has to fall back to a Copy on cross-filesystem), but should not fail doing so if it can't. |
Changes: mono/mono@8946e49...062f0ab Context: mono/mono#17133 * mono/mono@062f0ab: [sdks] Use Xcode 11.2 stable version for iOS/Mac SDKs * mono/mono@e1ef774: [2019-08] Bump CoreFX to pickup corefx PR xamarin#367 to fix #17133. (#17622) * mono/mono@6d1f88e: Bump msbuild to get SDK updates from mono/msbuild#150 * mono/mono@a3f3bfc: Bump nuget to the latest suggested version * mono/mono@9bd3079: [2019-08] bump msbuild with more p2 packages * mono/mono@6ac1ff7: [dim][regression] Explicit interface override (#17583) (#17627) * mono/mono@a119807: [acceptance-tests] Bump mono/roslyn to get build fix * mono/mono@d234d34: [2019-08][merp] Install sigterm handler in EnableMicrosoftTelemetry * mono/mono@444a9a3: [2019-08] [merp] Introduce a new 'dump mode' that allows different signal behavior when dumping (#17568)
Changes: mono/mono@8946e49...062f0ab Context: mono/mono#17133 * mono/mono@062f0ab: [sdks] Use Xcode 11.2 stable version for iOS/Mac SDKs * mono/mono@e1ef774: [2019-08] Bump CoreFX to pickup corefx PR #367 to fix #17133. (#17622) * mono/mono@6d1f88e: Bump msbuild to get SDK updates from mono/msbuild#150 * mono/mono@a3f3bfc: Bump nuget to the latest suggested version * mono/mono@9bd3079: [2019-08] bump msbuild with more p2 packages * mono/mono@6ac1ff7: [dim][regression] Explicit interface override (#17583) (#17627) * mono/mono@a119807: [acceptance-tests] Bump mono/roslyn to get build fix * mono/mono@d234d34: [2019-08][merp] Install sigterm handler in EnableMicrosoftTelemetry * mono/mono@444a9a3: [2019-08] [merp] Introduce a new 'dump mode' that allows different signal behavior when dumping (#17568)
Is this still broke for Linux systems? |
Changes: mono/api-snapshot@fc50bc4...45a61d9 $ git diff --shortstat fc50bc4f...45a61d93 22 files changed, 775 insertions(+), 474 deletions(-) Changes: dotnet/cecil@a6c8f5e...a6a7f5c $ git diff --shortstat a6c8f5e1...a6a7f5c0 55 files changed, 818 insertions(+), 530 deletions(-) Changes: mono/corefx@1f87de3...49f1c45 $ git diff --shortstat e4f7102b...49f1c453 38 files changed, 1171 insertions(+), 419 deletions(-) Changes: dotnet/linker@ebe2a1f...e8d054b $ git diff --shortstat ebe2a1f4...e8d054bf 137 files changed, 5360 insertions(+), 1781 deletions(-) Changes: mono/mono@8946e49...18920a8 $ git diff --shortstat 8946e49a...18920a83 1811 files changed, 47240 insertions(+), 48331 deletions(-) Changes: xamarin/xamarin-android-api-compatibility@a61271e...50a3c52 $ git diff --shortstat a61271e0...50a3c52d 1 file changed, 2 insertions(+), 791 deletions(-) Fixes: #3619 Context: https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1005448 Context: https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems/edit/967582 Context: https://github.com/dotnet/coreclr/issues/26370 Context: https://github.com/dotnet/coreclr/issues/26479 Context: https://github.com/dotnet/corefx/issues/40455 Context: https://github.com/dotnet/corefx/issues/40578 Context: mono/mono#7377 Context: mono/mono#12421 Context: mono/mono#12586 Context: mono/mono#14080 Context: mono/mono#14725 Context: mono/mono#14772 Context: mono/mono#15261 Context: mono/mono#15262 Context: mono/mono#15263 Context: mono/mono#15307 Context: mono/mono#15308 Context: mono/mono#15310 Context: mono/mono#15646 Context: mono/mono#15687 Context: mono/mono#15805 Context: mono/mono#15992 Context: mono/mono#15994 Context: mono/mono#15999 Context: mono/mono#16032 Context: mono/mono#16034 Context: mono/mono#16046 Context: mono/mono#16192 Context: mono/mono#16308 Context: mono/mono#16310 Context: mono/mono#16369 Context: mono/mono#16380 Context: mono/mono#16381 Context: mono/mono#16395 Context: mono/mono#16411 Context: mono/mono#16415 Context: mono/mono#16486 Context: mono/mono#16570 Context: mono/mono#16605 Context: mono/mono#16616 Context: mono/mono#16689 Context: mono/mono#16701 Context: mono/mono#16712 Context: mono/mono#16742 Context: mono/mono#16759 Context: mono/mono#16803 Context: mono/mono#16808 Context: mono/mono#16824 Context: mono/mono#16876 Context: mono/mono#16879 Context: mono/mono#16918 Context: mono/mono#16943 Context: mono/mono#16950 Context: mono/mono#16974 Context: mono/mono#17004 Context: mono/mono#17017 Context: mono/mono#17038 Context: mono/mono#17040 Context: mono/mono#17083 Context: mono/mono#17084 Context: mono/mono#17133 Context: mono/mono#17139 Context: mono/mono#17151 Context: mono/mono#17180 Context: mono/mono#17278 Context: mono/mono#17549 Context: mono/mono#17569 Context: mono/mono#17665 Context: mono/mono#17687 Context: mono/mono#17737 Context: mono/mono#17790 Context: mono/mono#17924 Context: mono/mono#17931 Context: https://github.com/mono/mono/issues/26758 Context: https://github.com/mono/mono/issues/37913 Context: xamarin/xamarin-macios#7005
Changes: mono/api-snapshot@fc50bc4...45a61d9 $ git diff --shortstat fc50bc4f...45a61d93 22 files changed, 775 insertions(+), 474 deletions(-) Changes: dotnet/cecil@a6c8f5e...a6a7f5c $ git diff --shortstat a6c8f5e1...a6a7f5c0 55 files changed, 818 insertions(+), 530 deletions(-) Changes: mono/corefx@1f87de3...49f1c45 $ git diff --shortstat e4f7102b...49f1c453 38 files changed, 1171 insertions(+), 419 deletions(-) Changes: dotnet/linker@ebe2a1f...e8d054b $ git diff --shortstat ebe2a1f4...e8d054bf 137 files changed, 5360 insertions(+), 1781 deletions(-) Changes: mono/mono@8946e49...18920a8 $ git diff --shortstat 8946e49a...18920a83 1811 files changed, 47240 insertions(+), 48331 deletions(-) Changes: xamarin/xamarin-android-api-compatibility@a61271e...50a3c52 $ git diff --shortstat a61271e0...50a3c52d 1 file changed, 2 insertions(+), 791 deletions(-) Fixes: #3619 Context: https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1005448 Context: https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems/edit/967582 Context: https://github.com/dotnet/coreclr/issues/26370 Context: https://github.com/dotnet/coreclr/issues/26479 Context: https://github.com/dotnet/corefx/issues/40455 Context: https://github.com/dotnet/corefx/issues/40578 Context: mono/mono#7377 Context: mono/mono#12421 Context: mono/mono#12586 Context: mono/mono#14080 Context: mono/mono#14725 Context: mono/mono#14772 Context: mono/mono#15261 Context: mono/mono#15262 Context: mono/mono#15263 Context: mono/mono#15307 Context: mono/mono#15308 Context: mono/mono#15310 Context: mono/mono#15646 Context: mono/mono#15687 Context: mono/mono#15805 Context: mono/mono#15992 Context: mono/mono#15994 Context: mono/mono#15999 Context: mono/mono#16032 Context: mono/mono#16034 Context: mono/mono#16046 Context: mono/mono#16192 Context: mono/mono#16308 Context: mono/mono#16310 Context: mono/mono#16369 Context: mono/mono#16380 Context: mono/mono#16381 Context: mono/mono#16395 Context: mono/mono#16411 Context: mono/mono#16415 Context: mono/mono#16486 Context: mono/mono#16570 Context: mono/mono#16605 Context: mono/mono#16616 Context: mono/mono#16689 Context: mono/mono#16701 Context: mono/mono#16712 Context: mono/mono#16742 Context: mono/mono#16759 Context: mono/mono#16803 Context: mono/mono#16808 Context: mono/mono#16824 Context: mono/mono#16876 Context: mono/mono#16879 Context: mono/mono#16918 Context: mono/mono#16943 Context: mono/mono#16950 Context: mono/mono#16974 Context: mono/mono#17004 Context: mono/mono#17017 Context: mono/mono#17038 Context: mono/mono#17040 Context: mono/mono#17083 Context: mono/mono#17084 Context: mono/mono#17133 Context: mono/mono#17139 Context: mono/mono#17151 Context: mono/mono#17180 Context: mono/mono#17278 Context: mono/mono#17549 Context: mono/mono#17569 Context: mono/mono#17665 Context: mono/mono#17687 Context: mono/mono#17737 Context: mono/mono#17790 Context: mono/mono#17924 Context: mono/mono#17931 Context: https://github.com/mono/mono/issues/26758 Context: https://github.com/mono/mono/issues/37913 Context: xamarin/xamarin-macios#7005
Changes: mono/api-snapshot@6f14e43...8ea1d66 $ git diff --shortstat 6f14e433...8ea1d663 21 files changed, 719 insertions(+), 444 deletions(-) Changes: mono/boringssl@4ca62c5...d7b108e $ git diff --shortstat 4ca62c57...d7b108ee 1 file changed, 1 insertion(+) Changes: dotnet/cecil@cb6c1ca...a6a7f5c $ git diff --shortstat cb6c1ca9...a6a7f5c0 47 files changed, 587 insertions(+), 444 deletions(-) Changes: mono/corefx@10a41e9...5940515 $ git diff --shortstat 10a41e9f...59405155 55 files changed, 1382 insertions(+), 369 deletions(-) Changes: mono/mono@18920a8...2edccc5 $ git diff --shortstat 18920a83...2edccc52 1393 files changed, 44742 insertions(+), 90381 deletions(-) Context: https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1005448 Context: https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1005448 Context: KSP-CKAN/CKAN#2881 Context: mono/mono#10559 Context: mono/mono#12249 Context: mono/mono#12337 Context: mono/mono#12995 Context: mono/mono#13777 Context: mono/mono#15006 Context: mono/mono#15010 Context: mono/mono#15181 Context: mono/mono#15805 Context: mono/mono#15845 Context: mono/mono#16026 Context: mono/mono#16206 Context: mono/mono#16410 Context: mono/mono#16513 Context: mono/mono#16557 Context: mono/mono#16588 Context: mono/mono#16632 Context: mono/mono#16701 Context: mono/mono#16778 Context: mono/mono#17053 Context: mono/mono#17084 Context: mono/mono#17133 Context: mono/mono#17151 Context: mono/mono#17161 Context: mono/mono#17190 Context: mono/mono#17278 Context: mono/mono#17278 Context: mono/mono#17317 Context: mono/mono#17367 Context: mono/mono#17389 Context: mono/mono#17546 Context: mono/mono#17549 Context: mono/mono#17569 Context: mono/mono#17601 Context: mono/mono#17665 Context: mono/mono#17687 Context: mono/mono#17737 Context: mono/mono#17790 Context: mono/mono#17869 Context: mono/mono#17878 Context: mono/mono#17916 Context: mono/mono#17924 Context: mono/mono#17926 Context: mono/mono#17931 Context: mono/mono#18213 Context: mono/mono#18221 Context: mono/mono#18273 Context: mono/mono#18276 Context: mono/mono#18317 Context: mono/mono#18388 Context: mono/mono#18455
This is a continuation of xamarin/xamarin-android#3426, narrowed down to the problematic system call.
This pinpointed test case does not demonstrate a regression between Mono versions because it explicitly invokes the problematic system call itself. But the real user-facing scenario from xamarin/xamarin-android#3426 is a regression starting in Xamarin.Android 9.4 (mono/2019-02) because Xamarin.Android 9.3 (mono/2018-08) and earlier did not yet use
SystemNative_CopyFile()
.It looks like
SystemNative_CopyFile()
might need to add some special handling for the unusual way that permissions behave in the external storage directory on Android.Steps to reproduce
Test case: AndroidExternalStorageFchmod.zip
The core of the test case is:
The intention is to demonstrate the result of the
fchmod()
call inSystemNative_CopyFile()
:https://github.com/mono/corefx/blob/cb2c46b3f2b0097ea4ccb71de296b7760eeda26e/src/Native/Unix/System.Native/pal_io.c#L1340
Problematic behavior on Android 7.1 Nougat (API level 25) x86 or x86_64 Google APIs emulator
The
fchmod()
call returns-1
with anerrno
ofMono.Unix.Native.Errno.EPERM
, so the test case throwsUnauthorizedAccessException
.After you run the test case, you also can optionally confirm the permissions restriction by hand using
adb
:Open Tools > Android > Android Adb Command Prompt.
Run the following command:
C:\> adb shell ls -l /storage/emulated/0/Temp/*.txt
Result:
(Note that the files are owned by root rather than the app itself.)
Run the following command:
C:\> adb shell run-as com.companyname.androidapp1 chmod g+w /storage/emulated/0/Temp/out.txt
Result:
Non-problematic behavior on Android 9 Pie (API level 28) x86 Google APIs emulator
The
fchmod()
call returns 0, so the test case does not throw any exceptions.As expected, a similar
adb
command also completes successfully:C:\> adb shell run-as com.companyname.androidapp1 chmod g+w /storage/emulated/0/Temp/out.txt
Result: Success (no output).
On which platforms did you notice this
[x] Android 7.1 Nougat (API level 25) x86 and x86_64 Google APIs emulators
[x] Android 4.1 Jelly Bean (API level 16) armeabi-v7a LG Optimus L9
I could not reproduce the permissions problem on:
Version Used:
Xamarin.Android 9.4.0.51 (mono/2019-02@e6f5369c2d24aa2de511a52c8a58956c3283c4e4)
Xamarin.Android 10.0.0.43 (mono/2019-06@7af64d1ebe9)
The text was updated successfully, but these errors were encountered: