android 11 again: permission denied while receiving files on the phone

[qorron]

Introduction: If you have issues with this fork, feel free to post them. I'll look into them if a good reproducer or log is provided.

Description of the issue

I'm using the version linked here because of the reasons described in the other bug report:
#704 (comment)
things are working good, syncthing keeps shoveling my pictures off the phone to the private storage.
I shared another directory that should be kept in sync over many devices, data goes to and fro this time.
but file will not roll in. directories are created though.

Reproducer

have a folder X on device B (any) share it with device A (Android 11, oneplus 8 in my case).
put a new file in X on B.
wait for it to show up on A
A gets permission denied.

actual error message from the web gui on the phone:
syncing: opening temp file: open /storage/emulated/0/Bunny Squad Common/Murmeltier/.syncthing.murmel_2020-12-24T04:59:06.mp3.tmp: operation not permitted
Notice: the directory was created, the file was not. unix permissions seem to be in order: 660/770, root:everybody

GF's oneplus 8, kept on android 10 does not show this problem.

Version Information

App Version: 1.12.0.0
Syncthing Version: v1.12.0-preview.3
Android Version: Android 11
Device manufacturer: Oneplus 
Device model: 8 (256gig)

Device platform info

adb shell "getprop | grep ro.product"

[ro.product.board]: [kona]
[ro.product.brand]: [OnePlus]
[ro.product.build.date]: [Thu Nov 19 01:17:19 CST 2020]
[ro.product.build.date.utc]: [1605719839]
[ro.product.build.fingerprint]: [OnePlus/OnePlus8_EEA/OnePlus8:11/RP1A.201005.001/2011190109:user/release-keys]
[ro.product.build.id]: [RP1A.201005.001]
[ro.product.build.tags]: [release-keys]
[ro.product.build.type]: [user]
[ro.product.build.version.incremental]: [2011190109]
[ro.product.build.version.release]: [11]
[ro.product.build.version.release_or_codename]: [11]
[ro.product.build.version.sdk]: [30]
[ro.product.cpu.abi]: [arm64-v8a]
[ro.product.cpu.abilist]: [arm64-v8a,armeabi-v7a,armeabi]
[ro.product.cpu.abilist32]: [armeabi-v7a,armeabi]
[ro.product.cpu.abilist64]: [arm64-v8a]
[ro.product.device]: [OnePlus8]
[ro.product.first_api_level]: [29]
[ro.product.locale]: [en-US]
[ro.product.manufacturer]: [OnePlus]
[ro.product.model]: [IN2013]
[ro.product.name]: [OnePlus8_EEA]
[ro.product.odm.brand]: [OnePlus]
[ro.product.odm.device]: [OnePlus8]
[ro.product.odm.manufacturer]: [OnePlus]
[ro.product.odm.model]: [IN2013]
[ro.product.odm.name]: [OnePlus8_EEA]
[ro.product.product.brand]: [OnePlus]
[ro.product.product.device]: [OnePlus8]
[ro.product.product.manufacturer]: [OnePlus]
[ro.product.product.model]: [IN2013]
[ro.product.product.name]: [OnePlus8_EEA]
[ro.product.property_source_order]: [odm,vendor,product,system_ext,system]
[ro.product.system.brand]: [OnePlus]
[ro.product.system.device]: [OnePlus8]
[ro.product.system.manufacturer]: [OnePlus]
[ro.product.system.model]: [IN2013]
[ro.product.system.name]: [OnePlus8_EEA]
[ro.product.system_ext.name]: [qssi]
[ro.product.vendor.brand]: [OnePlus]
[ro.product.vendor.device]: [OnePlus8]
[ro.product.vendor.manufacturer]: [OnePlus]
[ro.product.vendor.model]: [IN2013]
[ro.product.vendor.name]: [OnePlus8_EEA]
[ro.product.vndk.version]: [30]

Android Log

adb logcat v
or MatLog (search on GitHub) output

this should be the relevant part:

12-28 13:30:06.580  8651  8651 W libsyncthingnat: type=1400 audit(0.0:132622): avc: denied { bind } for scontext=u:r:untrusted_app:s0:c244,c257,c512,c768 tcontext=u:r:untrusted_app:s0:c244,c257,c512,c768 tclass=netlink_route_socket permissive=0 b/155595000 app=com.github.catfriend1.syncthingandroid
12-28 13:30:06.911  2442  2442 E EventProcessor: onItemFinished: Error "opening temp file: open /storage/emulated/0/Bunny Squad Common/Murmeltier/.syncthing.murmel_2020-12-25T04:59:07.mp3.tmp: operation not permitted" reported on file: /storage/emulated/0/Bunny Squad Common/Murmeltier/murmel_2020-12-25T04:59:07.mp3
12-28 13:30:06.913  2442  2442 E EventProcessor: onItemFinished: Error "opening temp file: open /storage/emulated/0/Bunny Squad Common/Murmeltier/.syncthing.murmel_2020-12-26T04:59:06.mp3.tmp: operation not permitted" reported on file: /storage/emulated/0/Bunny Squad Common/Murmeltier/murmel_2020-12-26T04:59:06.mp3
12-28 13:30:06.916  2442  2442 E EventProcessor: onItemFinished: Error "opening temp file: open /storage/emulated/0/Bunny Squad Common/Murmeltier/.syncthing.murmel_2020-12-27T08:16:59.mp3.tmp: operation not permitted" reported on file: /storage/emulated/0/Bunny Squad Common/Murmeltier/murmel_2020-12-27T08:16:59.mp3

[Catfriend1]

Hi, two questions ahead please.

• Does https://github.com/Catfriend1/syncthing-android/releases/download/v1.12.0.1/com.github.catfriend1.syncthingandroid_github_v1.12.0.1_d38f56f6.apk or https://github.com/Catfriend1/syncthing-android/releases/download/v1.12.1.0/com.github.catfriend1.syncthingandroid_github_v1.12.1.0_89e90b55.apk work better?

• Is the phone rooted and/or did you grant the alle files access?

[qorron]

the device is not rooted. all permissions are granted. (incl. all filesystem access)
unfortunately, none of the APKs worked. all show the same behavior.

[Catfriend1]

Ok, thanks for the follow up. I could try it again on the emulator during a future dev session. But no promise I could solve this, because the last emulator tests worked (as I posted on the Syncthing Forum a while ago). Unfortunately, I have no physical Android 11 device here to test a manufacturer specific implementation if it works.

[qorron]

I think, I have some new data:

some new files added to B are showing up on A. but the old ones don't.
history: I forgot to [x] ignore permissions on B when I created this.
files that are new in a directory I created after I ignored permissions on B show up on A.
they do this even if I move them in a dir that was created earlier.
on ht eother hand.. I can remove, re-add, rename the dir around on B it won't change anything on A. it gets renamed but the files won't show up. I eben deleted the dir on A and re-shared it from B to A. old files (the ones that already were on A) show up instantly, missing files from B don't. syncthing's caching mechanism is just too good. I can't seem to trick it into forget everything and just pull the files in without permissions.

ok, further experiments done:
when I touch a file it shows up.
when I cp a non-working file it shows up (mv does not do the trick)
files that do show up are generated by crond which calls a perl script which does come 'curl ... -o not_working_file.txt'
there is nothing odd with permissions when I do a 'ls -l'

I guess this ticket isn't app-related after all. shall I close it?

[Catfriend1]

Hm yeah, thanks anyways for the follow up. For a moment, I thought about database reset but better avoid this if you have another way to revive sync again without data loss. If the permission handling has to be improved, that's related to SyncthingNative, you could then head for forum.syncthing.net to get tips and tricks from users or file a bug.

[qorron]

solution is found, problem is solved:
https://forum.syncthing.net/t/permission-denied-on-syncthing-tmp-files-on-android/16096/5

tl;dr
android 11 apparently now uses a table to allocate files because it doesn't like colons in filenames any more.

[Catfriend1]

Thanks!

[jtagcat]

Could a better error message be added [without days of significant effort]?
I spent half an hour on trying to figure out, what the 'permission denied for .tmp files' meant (directory (with colon!) was created on the phone, yet files within it, without colons couldn't get downsynced).

[Catfriend1]

The wrapper displays what the SyncthingNative gives him via RestApi / EventsApi. I think a better error message should be emitted by the native, so Android and PC users can see it and not only users of this wrapper.

[oliver]

By the way, file names containing pipe characters (|) can cause this problem as well.

[Catfriend1]

@oliver Don't use special chars and feel safe :-)

[bxff]

Is there a way to solve this other than avoiding using special chars as I already heavily use them, and I am kinda of forced to with syncing Node libraries, I have many issues related this this syncing between my phone(S20U) and my laptop(MBP). Some of the examples are Vue Testing/testing/node_modules/.bin/parser, which I am not certain why is the case. I have certain examples with Node.js? which is also not parsed. Currently I am looking at using a solution such as Crytometer to encrypt and send files over syncing, but would love to know if there is a good fix yet.

[Catfriend1]

Well, this is not a Syncthing but a filesystem issue. If you're able to use a filesystem or mount point that supports special characters it might be okay, but its not officially supported by Android and any root tricks and modifications go beyond this project.

[guillaume-uH57J9]

Exact same problem here with "operation not permitted" on tmp files that contains date and time information (eg foobar_2020-12-24T04:59:06.ext).
Filenames are automatically generated based on current date and time, so I didn't realize there was any special character.
I scratched my head, searched the error online, and finally found this bug ticket that suggest a workaround (ie removing character ':').

It would be nice if syncthing was able to account for operating system limitation w.r.t filenames, in particular ':' and '|' on Android 11.
Syncthing's manpage give an example on Windows, where brackets '[' and ']' are forbidden. But the doc recommand ignoring the files (cf .stignores), which is a workaround more than a proper solution.

This was already suggested in #1734.