Use "Storage Access Framework" to allow storing maps on external sdcard

[cryptomilk]

On Android 5.0+ you need to use the Storage Access Framework to allow an app to store data on an external sdcard.

An app correctly implementing this is OpenCamera (GPL), see their FAQ section. With that app storing data on the sdcard simply works.

This will fix #3904 and #3790.

[vshcherb]

I don't think it is a working solution:

1. We need to have a NDK support to operate with C++ code
2. We need to iterate over all files
   3A) We need to have more or less permanent access in order to avoid unexpected problems with user have no maps at all
   3B) We need to get proper messages to display to user in case something is not operating normally.

[cabroncito]

I love OsmAnd (F-Droid) for its offline maps, I use it frequently in many occasions and even contribute to openstreetmap in order to improve the OsmAnd maps. However, it's a real pain in the donkey (substitute with synonym) that OsmAnd cannot write to SD cards [for years already!] in LineageOS (also didn't work on a stock Sony Xperia XA OS) - instead, these massive maps use up all of my scarce internal memory. Please, do something about this permission thing.

[Zahnstocher]

@cabroncito
I don't understand your issue.
On my devices (Android P and Android O) OsmAnd is able to use the external SD card.
"Storage Access Framework" (SAF) is not necessary, if you use the designated app path on the external SD card, on my devices it is /storage/<SD_card_id>/Android/data/net.osmand.plus/files.
This path is writable without SAF.
You should be able to select this path in OsmAnd:
Settings -> General settings -> Data storage folder -> External storage 2
If it has a different name on your device, select the storage location on your external SD card which contains /data/net.osmand/files (for OsmAnd) or /data/net.osmand.plus/files (for OsmAnd+).
According the APK name the F-Droid version should be net.osmand.plus.

But be aware, if you use this designated app path, all OsmAnd files will be deleted, if you clear OsmAnd app data or uninstall OsmAnd.
You can avoid the deletion of these files, if you unmount the SD card or just rename the folder /storage/<SD_card_id>/Android/data/net.osmand.plus/files for example to /storage/<SD_card_id>/Android/data/save_net.osmand.plus/files before you clear or uninstall OsmAnd.

[Zahnstocher]

@vshcherb
Android Q will enforce SAF also for internal storage, if the target API level is set to 29 (Android Q), except the app-specific directory (.../net.osmand*/...).
This affects "Shared memory" OsmAnd storage setting.
https://www.xda-developers.com/android-q-storage-access-framework-scoped-storage/

[cryptomilk]

@Zahnstocher That might be because your ROM altered the sdcard or media_rw permissions to assign it a gid which allows apps to write. However this is a hack and not the way Google anticipated this. They tell everyone to use the Storage Access Framework. As an example OpenCamera impelements storage with SAF and it works to store pictures on the sdcard.

I've just tried to put my maps on: /storage/<SD_card_id>/Android/data/net.osmand.plus/files and I get access denied as it doesn't use SAF on Android 9.0.

[Zahnstocher]

@cryptomilk

That might be because your ROM altered the sdcard or media_rw permissions to assign it a gid which allows apps to write.

I don't think that my SD card or media_rw permission is altered.

They tell everyone to use the Storage Access Framework.

The app should always be able to access the app-specific directory without SAF.

Also Android docs explicitly states that:

getExternalStorageDirectory
This method was deprecated in API level 29.
To improve user privacy, direct access to shared/external storage devices is deprecated. When an app targets Build.VERSION_CODES.Q, the path returned from this method is no longer directly accessible to apps. Apps can continue to access content stored on shared/external storage by migrating to alternatives such as getExternalFilesDir(String), MediaStore, or Intent#ACTION_OPEN_DOCUMENT.

https://developer.android.com/reference/android/os/Environment.html#getExternalStorageDirectory()

getExternalFilesDir returns the app-specific directory path /storage/<SD_card_id>/Android/data/net.osmand.plus/files, which can only be used with direct file access (not SAF).
https://developer.android.com/reference/android/content/Context.html#getExternalFilesDir(java.lang.String)

[cryptomilk]

Tell me how to turn on debug logs for osmand and then I can tell you why it doesn't work on my device.

[Zahnstocher]

@cryptomilk

Tell me how to turn on debug logs for osmand and then I can tell you why it doesn't work on my device.

I don't think that there is a setting to turn on debug.
You may look at logcat, maybe there are some hints?

By the way, I'm not able to reproduce this issue on Android emulator with official Google Android P and Q images:
OsmAnd (F-Droid) is able to read/write to the app-specific directory "External storage 2" (/storage/<SD_card_id>/Android/data/net.osmand.plus/files) without SAF.
But the access to any other directory on the external SD card "/storage/<SD_card_id>/" is denied.
It even works, if I revoke the storage permission for OsmAnd.
The app-specific directory should always be accessible (read and write) by the correspondent app without SAF and without any permissions.
This is exactly how it is intended by Google and how it also works on my devices.

I guess something is special with your ROM?

[vshcherb]

@Zahnstocher I'm concerned it will be even more cumbersome unfortunately especially for people who will run out of default path. Though even default path will be cumbersome.
So I don't know how 2 OsmAnd versions will be able to share same directory which was designed to be from day 0. Cause having shared maps / favorites export is great to have and also takes much less disk space

[Zahnstocher]

@vshcherb Google Play requires targetSdkVersion 28 for app updates by November 1 2019, thus it will very likely be 29 or above one year later.
https://support.google.com/googleplay/android-developer/answer/113469#targetsdk

If OsmAnd targets Android API 29, I think the only way to share the same directory (for maps / favorites) will be to use SAF.

A hint for NDK:
With SAF you get file/uri, which can be converted to a native fd:
int fd = getContentResolver().openFileDescriptor(file.getUri(), "w").getFd();
https://developer.android.com/reference/android/os/ParcelFileDescriptor.html#getFd()

In native C++ code you can open this fd:
FILE* file = fdopen(fd, ...);

You may find some SAF examples:
https://github.com/Jpub/Androidstudio3/blob/master/StorageDemo/app/src/main/java/com/ebookfrenzy/storagedemo/StorageDemoActivity.java

[cabroncito]

@Zahnstocher Thanks for replying!!
I am sort of an Android noob, so please forgive me if my descriptions are kind of weird. I'll try to describe the issue I, and, as far as I understood, many other LineageOS users, experience.
So I bought a Sony Xperia XZ1 compact, unlocked the bootloader, installed TWRP and finally LineageOS 15 (beta 7, later beta 8) according to these instructions:
https://forum.xda-developers.com/xperia-xz1-compact/development/lineageoslilac-t3745215
and finally updated to LOS 16 (RC1 first, and now RC2):
https://forum.xda-developers.com/xperia-xz1-compact/development/rom-lineageos-16-0-unofficial-todo-t3925675
Now, I do (as I did before with LOS15) the following:

1. restart phone
2. check OsmAnd (version 3.3.8 now) permissions via settings --> Apps --> OsmAnd: "Permissions: Location and Storage", detailed "modify or delete the contents of your SD card" and "read the contents of your SD card" are all shown to be enabled.
3. Start OsmAnd (F-Droid version installed to internal memory, 13 maps and 4 Wikipedia compendia installed).
4. Open up Settings --> General Settings --> Data storage folder --> edit (=pencil). Folder is set to "External storage 1"; That's the internal memory /storage/emulated/0/Android/data/net.osmand.plus/files, OsmAnd is working as expected.
5. I change the folder to "external storage 2" and being asked about 'Move OsmAnd data files to the new destination', which is /storage/FE81-BFCC/Android/data/net.osmand.plus/files, 25 GB free memory (more than enough). I tap "move maps" and get instantly a message "Moved 1 files (0 kB). I tap "App Restart", but nothing happens.
6. Same thing happens if I choose "external storage 3" (my SDcard has 2 FAT32 partitions). Now if I choose "multiuser storage 1", data is actually moved (223 files, 4617.9 MB to /storage/emulated/0/Android/obb/net.osmand.plus) and "App Restart" indeed restarts the app.
7. Choosing "multiuser storage 2" (or 3) is the same as "external storage 2" (or 3).

On a Sony Xperia XA (Stock ROM) about 1 year ago I installed F-Droid and then OsmAnd, set the map path to sdcard (only 1 partition FAT32) and downloaded a map, which, after finishing the download, did not appear in Osmand. I repeated the download several times, tried with different maps - nothing worked; until I changed the map data folder to internal. Then with the very next download the map was usable.
Then I moved the map to the respective OsmAnd folder on the SDcard with the "simple file manager" app (which correctly asks for write permission on the SDcard): OsmAnd is still usable, however the map cannot be updated by OsmAnd.

Other users who are reporting write permission issues with SDcards with the same phone/LOS:
https://forum.xda-developers.com/search/thread/3925675?query=sdcard
https://forum.xda-developers.com/search/thread/3745215?query=osmand
and several google hits for "osmand write permission sdcard"

If I remember correctly, especially Sony users have been experiencing problems with OsmAnd and SDcards.

If of any help, I can run some more tests next week with a Xiaomi Redmi Note 4 (Mediatek CPU, Android 6 Stock ROM), Unihertz Jelly Pro (Android 7 Stock ROM) and a Sony Xperia Z2 tablet (AICP ROM based on Pie).

[Zahnstocher]

@cryptomilk I just noticed that you are the LineageOS - Maintainer for Sony Xperia XZ1 Compact! 👍
A XDA user may have an explanation for the different behavior of LineageOS:
https://forum.xda-developers.com/showpost.php?p=79753264&postcount=193
To verify this, I made the emulator system image writable and removed encryptable=userdata from the SD card line (fstab).
After rebooting the emulator, OsmAnd was not able to access the "external storage 2" anymore, but the SD card is still available and mounted.
Maybe you may try to add encryptable=userdata on your device to check if this is the issue?

[cryptomilk]

@Zahnstocher There is no encryptable=userdata in the fstab, at least not in RC2, see https://github.com/cryptomilk/android_device_sony_yoshino/blob/lineage-16.0/config/init/fstab.yoshino

Moving the maps fails, no logcat error or warning.

[Zahnstocher]

@cryptomilk

There is no encryptable=userdata in the fstab, at least not in RC2, see https://github.com/cryptomilk/android_device_sony_yoshino/blob/lineage-16.0/config/init/fstab.yoshino

The problem might be that encryptable=userdata is missing in the fstab.
If you add encryptable=userdata back to the fstab, the app-specific directory should be writable again.
According to the investigation of the XDA user encryptable=userdata has been removed, which may cause this problem:
cryptomilk/android_device_sony_yoshino@73e044a

[cryptomilk]

I've added encryptable=userdata to the fstab and this doesn't fix the problem. It might be if you change the sdcard to adoptable storage but this needs kernel support which is only available on latest kernels. I don't know a device which offers that yet.

[scaidermern]

I wonder what makes your device different from others. I've never had these problems on LineageOS so far.

[cryptomilk]

@scaidermern Does your device have an external micro sdcard? If yes, which device is it?

[scaidermern]

Yes, it has. It is a Xiaomi A1 but I also used a Motorola G4 Play and a Samsung S4 with LineageOS and OsmAnd maps on an external SD card before.

[cryptomilk]

In Android 6.0, any device that is not adopted is considered portable. Because portable storage is connected for only a short time, the platform avoids heavy operations such as media scanning. Third-party apps must go through the Storage Access Framework to interact with files on portable storage; direct access is explicitly blocked for privacy and security reasons.

https://source.android.com/devices/storage/traditional

For adoptable storage, the sdcard needs to be encrypted. However that is not that easy with File-Based Encryption. The Xiaomi A1 is using FDE so probably you have adopted your sdcard and because of that it works ...

[cryptomilk]

The Storage Access Framework is the only way for apps to work with all your files in Android Q.
...
And developers will have to substantially recode apps to support it.

https://www.xda-developers.com/android-q-storage-access-framework-scoped-storage/

[vshcherb]

I think it is related only to external sdcard users, so with Android Q the default option where maps are stored in a specific folder for application will work as before.
SAF approach is already used to import maps / tracks when you click on the in browser or email.

[Lemmiwinks]

@cryptomilk : I still don't get it, there must be a reason why this does not work with LineageOS on the G8441... Any new insights?

[vshcherb]

I finally managed to go through it and I think this topic will be closed as having contradictive information.

1. On new Android devices it will be possible to use sdcard cause each app will have own location on the external sdcard as of today.
2. On new Android devices it might be not possible to use shared storage.
3. We can't use SAF (storage access framework) cause most of the code requires browsing list of the maps, indexing and etc. We can't call for each start a function and ask user to pick up a map to use it.

Main: There no much benefit of Shared System if Android Q will stop supporting it and we don't use it by default. Main benefit is to not store Maps in 2 locations (GPX files are lightweight and another solution could be applied).

The easiest option as it looks will be acquiring root on device and symlink folders.

[vshcherb]

Work in progress - 759b2d8#diff-830bf84e0e2639b27123090a5c2871f45009f1f27e93b49bddefb8bac017106aR211

We will also need to

• migrate C++ using FileDescriptor
• need to check if Sqlite maps are working
• Random access file to search POI / Address / init files

[mmarczell-graphisoft]

@vshcherb Based on my understanding, the ongoing work linked in the above diff is based on a wrong assumption that the File APIs are still available for files located in folders accessed via the Storage Access Framework. Specifically this method:

public static File toRawFile(@NonNull DocumentFile doc) {

will not work on Android 11.

[vshcherb]

It was tested on Android 11 emulator. Not all API available but read API still works

[Pastim]

What is the situation now with Android 11? Will there be an improvement in performance at some point?

I have to have my maps on SD card because of space constraints. Downloading and indexing maps does work, but is very slow. Opening OSMAND takes a minute or two before the map is visible, which is a daily nuisance.

I can't go back to Android 10 just for this, and don't wish to root for all the usual reasons.

If there is no future for OSMAND I'd like to know so I can start looking for other options. I really don't want to, since OSMAND has been my main mapping and navigation tool for several years now.

[vshcherb]

Situation is quite ambiguous, we will need to put a significant effort to implement it and I think it couldn't be expected earlier than 2021 Q3

[Pastim]

Situation is quite ambiguous, we will need to put a significant effort to implement it and I think it couldn't be expected earlier than 2021 Q3

Thanks for the quick response. If it's in the pipeline I'm happy.

I was really quite surprised by the related changes in Android 11. I had loads of trouble sorting out access to my SD card. I'm sure many apps have had problems with it.

[Pastim]

This seems to be fixed as of 21st July. There were some system updates this morning, after which OSMAND+ wouldn't run, so I reinstalled it and it now starts up more quickly. Thanks to whoever has done the necessary work.

[vshcherb]

#12046 There is a discussion going on related to Samsung devices on Android 11 with SDcard which represent > 50% of the following use cases.
Short summary: we don't see any performance drawback as for now with storage like /storage/XXXXX-XXXX/osmand and /storage/XXXXX-XXXX/Android/data/net.osmand.plus/files

[Pastim]

Thanks.

[Tombstone2K]

@vshcherb .
I have proposed a solution to this in another thread. Please have a look -> #16059 (comment)