Android: Scoped storage [To be merged in October]

[JosJuice]

From November 2021 and on, Google Play won't allow us to release updates for Dolphin unless we increase the app's targetSdkVersion to 30 (i.e. Android 11). This PR makes that change. On a normal year, that we're increasing the targetSdkVersion isn't something noteworthy for users, but this time it is, because setting the targetSdkVersion to 30 will force all new installs of Dolphin on Android 11 and up to use scoped storage.

Before we continue, let me explain what scoped storage is so that everyone is on the same page. Android users who have updated to Android 11 or bought a new phone with Android 11 pre-installed have experienced that some emulators which do not use the Storage Access Framework (including older versions of Dolphin) are no longer able to list files on SD cards, regardless of what the app's targetSdkVersion is. Some users have drawn the conclusion that this problem with SD cards is what scoped storage is, but that isn't correct. Scoped storage is when an app is unable to read any file at all outside its designated app-specific directory unless the app uses the Storage Access Framework (or in the case of media files, certain other APIs, but the kinds of files we are dealing with are not media files). Scoped storage is applied only in the following cases:

• The operating system is Android 10 or newer and the app's targetSdkVersion is 29 or higher and the app does not set requestLegacyExternalStorage
• The operating system is Android 11 or newer and the app's targetSdkVersion is 30 or higher and either the app does not set preserveLegacyExternalStorage, or scoped storage was active for the app at the time it was installed, or this is a new install

Considering that this logic and the value of targetSdkVersion aren't exposed to users in any meaningful way, can you blame them for thinking that the SD card problem was caused by scoped storage even though scoped storage actually wasn't active for the affected apps? Not really.

Now, with that out of the way, let's talk about what consequences scoped storage will have for Dolphin once it's actually applied. I've already made us handle WAD files, SD card images and games with the Storage Access Framework in pull requests #8962, #9221 and #9318 respectively, so those will keep working the same way with scoped storage applied as they are working now. The biggest remaining obstacle to adopting scoped storage is the user directory. Currently, the user directory is always a directory stored in the root of external storage with the name dolphin-emu. With scoped storage, we will not be able to access any location other than the app-specific directory created for us by Android unless we use the Storage Access Framework, and making our C++ code support the Storage Access Framework for everything stored in the user directory (there's a lot of different things handled by different pieces of code in there!) would both be a very big undertaking and would likely lead to severe performance problems in some situations. So we will have to use the app-specific directory as the user directory. Even if we could use the Storage Access Framework, the app-specific directory would have to be the default directory anyway unless we want to force the user to pick a directory before they can start using the app.

(To be clear, each app has one app-specific directory in internal storage and one app-specific directory in external storage. All my mentions of "the app-specific directory" refer to the one in external storage – we don't want to use the internal one because it has the huge drawback of not being user accessible.)

The kinds of things that we're storing in the user directory are in fact exactly the kinds of things that Google thinks should go in the app-specific directory – settings and save files and other app-specific data. One big reason why Google wants apps to use the app-specific directory is to keep the file system neat and tidy, instead of having apps placing their data in whatever random place they feel like, which I think is completely reasonable. However, the app-specific directory comes with something that can be considered both an advantage and a disadvantage: It is deleted when the app is uninstalled. In some cases this makes sense, as you usually don't want data from apps you uninstall to take up space anymore, but it can get annoying if you for instance need to briefly uninstall Dolphin in order to install an older version of Dolphin (for bisecting or whatever). In the grand scheme of things, I guess having to use the app-specific directory isn't too bad.

...or at least it wouldn't be if it wasn't for yet another restriction added in Android 11! In Android 11, regardless of the targetSdkVersion, apps can no longer access the app-specific directories of other apps. This includes file manager apps. According to Google, this is for privacy or security or whatever, but I find it frustrating that you can't override this even if both the user and both involved apps consent to it. It's as if apps that want to store data locally and have the user be in full control of the data aren't on the radar of whatever team at Google is making these decisions... Anyway, it's still possible to access app-specific directories on external storage by connecting your device to a PC using USB, and it seems like there's a file manager in the Android system settings that still lets you access them, so I guess it's manageable even though it's annoying and probably will cause some confusion with users. (The file manager in the system settings doesn't let me open INI files directly 🙁)

Either way, we will have to use the app-specific directory as the user directory when scoped storage is active, like it or not. But we can still provide the option of using the old dolphin-emu directory for users who don't have scoped storage applied to Dolphin, either because they're using an older version of Android or because they installed Dolphin before installing Android 11 or before updating Dolphin to a version that includes the change made in this pull request. The way I've implemented it is that if the app-specific directory is empty and the dolphin-emu directory exists and scoped storage is not active, the dolphin-emu directory will be used. Otherwise the app-specific directory will be used. This ensures that all existing dolphin-emu directories keep working (as long as you don't uninstall Dolphin and are using Android 11 or newer...), but makes new users use the app-specific directory unless they intentionally go through the effort of creating a dolphin-emu directory, because I think it's less confusing to make as many people as possible get used to the new behavior at once instead of having a transitional period where new users experience different things based on what version of Android they have. As a bonus: You can now reject Dolphin's request for external storage permissions that pops up when starting the app, and things will work fine anyway (albeit without the ability to use the dolphin-emu directory). If scoped storage is active, the permission request won't even pop up!

So, with the app-specific directory in use as the user directory, does everything in Dolphin work with scoped storage? Not quite, unfortunately. The settings Dump Path, Load Path, Resource Pack Path, and Wii NAND Root are not compatible with the Storage Access Framework and thus can't be changed from their defaults when using scoped storage. Just like I mentioned earlier with why it isn't realistic to let the user set a custom user directory with the Storage Access Framework, using the Storage Access Framework for these settings would both require a bit of extra work to implement and would likely cause performance problems. In this PR, I've included a commit that makes us show an explanatory pop-up dialog if the user tries to change one of these paths while scoped storage is active. I won't rule out the possibility that I actually end up implementing Storage Access Framework support for at least some of these settings in a separate PR – if I do, I can remove the pop-up for those settings when that separate PR gets merged. But I wouldn't count on the performance being reasonable enough that we would want to offer it as an option for users.

I will merge this pull request in October 2021. No sooner and no later. If you want to review it, make sure to do so before then.

[JosJuice]

I've added one more thing: A screen where the user can check what directory is currently being used as the user directory. Unfortunately I don't think it's possible to add an "open folder" button, because Android doesn't really have any "open folder" intent.

[mbc07]

About the "User Data" menu, it would be nice if a link to directly launch the system file manager could be added to that screen when using scoped storage. I haven't thoroughly investigated this but the system file manager seems to always use the same package name (com.android.documentsui), so this should be doable...

[JosJuice]

Oh, you mean it would be possible to create an intent for that activity specifically (rather than any activity in general which can display a folder)? I'll try to look into that.

[mbc07]

This seems to work from ADB Shell:
am start -n com.android.documentsui/.LauncherActivity

[mbc07]

So, I was messing around with a Samsung Galaxy device updated to Android 11 with their OneUI 3.0 interface and found out they buried the native file manager very deep. It's simply impossible to open it from the settings app, as it launches Samsung's own file manager instead, which can't access the Android folder despite being a system app.

Galaxy users need to resort to Activity Launcher or to ADB shell whenever they want to access Android's native file manager. Considering Galaxy devices respond for a big chunk of the Android smartphone market, this is mildly concerning and probably another good reason to allow launching DocumentsUI directly from the new User Data screen from Dolphin when applicable.

(on the device I tested -- a Galaxy A71 -- the package still is named com.android.documentsui and the .LauncherActivity still exists, despite being hidden)

[JosJuice]

What the hell?! Ugh...

Yeah, I really should look into adding a button for opening LauncherActivity then. I wish I didn't have to, though.

[JosJuice]

@mbc07 com.android.documentsui/.LauncherActivity does not exist on my device, but these ones do:

com.android.documentsui/.ScopedAccessactivity
com.android.documentsui/.ViewDownloadsActivity
com.android.documentsui/.files.FilesActivity
com.android.documentsui/.picker.PickActivity
com.android.documentsui/.selection.demo.SelectionDemoActivity

Do any of those exist on your device? com.android.documentsui/.files.FilesActivity seems to work for me when launched through Activity Launcher, though I haven't gotten it to work from inside Dolphin yet (probably a bug in my code).

[JosJuice]

I've made it use com.android.documentsui/.files.FilesActivity for now. This is what it looks like:

[mbc07]

I mismatched the activities on my previous comment, sorry. On newer Android versions (including Samsung on Android 11 and OneUI 3) the activity indeed is com.android.documentsui/.files.FilesActivity. The com.android.documentsui/.LauncherActivity is on my old device (Xperia X, stock-ish Android 8.0), which isn't affected by Scoped Storage anyways...

[JosJuice]

I've additionally made it so that the button doesn't show up if you're using an older OS than Android 11. (Users on older versions of Android don't need the button for anything anyway.) I tried to make it so that the button would be hidden depending on whether the activity actually existed, but I didn't manage to write any code for that that worked... Maybe that's for the better, so that in 5 years or whenever this activity ends up getting removed from Android, people will be less confused about why the button is suddenly gone.

[JosJuice]

I will merge this October 15th if there are no objections.

[ghost]

What day/month this change will include on google play store update?

[JosJuice]

Tentatively the beginning of November, but it depends on the progress report.

[ghost]

how about this now? 10/16/21 today in my country.