-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Android 12 scoped storage hell #11997
Comments
It seems there is a way to delay it even when targeting Q: But presumably not for R. -[Unknown] |
Would PPSSPP be able to do something like, on API levels up to 28 (or 29 with the flag @unknownbrackets references above) use the default structure. For 29+ with scoped storage, use This will allow PPSSPP to, via I recommend looking at DocumentFile though I'd recommend against using it directly. Its performance is not very great =( The other folders, including MUSIC, PICTURE, and VIDEO seem like they could be replaced with using MediaStore when inside scoped storage, or you could use a similar method to the "PSP" directory to get persistent access to those as well. I'm guessing you already know about Hopefully this is a bit of help. |
Well, it seems more complicated than that. Feel free to correct me if I'm misunderstanding. For example, PSP games will try to call sceIoLseek on returned PSP file handles. They may do something like this:
Those are just example offsets, but we can't deal with the concept that we "might" get a file descriptor that doesn't support seeking. So the experience probably becomes:
Unless I'm missing some way to filter the "trees", this will at best be a terrible experience, and at worst be straight up unusable on some devices. The alternative is to try to hack around the limitations in the document API. For example, if a file is opened for read and write, read the whole thing in, then open it again for writing and wait for the game to write (possibly with seeking.) There will be tons of bugs, save data corruption, and other issues. Sounds like some kind of nightmare. As long as we get a real file descriptor, then we should be fine I guess for saving. It's the path to get there that's the problem. Given all those problems, I assume we won't be making users select the PSP folder by default, and instead use the hidden away app-private folder. Then we need to handle "installing" gamefaqs zips of saves, and probably need a way to transfer data in out. Hopefully one that doesn't make PPSSPP require network for you to play a single-player game that autosaves. See #1019. And that's just about saves. Haven't even gone into how people will transfer ISOs onto their device, or navigate to homebrew. It has many of the same problems, although at least ISOs are read only (they absolutely require seeking, though.) So - yes. It can be worked around. As long as everything is device-local storage, it's not even a big technical problem. It's just going to be very confusing for anyone who doesn't understand storage really well. -[Unknown] |
Alright, so some news! https://developer.android.com/preview/privacy/storage has been updated with the changes for Android 11. We have a reprieve, in that as long as we only target Android 10, we can still opt out! So that pushes the issue to the day when Google Play will no longer accept apps targeting 10, or I guess when 12 comes out. So we can probably stumble along another year or so without changes. There's also a new MANAGE_EXTERNAL_STORAGE permission, that will be very cumbersome to explain to users how to enable though, which makes things easier for us if enabled. There's also support to use raw paths to read "external media libraries" whatever that is, might possible be useful? Either way, no longer an emergency but I should really start preparing :P |
Oops, this wasn't meant to be closed by that. -[Unknown] |
Bumping this to the next version, since we still have quite some time left... |
Alright, the final deadline has arrived, and it's November 2021. https://android-developers.googleblog.com/2020/11/new-android-app-bundle-and-target-api.html We need to find workaround and solutions for this before then, otherwise we won't be able to make new releases on Google Play. And if Android 12 is released before then, we'll also be in trouble there because they probably won't keep allowing 10-targeted apps to access storage freely. I plan to do the work early spring. |
According to this https://stackoverflow.com/questions/60360368/android-11-r-file-path-access
Well we'll still need to deal with Android Q |
There's still the "guess again, idiot" problem where users will be able to select folders (i.e. Dropbox) that don't support seeking. Example: a user will put an ISO on their Dropbox account. Then PPSSPP will ask where their ISOs are, and pop up a dialog with their Dropbox account in the list. However, selecting that account won't work. We won't have a File to back it, presumably. -[Unknown] |
I'm gonna use this comment as a notepad, adding findings as I find them. High level concernsWe will eventually need to configure PPSSPP to target Android 12, which will enforce Scoped Storage all the way back to Android 10. These devices need a nice migration path. Our use of on-device storageThere are two main things to deal with: Picking and loading ISO files
The PSP/ directory, containing saves, config, everything else
Current release planRelease PPSSPP 1.12 which will do the following:
Release PPSSPP 1.13 which does the following:
|
Hm, I thought I saw issues for some people - not on all devices - not having access to this specific path via USB storage. Could've misunderstood for sure. If that consistently works, that's a lot better (even if hard to find.)
We currently put "memstick_dir.txt" inside the "data" directory that is not normally accessible. Then we load the ini from that folder. I think this is better because then the ini is at least as accessible as the other files, and it's a straight-forward experience across platforms. -[Unknown] |
@nic0lette Turns out |
I'm comparing to direct network communication between a wifi mobile device and a wired network storage, which I've actually tested the latency of. My tests were mostly on 600 (N) and 1900 (AC) connections. But yes, if you use cache iso in RAM it may be fine (although 1.8 GB is a big cost on a 6 GB device, especially when that 6 GB is VRAM too.) Online services are likely even worse indeed, but I assume any online service will ultimately cache reads and buffer writes in some way. Though for ISOs, maybe that's a bad thing if it has to download the whole thing before reading any of it.
Well, that's not good. If we can still get seekable fds from files, but only through Java, it should be doable without crazy bugs. But it'll be a ton of work. For games, a lot could live in the async io manager, I guess, but we'd probably want to abstract that out... -[Unknown] |
I wonder how this gonna affect Android TV support? |
That is indeed a good question. @nic0lette will the ACTION_OPEN_DOCUMENT_TREE dialog be fixed in Android TV? I heard before that it doesn't work (though haven't tried it lately). |
From what I've found, no, it doesn't look like ACTION_ OPEN_DOCUMENT_TREE is supported on Android TV. |
@nic0lette that is unfortunate and potentially a huge problem. Why? Is it going to change? |
Probably only Nvidia Shield TV support it https://commonsware.com/blog/2017/12/27/storage-access-framework-missing-action.html On the issue tracker mentioned:
|
That post is old, and Android TV devices that old won't be affected anyway since they can just keep using the old methods of accessing storage. What I'm concerned about is Android 10 and Android 11 TV devices, once we change our target version of Android to 11 or later which will make scoped storage / SAF mandatory for them. |
I added a comment to https://issuetracker.google.com/issues/169471812 . |
Ah I thought it gonna affect ones before Android 10 (like mine which uses 9) but I it seems like its not. |
I think we can preserve data on Android TV installs, and from other existing installs on other devices, by using preserveLegacyExternalStorage when targeting Android 11 or higher. This allows the previous permission model until the next uninstall, so we'll simply take the opportunity and offer the user to migrate all data to the allowed external directory (android/Data/org.ppsspp.ppsspp) if we detect that we can still access the old PSP directory. Unfortunately this still has the problem that data will disappear if the user then uninstalls the app, which some users might not be used to. But it's just the way it will be. We can potentially offer users on non-TV devices to use a more permanent directory for memstick storage using ACTION_OPEN_DIRECTORY_TREE but not sure if worth the trouble and confusion.... |
Just documenting some path formats, trying to understand content URIs. We probably should not rely on the formats of these strings anyway, instead only use full strings as returned from APIs, but this is a bit impractical when browsing a directory subtree that we have full permissions for anyway, we'll need so many redundant file listings... (notes on URL encodings: Browsed tree uri: One level deeper: So effectively these are files in the "tree" "primary:PSP ISO". The divider between the "tree" specifier and the internal path specifier appears to be simply "/document/". Makes me wonder what happens if you have a directory called "document", as previously mentioned, which is not really hard to imagine - that has to be a bit of a design flaw. I suppose you know that it'll be followed by "primary" if there was a "primary" after tree though, so maybe it's not as ambiguous as I first though. Still, yuck. |
I don't believe it actually is ambiguous. A content URI has two or four components, with the first component always being either "tree" or "document" and the third one always being "document" (if there are more than two components). To determine whether something is a tree and/or a document, Android specifically checks the first and third component. It doesn't scan through all components looking for one that says "document". |
Ooh I see, the slashes are URL-encoded away inside the component, and the slashes separate the components. Right.. |
Alright, managed to add some code in #14232 that improves file system navigation below tree URIs (can now go upwards). So that's a tiny bit of progress... |
@nic0lette A somewhat unrelated issue but I know you're watching :) https://issuetracker.google.com/issues/163120692?pli=1 is a critical bug where accessibility settings and apps breaks game controllers on Android. There seems to be no movement on this, but it even breaks Stadia on some devices. Can you confirm whether the team is aware of it and has a fix for Android 12? In other news I'm making progress on this, we're probably gonna end up with a workable experience it seems, even though things won't be as smooth as it used to. |
As discussed on Discord, some ideas about the root directory issue: You can't select the root storage, you must select a subfolder. For new users, this isn't much of an issue. They can create a new folder with any name, or potentially we could encourage them to name it PSP (we don't really care, but existing tutorials around the Internet might be easier to follow if they used "PSP".) For existing users, this means they would be selecting PSP. However, PPSSPP would normally set this as the root of the memory card, which includes PSP itself, and can potentially include other files. Some specific examples of things outside PSP:
I've marked the likely most concerning cases, but even those are definitely edge cases. So what we might do is the following:
I suppose we could handle this by having a separate To avoid confusion it might be best if this behavior was cross platform (i.e. even on Windows, if you had My Documents/PPSSPP /SYSTEM/ppsspp.ini.) That way, if someone does something on Android and it works, they won't get confused when it works differently on another device. -[Unknown] |
Great summary! Can also, on Android, check the name of the root folder so we don't end up with PSP/PSP , if neither file exists - good for first time setup. |
@nic0lette I have another bug to report in the folder browser. After you create a folder in it using the New Folder button, the button "Use this folder" at the bottom of the screen doesn't appear immediately. You have to navigate out of it and back in again. In the latest update for my Pixel, the previous issue seems to be fixed at least (where you can't use the folder it's opened at). |
Is there an issue for this? (Could you open in this component if not?)
The team is aware of it, but this is outside the teams I communicate with (Android is big) and so other than I don't know what's going on. Sorry. (My suggestions would be to add a comment to the bug, or +1 it at least, and then add yourself in the 'cc' to get updates?) |
I'm actually having trouble reproducing the new folder bug. Maybe it was just some wacky transient thing .. So I guess let's forget about that one, unless I have it happen again - in which case I'll investigate properly. Right, I know Android is big :) I added a comment. |
This is now pretty much resolved with #14619 merged. There may still be a few issues remaining, of course, but we'll fix them one by one. Closing, finally. |
Android Q will introduce "scoped storage", which imposes really heavy restrictions on external storage. And in the version after Q, it will be a requirement for all apps, even those who target Android-28 (which PPSSPP will do as long as possible). So we will be able to limp along for a while longer without it, but next year, new devices will start requiring it, and it'll greatly worsen the user experience. We need to find ways to make it as workable as possible.
For us, the filesystem really is part of the UX to match the real PSP with an USB connection.
Users will additionally have problems if they install the app on an older OS, upgrade to Q, and then downgrade to an older version of the app (which is done sometimes to bisect bugs, etc).
The one bright point is that the scoped storage APIs allows getting file descriptors so I/O performance will at least not be hampered much - assuming the file descriptors are unrestricted with regards to seek etc.
https://android-developers.googleblog.com/2019/04/android-q-scoped-storage-best-practices.html
EDIT: I have requested a reprieve through AndroidManifest.xml, the hammer drops with Android 12.
The text was updated successfully, but these errors were encountered: