File access and Android 11+ #1743
Comments
|
for starters, we can make a new config manager class (or adjust the existing one) that handles api 30+ and only looks in app private storage, bypassing the storage utils class. This should resolve a lot of issues people are having with the current setup. |
|
For starters, I think the system already in place (osmdroid locating its cache directory in the app private storage) is the good solution. The issue is much more for advanced users, wanting to manage the all kinds of offline content supported by osmdroid (huge work behind that!):
|
Unless you copy the file in the app private storage. Is it possible under the new rules to have to end-user select a file wherever and read it? (in order to copy it) |
|
My understanding of the new rules:
|
|
I guess we must slalom around each details: there's no way the new rules would prevent an end-user to select a picture in the Gallery and save it in the app private storage. |
|
Unfortunately, there is not way for a user to select a picture in the Gallery and save it in the app private storage... |
|
Really? How do you add your profile pic to an app like whatsapp? |
|
Whatsapp is certainly using the SAF framework, which exists since Android 4.4. |
That's what I had in mind: the end-user selecting the file. That looks OK for the read part, doesn't it? |
|
Yes, as long as you replace everything based on java.io.File by something based on InputStream. |
|
I meant to say that it was possible, not quickly done. I understood that if you invoke the file picker, as the end-user is involved, you're in a grace period long enough to copy the file. But you cannot reuse the file path later for instance. Disclaimer: I have only an old Lollipop smartphone, and I only care about "standard" files like pictures and MP3. But my tests with API30 on emulators are OK (so far). |
|
That's a way. Drawbacks:
A point: osmdroid method to choose its cache directory do not warranty the same place at each launch. The offline files may be copied to a directory which will not be used later. I was more in favor of a real move to SAF. Note that the app can reuse later a file which has been authorized once. With additional complications after a reboot. |
|
Mapsforge team already worked on supporting SAF: mapsforge/mapsforge#1186 |
|
@MKergall Fair enough. Actually I don't use offline maps; I was just reacting to your rather alarming OP.
I'm not sure it's correct: maybe the very first time, but then the path is stored in the preferences and is reused the next time. To be checked. |
I just checked: no storage of the path in preferences. |
|
@MKergall Maybe we're talking about different things, but this is what I see in https://github.com/osmdroid/osmdroid/blob/master/osmdroid-android/src/main/java/org/osmdroid/config/DefaultConfigurationProvider.java#L328: if (!prefs.contains("osmdroid.basePath")) {
//this is the first time startup. run the discovery bit
File discoveredBasePath = getOsmdroidBasePath(ctx);
// ...
} else {
//normal startup, load user preferences and populate the config object
setOsmdroidBasePath(new File(prefs.getString("osmdroid.basePath", getOsmdroidBasePath(ctx).getAbsolutePath()))); |
|
Oh, OK, thanks. I never noticed the introduction of this Configuration.getInstance().load method, so I wasn't calling it. |
|
I'm attempting to fix the Android 10+ Storage Model issue on my own using the SimpleStorage third-party package, which makes extensive use of the AndroidX DocumentFile class. Perhaps we can start using the Because the If the /**
* to scan for folder paths that match "tiles" to find OSMDroid database files
*
* @return maps a set of DocumentFiles, each store the found folder path
*/
protected Set<DocumentFile> findCacheFiles(@NotNull Context context, DocumentFile path) {
Set<DocumentFile> maps = new HashSet<>();
if (path != null) {
Regex regex = new Regex("^.*tiles.*$");
String[] mimetypes = {MimeType.UNKNOWN};
List<DocumentFile> list = DocumentFileUtils.search(path, true, DocumentFileType.FOLDER, null, "", regex);
LOG.log(Level.INFO, "OSM GPKG file list size: " + list.size());
list.stream().forEach((c) -> {
LOG.log(Level.INFO, "OSM GPKG file item: " + DocumentFileUtils.getAbsolutePath(c, context));
});
maps.addAll(list);
}
return maps;
}/**
* to scan for file paths that match "*.gpkg" to find GeoPackage files
*
* @return maps a set of DocumentFiles, each store the found file path
*/
protected Set<DocumentFile> findMapFiles(@NotNull Context context, DocumentFile path) {
Set<DocumentFile> maps = new HashSet<>();
if (path != null) {
Regex regex = new Regex("^.*gpkg$");
String[] mimetypes = {MimeType.UNKNOWN};
List<DocumentFile> list = DocumentFileUtils.search(path, true, DocumentFileType.FILE, null, "", regex);
LOG.log(Level.INFO, "OSM GPKG file list size: " + list.size());
list.stream().forEach((c) -> {
LOG.log(Level.INFO, "OSM GPKG file item: " + DocumentFileUtils.getAbsolutePath(c, context));
});
maps.addAll(list);
}
return maps;
}//Assume the default Android public Documents folder at "/storage/emulated/0/Documents"
//set OSMDroid default folder at "/storage/emulated/0/Documents/APP/maps"
Stream<String> mapTree = Arrays.asList(SimpleStorage.getExternalStoragePath(), PublicDirectory.DOCUMENTS.getFolderName(), Constants.APP_NAME, "maps").stream();
String mapPathTree = mapTree.collect(Collectors.joining(File.separator));
Configuration.getInstance().setOsmdroidBasePath(new File(mapPathTree));
LOG.log(Level.INFO, "OSM GPKG baseDir created: " + Configuration.getInstance().getOsmdroidBasePath(MainApp.context()));
Boolean hasAccess = SimpleStorage.hasStorageAccess(MainApp.context(), mapPathTree);
LOG.log(Level.INFO, "OSM GPKG SimpleStorage.hasStorageAccess("+mapPathTree+"): " + hasAccess);
DocumentFile osmdroidDefaultFolder = DocumentFileCompat.fromFullPath(MainApp.context(), mapPathTree);
Set<DocumentFile> tileCachefiles = findCacheFiles(MainApp.context(), osmdroidDefaultFolder);
File [] tileCaches = tileCachefiles.stream().map((c)-> DocumentFileUtils.toRawFile(c, MainApp.context())).toArray(File[]::new);
if (tileCaches.length >= 1) {
LOG.log(Level.INFO, "OSM GPKG cacheDir found: " + tileCaches[0].getAbsolutePath());
Configuration.getInstance().setOsmdroidTileCache(tileCaches[0]);
} else {
Configuration.getInstance().setOsmdroidTileCache(new File(mapPathTree + "/tiles"));
LOG.log(Level.INFO, "OSM GPKG cacheDir created: " + Configuration.getInstance().getOsmdroidTileCache(MainApp.context()));
}
//now, save changes to the base and cache database path
Configuration.getInstance().save(MainApp.context(), PreferenceManager.getDefaultSharedPreferences(MainApp.context()));
Set<DocumentFile> mapfiles = findMapFiles(MainApp.context(), osmdroidDefaultFolder);
File [] maps = mapfiles.stream().map((c)-> DocumentFileUtils.toRawFile(c, MainApp.context())).toArray(File[]::new);
LOG.log(Level.INFO, "OSM GPKG maps size " + maps.length);
if (maps.length == 0 && hasAccess) {
//show a warning that no map files were found...
}
surfaceView = view.findViewById(R.id.drawingOsmSurfaceView);
if (surfaceView != null) {
geoPackageProvider = new GeoPackageProvider(maps, this.getContext());
// do whatever we want with GeoPackage files...
}The app will output the following logs if the user has granted the appropriate storage permission access... Otherwise, UPDATE (2021-09-23): The raw I believe the OSMDroid library should provide full SAF support via |
|
I found this article by @CommonsWare about undocumented Documents support in The point to underline here is that OSMDroid may need to figure out how to establish the cache database as well as to read and write folders and files that aren't in the app's "private" folders, such as the shared public Documents folder. NOTES:
Stream<String> baseTree = Arrays.asList(Constants.APP_NAME, "maps").stream();
String basePathTree = baseTree.collect(Collectors.joining(File.separator));
LOG.log(Level.INFO, "OSM GPKG basePathTree: " + basePathTree);
Stream<String> docTree = Arrays.asList(PublicDirectory.DOCUMENTS.getFolderName(), basePathTree).stream();
String docPathTree = docTree.collect(Collectors.joining(File.separator));
LOG.log(Level.INFO, "OSM GPKG docPathTree: " + docPathTree);
//Assume the default Android public Documents folder at "/storage/emulated/0/Documents"
//set OSMDroid default folder at "/storage/emulated/0/Documents/APP/maps"
Stream<String> mapTree = Arrays.asList(SimpleStorage.getExternalStoragePath(), docPathTree).stream();
String mapPathTree = mapTree.collect(Collectors.joining(File.separator));
LOG.log(Level.INFO, "OSM GPKG mapPathTree: " + mapPathTree);
LOG.log(Level.INFO, "OSM GPKG mapPathTree DocumentFileCompat.doesExist(mapPathTree): " + DocumentFileCompat.doesExist(MainApp.context(), mapPathTree));
if (Build.VERSION.SDK_INT >=29 )
{
String filename = "test.txt";
ContentValues contentValues = new ContentValues();
contentValues.put(MediaStore.Files.FileColumns.DISPLAY_NAME, filename);
contentValues.put(MediaStore.Files.FileColumns.MIME_TYPE, MimeType.TEXT);
contentValues.put(MediaStore.Files.FileColumns.DATE_ADDED, System.currentTimeMillis()/1000);
contentValues.put(MediaStore.Files.FileColumns.DATE_MODIFIED, System.currentTimeMillis()/1000);
contentValues.put(MediaStore.Files.FileColumns.RELATIVE_PATH, docPathTree); // "Documents/APP/maps"
contentValues.put(MediaStore.Files.FileColumns.IS_PENDING, true);
Uri uri = getContext().getContentResolver().insert(MediaStore.Files.getContentUri(MediaStore.VOLUME_EXTERNAL), contentValues);
if (uri != null) {
LOG.log(Level.INFO, "OSM GPKG MediaStore uri: " + uri);
try {
OutputStream out = getContext().getContentResolver().openOutputStream(uri);
out.write("Hello\n".getBytes(StandardCharsets.UTF_8));
out.close();
LOG.log(Level.INFO, "OSM GPKG MediaStore written into: " + uri);
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
//let's verify the file exists and get its properties!
//do note that it is not a raw file but a file that contains "content://" URI scheme!
DocumentFile testFile = DocumentFileCompat.fromUri(MainApp.context(), uri);
if (testFile != null) {
LOG.log(Level.INFO, "OSM GPKG testFile.getUri(): " + testFile.getUri());
LOG.log(Level.INFO, "OSM GPKG testFile.exists(): " + testFile.exists());
LOG.log(Level.INFO, "OSM GPKG testFile.isDirectory(): " + testFile.isDirectory());
LOG.log(Level.INFO, "OSM GPKG testFile.isFile(): " + testFile.isFile());
LOG.log(Level.INFO, "OSM GPKG testFile.getParentFile(): " + testFile.getParentFile());
LOG.log(Level.INFO, "OSM GPKG testFile.canRead(): " + testFile.canRead());
LOG.log(Level.INFO, "OSM GPKG testFile.canWrite(): " + testFile.canWrite());
}
}
contentValues.clear();
contentValues.put(MediaStore.Files.FileColumns.IS_PENDING, 0);
getContext().getContentResolver().update(uri, contentValues, null, null);
if (uri != null) {
//let's open this file as raw file instead by using the actual raw file path (should have "file://" URI scheme)
Stream<String> fileTree = Arrays.asList(mapPathTree, filename).stream();
String filePathTree = fileTree.collect(Collectors.joining(File.separator));
LOG.log(Level.INFO, "OSM GPKG filePathTree: " + filePathTree);
LOG.log(Level.INFO, "OSM GPKG filePathTree DocumentFileCompat.doesExist(filePathTree): " + DocumentFileCompat.doesExist(MainApp.context(), filePathTree));
DocumentFile rawFile = DocumentFileCompat.fromFullPath(MainApp.context(), filePathTree);
if (rawFile != null) {
LOG.log(Level.INFO, "OSM GPKG rawFile.getUri(): " + rawFile.getUri());
LOG.log(Level.INFO, "OSM GPKG rawFile.exists(): " + rawFile.exists());
LOG.log(Level.INFO, "OSM GPKG rawFile.isDirectory(): " + rawFile.isDirectory());
LOG.log(Level.INFO, "OSM GPKG rawFile.isFile(): " + rawFile.isFile());
LOG.log(Level.INFO, "OSM GPKG rawFile.getParentFile(): " + rawFile.getParentFile());
LOG.log(Level.INFO, "OSM GPKG rawFile.canRead(): " + rawFile.canRead());
LOG.log(Level.INFO, "OSM GPKG rawFile.canWrite(): " + rawFile.canWrite());
try {
byte[] bytes = new byte[(int) rawFile.length()];
InputStream in = getContext().getContentResolver().openInputStream(uri);
in.read(bytes, 0, (int) rawFile.length());
in.close();
LOG.log(Level.INFO, "OSM GPKG read from rawFile: " + DocumentFileUtils.getAbsolutePath(rawFile, MainApp.context()));
LOG.log(Level.INFO, "OSM GPKG content in rawFile: " + new String(bytes));
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}With the Allow management of all files option enabled, The codes successfully created and read the test.txt file from the shared public Documents folder, resulting in the following logs:
|
|
When the mTileProviderList.add(MapTileProviderBasic.getMapTileFileStorageProviderBase(pRegisterReceiver, pTileSource, tileWriter));
geopackage = new GeoPackageMapTileModuleProvider(databases, pContext, tileWriter);If the Storage permission is set to "Allow access to media only" or "Allow management of all files", the following will happen (Android 11 on Samsung A51): I suspect this problem will also affect the GeoPackageFactory class in the Geopackage Android library. |
|
public boolean importGeoPackage(DocumentFile file);I'm not sure if other offline tile database frameworks support |
|
I'm not familiar with the git system (I'm only familiar with SVN), so I just made a unified diff since HEAD patch for the osmdroid-geopackage update to version 6.0.2, which adds support for the geopackage-update-to-6.0.2.zip (updated with a note below) As a result, we can pass //using anggrayudi/SimpleStorage API
DocumentFile osmdroidDefaultFolder = DocumentFileCompat.fromUri(getContext(), uri);
/**
* to scan for file paths that match "*.gpkg" to find GeoPackage files
*
* @return maps a set of DocumentFiles, each stores the found file path
*/
protected Set<DocumentFile> findMapFiles(@NotNull Context context, DocumentFile path) {
Set<DocumentFile> maps = new HashSet<>();
if (path != null) {
Regex regex = new Regex("^.*gpkg$");
String[] mimetypes = {MimeType.UNKNOWN};
List<DocumentFile> list = DocumentFileUtils.search(path, true, DocumentFileType.FILE, null, "", regex);
maps.addAll(list);
}
return maps;
}
Set<DocumentFile> mapfiles = findMapFiles(getContext(), osmdroidDefaultFolder);
DocumentFile[] mapDocs = mapfiles.stream().toArray(DocumentFile[]::new);
geoPackageProvider = new GeoPackageProvider(mapDocs, getContext());
boolean sourceSet = false;
List<GeopackageRasterTileSource> tileSources = geoPackageProvider.geoPackageMapTileModuleProvider().getTileSources();
if (!tileSources.isEmpty()) {
surfaceView.setTileProvider(geoPackageProvider);
surfaceView.setTileSource(tileSources.get(0));
surfaceView.zoomToBoundingBox(tileSources.get(0).getBounds(), true);
surfaceView.getController().setZoom(tileSources.get(0).getMinimumZoomLevel());
sourceSet = true;
}
// Get a manager
GeoPackageManager manager = GeoPackageFactory.getManager(getContext());
// Available databases
List<String> databases = manager.databases();It's worth noting that the file extension from the filename must be omitted when comparing a |
|
Hello, Because today, it's getting critical for any apps to upload to Google Play Thanks ! |
|
@mguignes There really isn't a fix for this issue. The closest I have gone is to do something like this: ryanbehr@13e5fbe where you can request permission to access a directory and provide the DocumentFile to the MBTiles file to OSMDroid. After that it will copy the document file into a temporary app storage, then access that file using the usual "File" method. It works great for smaller MBTiles, but if you're using 50 GB MBTiles files, it's rough. |
|
@MKergall , have you solved the issue? I work on a completely different project, but have a similar issue and it got even worse with Android 13. I think all android developers should unite and send a petition to Google to reconsider current approach. It didn't make Android safer, but killed many good apps. So far, Android clearly loses to iOS specifically for bad designed API. |
|
I solved it on important sections of my own app (OsmNavigator). |
From Android 11, Google introduced drastical constraints on file access.
In short, an app cannot access any file, except files located in its "private" directory, directory which is not accessible to other apps, including File Explorers.
And you cannot publish an app on Google Play anymore, if the app is not targeting Android 11.
This raises various issues for apps using osmdroid features.
Good news: If you let osmdroid define its default cache directory, it will target the app "private" directory => usual file read/write is allowed, online tile sources will work fine.
Bad news:
Offline tiles: outside the app (using File Explorer), you cannot drop files in this cache directory.
MapsForge maps: outside the app (using File Explorer), you cannot put MapsForge files in a directory than could be read by the app.
Geopackage maps: same issue.
Workaround: you could request MANAGE_EXTERNAL_STORAGE permission. But again, when publishing your app on Google Play, you can anticipate that it will never be approved (trust me, I tried...).
The "good" direction defined by Google for file access is based on SAF framework: forget java.io.File class...
I think this subject needs investigations and suggestions for solutions.
Issue Type
[X] Bug / Improvement
If it's a bug, version(s) of android this affects:
Android 11+ (API Level 30 and above)
Version of osmdroid the issue relates to:
At least up to 6.1.11
The text was updated successfully, but these errors were encountered: