Offline map caches like zip or mbtiles should not be considered for cache trimming

[lukepot]

This issue is pretty much as the title states:
Using MOBAC, I initially created an mbtiles database that contains the tiles that I'm interested in displaying offline. The file is large, but not excessively so at ~1.3GB.
If I place this mbtiles database on the SD card at /sdcard/osmdroid/tiles it is immediately deleted the next time I start up osmdroid. If I use MOBAC to pack an OsmDroid ZIP archive, the zip is also deleted on the next startup of osmdroid.

The telltale clue I finally got with the mbtiles, after several red herrings, is a pair of lines from logcat (edited):
D OsmDroid: Trimming tile cache from 1358250840 to 524288000
and then eventually D OsmDroid: Finished trimming tile cache.
The next few lines on logcat spell out the fact that the SQLite cannot find the mbtiles file:
E SQLiteLog: (1802) os_unix.c:29136: (2) fstat (/sdcard/osmdroid/tiles/xxx.mbtiles)
E SQLiteLog: (1802) disk I/O error.
W OsmDroid: Error getting db stream: /17/76816/73878

Another thing I should probably mention is that I'm using a hybrid of online and offline tiles using the basic map tile provider as it provides the filesystem provider, the archive provider and the downloader provider without having to do any additional work. For the online tiles, I simply use Mapnik.

I can understand this being very helpful for purging the older / lesser used individual tiles from online caches, but surely if I'm supplying the bulk of the map in the form of a package, be it mbtiles or zip, I don't want that purged :)

At the moment, the work-around that I'm using is to just set the cache larger via OpenStreetMapTileProviderConstants.setCacheSizes() and setting the cache and trimming sizes a fair amount larger than I actually need.

[spyhunter99]

You're putting the database in the wrong spot. Try /sdcard/osmdroid/, not
the tiles folder
On Nov 28, 2015 9:50 AM, "TriptoKnocker" notifications@github.com wrote:

This issue is pretty much as the title states:
Using MOBAC, I initially created an mbtiles database that contains the
tiles that I'm interested in displaying offline. The file is large, but not
excessively so at ~1.3GB.
If I place this mbtiles database on the SD card at /sdcard/osmdroid/tiles
it is immediately deleted the next time I start up osmdroid. If I use MOBAC
to pack an OsmDroid ZIP archive, the zip is also deleted on the next
startup of osmdroid.

The telltale clue I finally got with the mbtiles, after several red
herrings, is a pair of lines from logcat (edited):
D OsmDroid: Trimming tile cache from 1358250840 to 524288000
and then eventually D OsmDroid: Finished trimming tile cache.
The next few lines on logcat spell out the fact that the SQLite cannot
find the mbtiles file:
E SQLiteLog: (1802) os_unix.c:29136: (2) fstat
(/sdcard/osmdroid/tiles/xxx.mbtiles)
E SQLiteLog: (1802) disk I/O error.
W OsmDroid: Error getting db stream: /17/76816/73878

Another thing I should probably mention is that I'm using a hybrid of
online and offline tiles using the basic map tile provider as it provides
the filesystem provider, the archive provider and the downloader provider
without having to do any additional work. For the online tiles, I simply
use Mapnik.

I can understand this being very helpful for purging the older / lesser
used individual tiles from online caches, but surely if I'm supplying the
bulk of the map in the form of a package, be it mbtiles or zip, I don't
want that purged :)

At the moment, the work-around that I'm using is to just set the cache
larger via OpenStreetMapTileProviderConstants.setCacheSizes() and setting
the cache and trimming sizes a fair amount larger than I actually need.

—
Reply to this email directly or view it on GitHub
#198.

[lukepot]

It strikes me as odd that it does work from the tiles directory, though.
With osmdroid 5.0.1, If I place the mbtiles file in /sdcard/osmdroid, my tiles are not shown - only if I place the mbtiles file in /sdcard/osmdroid/tiles do I see anything. If I revert back to osmdroid 4.3, the expected path of /sdcard/osmdroid works fine.

[spyhunter99]

They wouldn't be shown if if the tile source name given to osmdroid does not match the source name listed in the zip, sqlite, etc. I think mbtiles is not affected by this since they don't store the tile source name in the database.

You can also try using the OfflineTileProvider, which ignores the tile source entirely and just loads up whatever zip or database that you give it

[lukepot]

In MapTileArchiveProvider.java (in package org.osmdroid.tileprovider.modules)

private void findArchiveFiles() {

        mArchiveFiles.clear();

        if (!getSdCardAvailable()) {
            return;
        }

          // path should be optionally configurable
          File cachePaths = OpenStreetMapTileProviderConstants.TILE_PATH_BASE;
          final File[] files = cachePaths.listFiles();
          if (files != null) {
               for (final File file : files) {
                    final IArchiveFile archiveFile = ArchiveFileFactory.getArchiveFile(file);
                    if (archiveFile != null) {
                         mArchiveFiles.add(archiveFile);
                    }
               }
          }
    }

cachePaths is set to the tiles path to search for the archives / databases. If you change this to

File cachePaths = OpenStreetMapTileProviderConstants.OSMDROID_PATH;

then everything works as you previously described.

They wouldn't be shown if if the tile source name given to osmdroid does not match the source name listed in the zip, sqlite, etc. I think mbtiles is not affected by this since they don't store the tile source name in the database.

I have verified this by creating and then independently loading a zip file, an sqlite db, and an mbtiles db - as long as the source name is correct for whichever format you use, everything is fine. I can also verify that the .mbtiles file name must match the source name in order for it to be displayed.

[spyhunter99]

The cache path typically maps to /sdcard/osmdroid/tiles where as the OSMDROID_PATH is /sdcard/osmdroid.

I'm confused and I'm not too sure what the issue is at this point. Is the root problem that sqlite, zip, etc are loaded from within /sdcard/osmdroid/tiles/? or that your database was deleted because it was in /sdcard/osmdroid/tiles/?

[lukepot]

Apologies, I think I've morphed the problem after digging through the code.

The problem is two-fold:

1. The original problem: Databases and zips don't load from anywhere but /sdcard/osmdroid/tiles. If you do place them there and they are are larger than the cache size boundaries, the databases / zips are deleted by the cache trimmer.
2. Following from point 1, we both agree that the databases and zips should be stored in /sdcard/osmdroid and not in the tiles directory.

In the code block I posted in my previous post, I noticed that the archive loader defaults to loading the archives / databases from the tiles directory. Using the fix detailed in my previous post should fix both problems:

1. The archives / databases will no longer have to be placed in the tiles directory, and won't be trimmed when they exceed the cache.
2. The archives / databases will be loaded from the expected directory as per the old osmdroid 4.3 behaviour (specifically, /sdcard/osmdroid/xxx.zip or /sdcard/osmdroid/xxx.sqlite)

Sorry for being an insufferable pedant :)

[spyhunter99]

ahh i see the problem...

[spyhunter99]

Looks like it was me that buggered it up back in august. I pushed the changes to branch #198. Would you mind testing it on your end?

[lukepot]

I've given it a reasonable test - everything now works as expected. Thanks!

[spyhunter99]

merged. thanks for testing!