Handling Device Low Disk Storage #435

Closed
InI4 opened this Issue Oct 6, 2016 · 10 comments

Comments

Projects
None yet
3 participants
@InI4

InI4 commented Oct 6, 2016

This is actually kind of a feature request. I am observing below exception on an emulator, but it could well be on a real world device and I wonder, if osmdroid could do some checking of free memory or at least cleanup, when below the exception occurs:

10-06 04:27:01.710 7978-9239/de.spieleck.app.badgers.debug E/SQLiteDatabase: Error inserting expires=1475812252000 key=47358738 tile=[B@27aea82 provider=Mapnik android.database.sqlite.SQLiteFullException: database or disk is full (code 13) at android.database.sqlite.SQLiteConnection.nativeExecuteForLastInsertedRowId(Native Method) at android.database.sqlite.SQLiteConnection.executeForLastInsertedRowId(SQLiteConnection.java:780) at android.database.sqlite.SQLiteSession.executeForLastInsertedRowId(SQLiteSession.java:788) at android.database.sqlite.SQLiteStatement.executeInsert(SQLiteStatement.java:86) at android.database.sqlite.SQLiteDatabase.insertWithOnConflict(SQLiteDatabase.java:1471) at android.database.sqlite.SQLiteDatabase.insert(SQLiteDatabase.java:1341) at org.osmdroid.tileprovider.modules.SqlTileWriter.saveFile(SqlTileWriter.java:133) at org.osmdroid.tileprovider.modules.MapTileDownloader$TileLoader.loadTile(MapTileDownloader.java:230) at org.osmdroid.tileprovider.modules.MapTileModuleProviderBase$TileLoader.run(MapTileModuleProviderBase.java:296) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588) at java.lang.Thread.run(Thread.java:818)

@neilboyd neilboyd added the enhancement label Oct 6, 2016

@spyhunter99

This comment has been minimized.

Show comment
Hide comment
@spyhunter99

spyhunter99 Oct 18, 2016

Collaborator

#448 related issue

and roger this. Great idea. What should the expected behavior be then?

Maybe on start up, set the cache max size to the current size + free disk space. This should force the trim mechanism however this is only triggered on application startup. As such, we should probably do some trickery to have the vacuum functions run periodically. It should be quick enough to attach to every tile insert event.

Collaborator

spyhunter99 commented Oct 18, 2016

#448 related issue

and roger this. Great idea. What should the expected behavior be then?

Maybe on start up, set the cache max size to the current size + free disk space. This should force the trim mechanism however this is only triggered on application startup. As such, we should probably do some trickery to have the vacuum functions run periodically. It should be quick enough to attach to every tile insert event.

@spyhunter99 spyhunter99 changed the title from Handling Device Low Storage Memory to Handling Device Low Disk Storage Oct 22, 2016

spyhunter99 added a commit that referenced this issue Oct 23, 2016

#435 low disk space. this resolves a number of problems with the sam…
…ple app + the library not being disk space aware. Also fixes a bug in the sqlitetilewriter's clean up mechanism for trimming. Preferences for developers to turn on and off the debug settings without recompiling the sample app. User selection of cache location and how to work with reconfiguring the cache settings. This kind of highlights that we need to rework osmdroid to have some kind of configuration class to manage all the settings, storage locations and to bounce sqlite writers and more

spyhunter99 added a commit that referenced this issue Oct 23, 2016

#435 low disk space. this resolves a number of problems with the sam…
…ple app + the library not being disk space aware. Also fixes a bug in the sqlitetilewriter's clean up mechanism for trimming. Preferences for developers to turn on and off the debug settings without recompiling the sample app. User selection of cache location and how to work with reconfiguring the cache settings. This kind of highlights that we need to rework osmdroid to have some kind of configuration class to manage all the settings, storage locations and to bounce sqlite writers and more

@spyhunter99 spyhunter99 added this to the v5.6 milestone Oct 23, 2016

@spyhunter99

This comment has been minimized.

Show comment
Hide comment
@spyhunter99

spyhunter99 Oct 23, 2016

Collaborator

@InI4 might want to look at these changes. could always use another set of eyes

Collaborator

spyhunter99 commented Oct 23, 2016

@InI4 might want to look at these changes. could always use another set of eyes

@InI4

This comment has been minimized.

Show comment
Hide comment
@InI4

InI4 Oct 23, 2016

I had a glance at the code and it seems, that it is trying to do some async database cleaning on startup and on exception?! Vacuum still commented out. But looks like a step in the right direction.

I need to merge this into my (local) branch for unclear fixes of #411 and then build it into my product. I will keep you posted.

InI4 commented Oct 23, 2016

I had a glance at the code and it seems, that it is trying to do some async database cleaning on startup and on exception?! Vacuum still commented out. But looks like a step in the right direction.

I need to merge this into my (local) branch for unclear fixes of #411 and then build it into my product. I will keep you posted.

@InI4

This comment has been minimized.

Show comment
Hide comment
@InI4

InI4 Oct 23, 2016

"That" emulator stays calm now. Can it be, that some memory to database code was also helping, tile files are gone now.

InI4 commented Oct 23, 2016

"That" emulator stays calm now. Can it be, that some memory to database code was also helping, tile files are gone now.

@spyhunter99

This comment has been minimized.

Show comment
Hide comment
@spyhunter99

spyhunter99 Oct 23, 2016

Collaborator

The vacuum command apparently isn;t supported by sqlite. But basically on start up, it will purge expired tiles, which may or may not be desirable. The old tile system writer did the same task on startup so i just mirrored the logic. Now i'm thinking the logic may be wrong.

On low disk space it will trim, also on every insert, it will check the database size and trim if needed. I also added a bit of code to auto resize the max database size based on the available free space on the storage media.

Collaborator

spyhunter99 commented Oct 23, 2016

The vacuum command apparently isn;t supported by sqlite. But basically on start up, it will purge expired tiles, which may or may not be desirable. The old tile system writer did the same task on startup so i just mirrored the logic. Now i'm thinking the logic may be wrong.

On low disk space it will trim, also on every insert, it will check the database size and trim if needed. I also added a bit of code to auto resize the max database size based on the available free space on the storage media.

spyhunter99 added a commit that referenced this issue Oct 23, 2016

#435 fixing the build
#455 adding extra checks to improve performance

spyhunter99 added a commit that referenced this issue Oct 23, 2016

Merge branch 'master' into feature/#435-2
# Conflicts:
#	OpenStreetMapViewer/src/main/AndroidManifest.xml
#	OpenStreetMapViewer/src/main/java/org/osmdroid/MainActivity.java
#	OpenStreetMapViewer/src/main/java/org/osmdroid/samplefragments/SampleFollowMe.java

spyhunter99 added a commit that referenced this issue Oct 23, 2016

@spyhunter99

This comment has been minimized.

Show comment
Hide comment
@spyhunter99

spyhunter99 Oct 23, 2016

Collaborator

merged

Collaborator

spyhunter99 commented Oct 23, 2016

merged

@spyhunter99 spyhunter99 reopened this Oct 23, 2016

@spyhunter99

This comment has been minimized.

Show comment
Hide comment
@spyhunter99

spyhunter99 Oct 23, 2016

Collaborator

and my fix causing all kinds of issues, disabling the auto trimming for now

Collaborator

spyhunter99 commented Oct 23, 2016

and my fix causing all kinds of issues, disabling the auto trimming for now

spyhunter99 added a commit that referenced this issue Oct 23, 2016

@InI4

This comment has been minimized.

Show comment
Hide comment
@InI4

InI4 Oct 29, 2016

Spontaneous question: Does a trim without vaccum shrink the database file size?

InI4 commented Oct 29, 2016

Spontaneous question: Does a trim without vaccum shrink the database file size?

@spyhunter99

This comment has been minimized.

Show comment
Hide comment
@spyhunter99

spyhunter99 Oct 29, 2016

Collaborator

From what I've seen, yes. But it may vary by android version

On Oct 29, 2016 6:04 AM, "InI4" notifications@github.com wrote:

Spontaneous question: Does a trim without vaccum shrink the database file
size?


You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub
#435 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB4kGWYw4wK6TQEui5CeqlKi4t1tz6A7ks5q4xoIgaJpZM4KPiZO
.

Collaborator

spyhunter99 commented Oct 29, 2016

From what I've seen, yes. But it may vary by android version

On Oct 29, 2016 6:04 AM, "InI4" notifications@github.com wrote:

Spontaneous question: Does a trim without vaccum shrink the database file
size?


You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub
#435 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB4kGWYw4wK6TQEui5CeqlKi4t1tz6A7ks5q4xoIgaJpZM4KPiZO
.

@spyhunter99

This comment has been minimized.

Show comment
Hide comment
@spyhunter99

spyhunter99 Oct 29, 2016

Collaborator

Actually this ticket needs more work. App procure storage isn't working

Collaborator

spyhunter99 commented Oct 29, 2016

Actually this ticket needs more work. App procure storage isn't working

spyhunter99 added a commit that referenced this issue Oct 29, 2016

#435 more fixes for the default app, sample app should now work when…
… there's no external storage available and when there storage permission is denied. woohoo!. Also added some hacks for androidx86 functionality and examples for keyboard/mouse wheel events.


#236 zoom to bounds example
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment