Option to shrink unused space

[livotov]

Did not find any info on that, so decided to ask here - is there (or planned) an option to shrink (e.g. compact the database)?

We're using objectbox for quite a long time for a production app one entity there is used to collect various loggable information for further sending it to a server. Successfully sent entities gets removed from the database, however, the physical db file still grows constantly. (which is understandable)

I understand we can always delete the file before creating the ObjectBox instance in order to reduce the unused space, however there are other entities there so we'll need save and recreate them somehow in the new (emptied) datastore. Thus, kind of an option to the builder like .shrinkOnOpen() would be nice to have, so app may check the database size during the initialization process and shrink it when necessary.

Thanks!

[crispert]

I placed the entities which are most frequently deleted in a separate BoxStore and when this reaches a certain size, the entities in it are sent to server and the BoxStore is recreated by calling deleteAllFiles() and then rebuilding it. Additionally the entities that need to be retained could be read and kept in RAM (provided it is big enough) while the BoxStore is being shrunk and then written again to the new tighter store.

[greenrobot]

I just want to make sure that the DB size is growing within expected bounds: basically, if you remove and put comparable sized entities in the same quantities, the required space should not grow anymore at a certain point. Is that what you are seeing, too?

[crispert]

Yes, indeed, the size of the BoxStore is stable, when some entities are deleted and new ones are saved the size remains constant, so there is absolutely no problem with the database. The only thing is the BoxStore file is not trimmed after deleting entities which is perfectly normal from a systems' point of view since it will grow right back afterwards and it will add unneeded complexity and overhead to the engine.

All my fiddling with the BoxStore was due to a business requirement that the size of the app storage must not exceed an arbitrary limit and in case that limit is reached no more entities should be created. Since the usage patterns of the application are quite diverse in the few cases that limit was reached the data was removed from the BoxStore but its size remained the same, so the app would still not create new entities. That was the only reason for deleting the file and recreating it, no 'leaks' or 'fragmentation' as far as I could observe, the database works flawlessly and I hardly even notice it.
Perhaps there is a simpler way to answer the request by simply exposing some low level attributes of the BoxStore like overall byte size and used byte size or loading factor (%).

In the end thank you very much for this awesome library which is truly a game changer.

[livotov]

yes, exactly like @crispert wrote :)

[eth5]

ObjectBox will automatically use the free space inside the database after removing some entities? I understand this correctly? I used very large entities with binary data and my database grew very quickly, after removing some entities from the database, I added more new entities to exactly repeat those I deleted. I expected the new entities to use the remote place, but my base has become even more

[crispert]

Reuse of space previously occupied by deleted entities is more of an observation, not an advertised feature of the platform and it's probably a best effort attempt, not a guarantee. Moreover this observation was made without considering binary data.
My guess is that storage space for binary fields (byte[]) is shared for several/all entities, so it may be difficult to anticipate if the available space can be reused at all because it might get fragmented by insertion of other entities which even if they use only a fraction of the overall space would still make it insufficient for the re-inserted entity that had been deleted in the beginning.
Also are you sure the new entity is exactly the same as the deleted one? You might read the same data but the array holding it could be larger.

ObjectBox will automatically use the free space inside the database after removing some entities? I understand this correctly? I used very large entities with binary data and my database grew very quickly, after removing some entities from the database, I added more new entities to exactly repeat those I deleted. I expected the new entities to use the remote place, but my base has become even more

[greenrobot]

@crispert Thanks for the response. It's mostly correct. Space is reclaimed, but the behavior is too complex to predict (b-tree, pages...).

binary fields (byte[]) is shared for several/all entities

There no such thing, each object is stored independently from each other.

[mohib-irshad-10p]

Do we have a way to shrink the database now? My app currently write data worth an MB every 2-3 seconds and after a few minutes, it sends the data to the server and deletes everything from the database. However, app size does not reduce at all.

[greenrobot]

it sends the data to the server and deletes everything from the database

In that case you could delete the DB completely instead:

boxStore.close();
boxStore.deleteAllFiles();

[mohib-irshad-10p]

Thanks but actually there is a concurrency issue with that for me as while I am deleting all of the data that got uploaded to the server, new data is still getting stored in the database. So just deleting the database would result in loss of this new data and also, while I am deleting and recreating the database, new data cant get stored because the database is closed.

SOLUTION: I did end up using your advice but with an additional workaround. What I did was, I created a secondary database for temporary storage. So, while the database is getting deleted and recreated, I store every new data into this secondary storage and as soon as the recreation is done, I migrate all of the data back into my primary database and then recreate this secondary database as well to reclaim space occupied by it.