delete messages from the device does not free up storage space on the device

[darhma]

I had 5 DC profiles on the device and in particular one of tests where I used bots for feeds and mastodon, so I received many messages but also periodically deleted them from the device but continued to occupy the storage space, to the point that looking at the storage space occupied by all the apps was 69GB of which 51 from DC.

I tried to "clean up" all profiles by deleting from the device all the messages I was not interested in holding, and making a backup of the various profiles, deleting the app data and re-importing backups:
of the test profile I was talking about before I couldn't back up because the app crashed, the others went well.
I deleted the data of the app and I re-imported the profiles, except the test one, now the storage space is ~8GB that however are many, too many, compared to the messages actually visible on the device.

• Operating System (Linux/Mac/Windows/iOS/Android):
• Core Version:
• Client Version: 2.11 (from f-droid)

Expected behavior

By deleting one or more messages (even thousands) from the device you should free up the storage space of the device

What happened instead?
deleting the messages from the device, does not free up storage space on the device itself

[Simon-Laux]

Is it only too large on device or is the backup also too large?
Can you check the backup, if the space is used by the database of by attachments/blobs? (extract the tar file and look at the file/folder sizes).

[darhma]

I'm on android but the problem was initially reported by another user on desktop, it looks like a db problem, even deleting the attachments does not seem to recover storage space.

[vinniec2]

A few months ago I tried to do the same procedure exclusively on the desktop and the problem is the same.
By memory:

1. I had created a full copy of all DC data
2. Start the copied client and remove all local data for each profile (options > chat & media > delete data from the device)
3. then tried to share profiles with a new device (options > add a second device)

And even though the content of the chats was empty (but all the contacts were there), the passage was always very long and DC had shared the full content (even though the chats were still empty).

This brings several problems:

1. deltachat data grow endlessly
2. security problem if you delete the data but these in reality still remain present

As for point one, at some point I was saturated with the disk and for some reason that i didn't know, I lost access to all the profiles... fortunately that I still had the old backup.

P.s. Beyond the problem, it would be nice to add the option to share with another device only accounts and contacts without data.

[Simon-Laux]

I guess at some point it would be nice to have some disk/storage usage info screen inside of delta chat.

even deleting the attachments does not seem to recover storage space.

Does not tell me much, nowadays blobs are deduplicated, so the actual file would only be deleted if all messages referencing it are deleted. I don't know if that is the case for you, though. (because I don't know if you have sent/forwarded/received those attachments in multiple chats)

It would be helpful to see if the size comes from the database or from the blobs folder. and if it comes from the db, ideally from which table it comes from, but I guess you'd need a special tool to find out the size usage per table. (https://sqlite.org/sqlanalyze.html)

[darhma]

when I delete a message, I should also delete the attachments/blobs unless I have exported them, but even in this case they should actually be deleted from the memory of the app, because I find them I exported

[darhma]

on androi where do I find the blob folder?

[Simon-Laux]

when I delete a message, I should also delete the attachments/blobs unless I have exported them,

If you receive the same image or document in multiple chats, then only one copy is saved, this is the deduplication.
So in this case you need to delete all messages that reference it, before the file is actually deleted.

on androi where do I find the blob folder?

if you extract the backup tar, there is a blobs folder and a database file in it.

---

I just did some small test where I replicated part of vinniec2's experiment: I took one of my accounts, exported a backup, then re-imported and set autodeletion from device to 1hour and exported the backup again.
It did not save as much space as I had expected, but maybe part of that could be that this settings does not affect the saved/self-messages chat (and my saved messages chat is quite big).

Anyways I ran https://sqlite.org/sqlanalyze.html on it and it discovered many unused pages. I tried running VACUUM on the db, but the db file size stayed the same.

EDIT: according to link2xt it is only 2mb, so I guess unused pages are not the issue

Maybe this info can help the more SQLite experienced devs here.

Maybe disk usage and count of unused pages could also be an interesting metric for the opt-in statistics at some point.

[vinniec2]

but maybe part of that could be that this settings does not affect the saved/self-messages chat (and my saved messages chat is quite big)

I don't think because my saved messages chat is very small, almost unused

[adbenitez]

for the record I also received several reports and complains of users about the account size going crazy sometimes even +50GB

[link2xt]

Anyways I ran https://sqlite.org/sqlanalyze.html on it and it discovered many unused pages. I tried running VACUUM on the db, but the db file size stayed the same.

leaf_unused is the number of unused bytes according to https://sqlite.org/sqlanalyze.html On your screenshot you have less than 2 MB of unused bytes in msgs table.

[link2xt]

We cannot debug this without at least someone looking into backup and answering this question:

It would be helpful to see if the size comes from the database or from the blobs folder. and if it comes from the db, ideally from which table it comes from, but I guess you'd need a special tool to find out the size usage per table. (https://sqlite.org/sqlanalyze.html)

This problem has been reported multiple times in the chats before, but we never got these details.

It is unlikely to be webxdc updates of deleted webxdc apps, because we cleanup old updates not linked to any apps during housekeeping:

core/src/sql.rs

Lines 894 to 904 in f428033

	context
	.sql
	.execute(
	"DELETE FROM msgs_status_updates WHERE msg_id NOT IN \
	(SELECT id FROM msgs WHERE chat_id!=?)",
	(DC_CHAT_ID_TRASH,),
	)
	.await
	.context("failed to remove old webxdc status updates")
	.log_err(context)
	.ok();

Unless this request fails for some reason.

[darhma]

These days I do not have the computer available, only the phone, from which I do not have access to the problematic backup, in the next few days, as soon as I have the laptop available again I try to give more info.

Actually the account that had gone decidedly out of control was the one I used mainly with bots and webxdc

[MS702]

I tried to verify it and create a new account on my android (2.22.0) and stored a 1GB file in saved messages, after deleting it again, the disk usage was taking the same extra 1 GB and in the backup it also remained and I can verify there wasn't any other reference to this file on this profile (because it was new and saved messages the only chat).