Backup: Do not exclude /data/media

[leo-bogert]

When doing a backup on my Galaxy Nexus, TWRP 2.7.0.0 will exclude anything in /data/media because it puts its backup in /data/media/TWRP (or /data/media/0/TWRP).

But /data/media typically contains a lot of user data, especially the camera pictures.
Further, lots of Android apps have the bad programming style of storing their data there, for example WhatsApp.

This renders the TWRP backups rather incomplete. Can you please change it to only exclude its own backup folder and include everything in /data/media?

[zeorin]

+1

I know I've seen that the prevailing opinion has been that there are other ways to back this up (e.g. just copy stuff over onto a PC), but many apps do indeed store their stuff in that folder.

Also, it's not very obvious to a user that when they're selecting the DATA partition for backup, that it's not actually the DATA partition that's backed up, just parts of it.

I myself never employed any of the 'alternative backup methods' for /data/media because I was (reasonably, IMO) under the impression that I was often backing that up when I selected the DATA partition.

I've now lost many sentimentally valuable photographs. I know people say you should back that stuff up, but that's what I thought I was doing.

So, either include /data/media (except other backups), or make it very clear what is actually backed up when someone selects the DATA partition for backup.

[tdussmann]

Def. +1

I was just wondering whether this was the case for a backup of the data partition or now.
Sure, when creating a backup of /data onto /data/media it would create a loop,
but for all other targets, like external SD-Card or USB OTG, it would be possible to copy the 'external storage' as well...

It could be shown only when selecting a supported backup location and to make it clear, that media/ is not included, it could be greyed out when backing up to it...

So, yeah, we can still take a full backup by backing up storage manually after a TWRP backup, but for a full blown backup/restore solution, it should be able to support this. After all, a lot of people use TWRP for it's extensive functionality.

[prataczak]

I was actually just getting ready to flash Odin, with the intention of flashing back to Cyanogenmod 12.1, but on a hunch, I decided to research this issue (because I remembered my last restore said something about excluding data/media).

I guess it's a good thing I did, seeing as flashing the stock ROM will result in being stuck in boot loop unless I do a factory reset on my Samsung Galaxy S4.

I'm new to using TWRP, and I have to say I already like it better than CWM, for the simple fact that every single device I have used CWM on has resulted in failed restores and hours of frustration manually restoring apps.

But this right here... this defies logic. This is not intuitive at all. With all due respect... who honestly decided it would be a good idea to skip backup of a folder that contains tons of data (I just checked mine - it had 1.6 gb of app data)?

And I never would have thought to 'manually' back up that folder. I shouldn't have to. I run all my backups to an external SD card just because I am paranoid about losing anything.

This should be fixed. Once again... this defies logic.

[smemsh]

Just lost a bunch of important files because of this on my titan board. Thought I was backing it up because:

1. I selected to backup /data,
2. there was a "microsd" checkbox (external sdcard), but
3. there was not any checkbox for internal sdcard
4. I presumed the list was exhaustive (and would backup everything if all were checked)

so my assumption was that internal sdcard contents were included in /data (since it actually is within /data filesystem in /data/media/0 this seems like a reasonable assumption)

definitely at least the user should be warned, I would have backed it up manually had I known. Only after wiping and restoring, but having all kinds of errors coming back up (even google apps are storing things here) did I inspect the tar file and find that entire section was missing!

[biji]

Yeah ... Dev please implement this, lost my media files, I thought backup /data is enough

[kdrag0n]

+1

Lost all my game progress, documents, and photos. At least warn that the internal storage will not be backed up. Maybe add another option to back up "Internal Storage" (the term used for /data/media in the rest of TWRP)

Please fix, TWRP.

[niker]

On android 5.x the problem is even worse, since the access to this internal storage is limited via permissions/SELinux for all apps. I copied all contents via ADB as a backup, but folders created by an app must have the correct permission/context set AFTER the restore or the app can only access said folder in read-only mode. This means even though I did backup to PC, the backup was worthless and after restore half of my apps kept crashing or behave strangely due to permission problems. Backing this up conveniently in recovery should definitely be an option.

[lchiocca]

This would be a really nice addition. But I have to admit that this would only be a use-case when backing up to an external device (via usb-otg or an external sd-card). I've been looking into the code and it doesn't seem easy to "pull up" this parameter to the UI. As a quick workaround, would adding a global variable that would override this behaviour be an option (e.g. TW_FORCE_BACKUP_OF_MEDIA_DATA)?

[JustMyGithub]

Another +1 from me.
surely you can backup it to a connected computer, but:
a) MTP is slow for a large number of files (much faster for an archive)
b) application data is stored on internal storage (you backup android.secure, don't you? - treat app data on internal storage differently)

I just recently lost app data due to a bug in the app. If the backup would have included media folder, I could restore the app data now....

Of course you better exclude the default TWRP and CWM backup paths ;D

[wdormann]

As a workaround, I find it quite reasonable to rsync the sdcard periodically. Being rsync, the incremental backups are super fast. And the combination of a restore from a nandroid backup plus a reverse rsync will put you right back where you were. https://play.google.com/store/apps/details?id=eu.kowalczuk.rsync4android&hl=en

[eulerreich]

+1 for me as well. Lost a lot of data because I thought TWRP got my back on /data/media

[leo-bogert]

Can someone please fix this ASAP?
I need this really urgently. Please help.

[leo-bogert]

Or even more easy:
Can someone please merge the fix for this?
Github automatically listed some commits above from another repository which seem to fix this.
Please, I need this ASAP, it would be of great help to me :)

[CaptainThrowback]

No one has submitted any patches to gerrit for this.

[leo-bogert]

I'm willing to pay a $ 15 bounty in Bitcoin to whoever fixes this.

Please prove it by linking the merged pull request with your fix.

EDIT: Bounty currently not available anymore, sorry.

[cmorlok]

Some information regarding /data/media: https://twrp.me/faq/datamedia.html

[blackwind]

That this is still an issue over two years later is just plain irresponsible on the part of the developers. I was fortunate that I had just begun my Android journey when first afflicted. Others here, apparently, weren't so lucky.

As noted by others above, if you're not going to include /data/media, at least inform the user in no uncertain terms. But you should include /data/media. That's what 100% of your users are expecting, and no one should have to learn such an important lesson the hard way.

[kitambi]

100% agree here. I probably should have read this (two year old) issue before trying this, but let's just sum it up here:
All of my photos from the last several months are gone because TWRP failed to back them up.
I have Titanium Backup.
My phone has an external SD card.
I don't give a crap about anything in /data except...oh yeah, /data/media.

So, thanks for that. Every once in a while, I consider just dumping Android completely and getting a flip phone. This is one of those times.

Note: FIX THIS. NOW. PEOPLE STORE IMPORTANT THINGS IN /data/media! I'm afraid that I can't recommend TWRP to anyone in good faith until this is fixed.

[blackwind]

Tagging the most active contributors since this issue seems to be buried deep: @Dees-Troy @that1 @bigbiff @mdmower @Tasssadar @enh @z31s1g @andi34

[JustMyGithub]

what about using a separate file for /data/media-backups.
that way people can unselect or choose to backup the internal sdcard easily.

[kitambi]

A big improvement (and would probably take about zero minutes of effort) would simply be to change "data" to "data (excluding data/media)". This way, there's disclosure, and in my case, I'd simply have tarred it up and copied it somewhere. Most people advanced enough to install TWRP probably have the skillset to perform a non-recovery based workaround (or just adb shell into TWRP and copy /data/media to the external sd card.)

Also, WTF...someone else already did the work for this?

[CaptainThrowback]

@kitambi If it would take zero minutes of effort, feel free to submit the change to Gerrit for review. TWRP is open-source, after all....

[Dees-Troy]

We made this design decision years ago when emulated storage first came about. I'm not entirely opposed to revisiting the decision as emulated storage is widely used now, but there's more to the decision than just, "do we or don't we".

For example, when you restore a backup of the system partition, we wipe the system partition first, then do the restore. If the user decides to restore a backup of the media folder, do we wipe it first? If we wipe it, are we wiping out newer photos, documents, downloads, etc? The whole issue gets a lot more murky when you start to consider these implications.

And no, the average TWRP user can't even spell adb. I mean literally, they can't even spell it. The average TWRP user is running MS Windows. This same user doesn't have adb drivers installed and they don't even know that they need drivers or what a driver is. Our average user doesn't have adb binaries downloaded either.

When the average user makes a backup, they either leave the defaults set or they check every box present in the GUI because more backup is better. This same user won't pay the least bit attention to the implications of restoring a backup of media and how restoring that backup may or may not wipe out their most recent precious pictures of their newborn child.

Let's be real... the average TWRP user doesn't even know how emulated storage works. They have no idea that their /sdcard is really located in /data/media. For a while we tried not creating /sdcard in TWRP for emulated storage devices. Gobs of people complained that their /sdcard was missing and no amount of explaining helped.

[cmorlok]

But the same implications already apply when the user restores /data. That wipes out the latest texts, call logs, whatsapp messages, game credits,... (unless his apps sync with an online service). How does the average user deal with that?

[kitambi]

I suppose that makes sense when we consider the historical implications. However, as far as /data/media goes, TWRP doesn't currently wipe /data/media at all unless the user specifically requests it, or they format /data (which I'm assuming is a power user feature, but now I'm losing even more faith in humanity with the knowledge that people are using Android recoveries without having the slightest inkling about what they're doing...also, how do they install TWRP without Android platform tools?)

I think an acceptable compromise, in the immediate, is to add a tag to the partition list that says "excludes /data/media" next to the "Data" name -- after all, if we're talking "average users" here (of which skill level is assumed to be very, very far from a normal distribution), that should get them asking questions. Also, since "Data" is a pretty broad term, most people ( (set of all users) - (set of twrp developers) - (set of people who lost everything in /data/media) ) expect a backup of "Data" to backup all of their data.

The current behavior is the equivalent of a Windows backup program backing up the entire system except for "C:\Users", or a Linux archive program operating on the entire system, but excluding "/home". Unintuitive at best, destructive at worst.

I think I'm done adding an "excludes" tag to the partition list, but now need to free up enough space for a test build to run in an emulator. :) This should allow people who know how to archive their /data/media in a different way, while (hopefully) prompting the "average user" to ask questions about it before they lose everything.

[rernko]

Just spent all day trying to figure out how to make a proper backuo of /data/media. Hopefully this makes sense:

adb forward tcp:7080 tcp:8080 &&\
adb shell 'tar -zcvf - /data/media | nc -p 8080 -l 1>/dev/null' &\
sleep 1;\
nc localhost 7080 > media.tar.gz &&\
adb forward --remove tcp:7080

[jcadduono]

@rernko interesting way to accomplish this, have you tested it with >4GB stream?
I used to just do adb exec-out 'tar cf - /data/media | gzip' > sdcard.tar.gz but always ended up with ADB timeouts or failures near the 4 GB mark, I assume netcat gets past that issue, nice way around it

[rernko]

The resulting 11.1 GB archive seems ok. Haven't tried putting it back tough...

[nailyk-fr]

This 4gb problem looks like a computer fs problem. Can you give a try with some ext4 or f2fs fs? Le 20 juillet 2017 21:36:09 GMT+02:00, James Christopher Adduono <notifications@github.com> a écrit :

…

@rernko interesting way to accomplish this, have you tested it with >4GB stream? I used to just do `adb exec-out 'tar cf - /data/media | gzip' > sdcard.tar.gz` but always ended up with ADB timeouts or failures near the 4 GB mark, I assume netcat gets past that issue, nice way around it -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #276 (comment)

-- Envoyé de mon appareil LineageOS avec K-9 Mail. Veuillez excuser ma brièveté.

[jcadduono]

@nailyk-fr errrrrrrr, do you figure i'm running windows 98 with fat32? lol
testing now it actually looks to be completely random where adb exec-out times out, anywhere from 60MB to 2GB, and when it does I can't reconnect...this is a completely different issue from the thread though and is probably just an adb server/client daemon issue :/

[transplier]

You can also consider using rsync and an external USB drive. See my post above from April 20.

[nailyk-fr]

It wont compress but
adb pull /dev/block/platform/*/by-name/userdata .
should work.

Nice netcat trick !

[eltznth]

I came up with a good solution so I hope the devs don't screw this up in the next release. The solution is to rename /data/media to /data/media123 or whatever prior to backup. After backup or restoration, remember to rename /data/media123 back to /data/media.

Easy or what?

[eocanha]

The script to restore the media.tar.gz backed up by rernko's netcat script (thanks!) would be this one:

adb forward tcp:7080 tcp:8080 &&\
adb shell 'nc -p 8080 -l | tar -zxvf -' &\
sleep 1;\
nc localhost 7080 < media.tar.gz &&\
adb forward --remove tcp:7080

[WLF0X]

I came up with a good solution so I hope the devs don't screw this up in the next release. The solution is to rename /data/media to /data/media123 or whatever prior to backup. After backup or restoration, remember to rename /data/media123 back to /data/media.

How do I know if it still works? Only September build?

[rernko]

Put it to the test! (And post results here 😉.)

[mikhoul]

I lost everything in my "/data/media/" and I'm far than being a newbie with computers and programming, 99% of the review of TWRP said it backup EVERYTHING/Mirror Copy to don't worry only to find that it's not true and you lost many precious files 😞 .

Most casual/regular users don't know what is "/data/media/" when they use TWRP they only want like any general full backup solution to have it to backup ALL the files and NOT all the files except some.... it make no sense.

Even the warning mean nothing for 70% of the user since they don't really know what is "/data/media/".

The current behavior is the equivalent of a Windows backup program backing up the entire system except for "C:\Users", or a Linux archive program operating on the entire system, but excluding "/home". Unintuitive at best, destructive at worst.

Why being so STUBBORN and don't add the OPTION to backup EVERYTHING including "/data/media/" ? 🤕

[bes1002t]

The main purpose of an Backup is to save all data especially Photos and Videos! That TWRP does not include /data/media means that the backup mechanism of TWRP is not capable to create Backups which fulfill the main purpose of an Backup.

Please implement a way to include /data/media.

[CaptainThrowback]

It's open source. Anyone can submit a patch for review for this. Please add me as a reviewer when you do. Thanks.

[steadfasterX]

It's in the works.. I was sick of not having it and no - I am not a TWRP developer and even not a CPP one and still.. its opensource, I love opensource and so I contribute like everyone could do..

https://github.com/steadfasterX/bootable_recovery_twrp_fork/wiki/Internal-Storage-(data-media)-Backup

as said its in the works so may need a lot of work before merged but I really hope I can achieve this one day.

[leo-bogert]

@steadfasterX:
Thanks for being willing to deal with this!
Are you aware that above in this issue, on Sept 9, 2015, someone referenced it from commits which likely implement the necessary code You could have a look at those commits to spare you some or all work.

[steadfasterX]

@steadfasterX:
Thanks for being willing to deal with this!
Are you aware that above in this issue, on Sept 9, 2015, someone referenced it from commits which likely implement the necessary code You could have a look at those commits to spare you some or all work.

No I have re-implemented it from scratch..
and tbh the main parts are done and working fine already (as you can see at the linked wiki page) and only some smaller bugs are open but keep in mind:

• there are still a lot of tests to do (which all can cause delays as things need to be implemented)
• even the actual code merge can take time as it must fit the needs of the TWRP lead devs ofc.

[beerisgood]

Maybe work together with https://github.com/kdrag0n/tipatch ?
@kdrag0n

[steadfasterX]

Basically his tipatch was the reason why I started it right now. I don't wanna say it's bad how tipatch works but.. Let's say it's a complete different approach and works completely different... and is imho far away from being ideal..
Don't get me wrong I love to see ppl find solutions which work! I do the same often enough but it's not the way I wanna see it in TWRP..

[kdrag0n]

Author of Tipatch here. I actually ended up making Tipatch after losing data and commenting back in 2015. After watching this issue, I was disappointed that the upstream TWRP developers did not want to add an option or something to include internal storage Instead, they just added a warning. I agree that my solution is not ideal. It's meant as a stopgap until upstream changes their mind or a major fork arises with the feature. I've considered forking and adding this feature as a seperate "Internal Storage" option on the Backup screen, avoiding the issues of Tipatch's binary patching. Let me know if anyone wants me to try that. @steadfasterX I don't think adding a setting for it is a very good approach because it may not be obvious to first-time users. Since "Internal Storage" is already used for /data/media everywhere in TWRP, we should show a backup option with that name. Edit: sorry about the spacing, caused by using email.

…

On Wed, Sep 12, 2018, 12:21 PM stay steadfast ***@***.***> wrote: Basically his tipatch was the reason why I started it right now. I don't wanna say it's bad how tipatch works but.. Let's say it's a complete different approach and works completely different... and is imho far away from being ideal.. Don't get me wrong I love to see ppl find solutions which work! I do the same often enough but it's not the way I wanna see it in TWRP.. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#276 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHkBf8NKMSUHJSvm8PgT1G8yQSp-bLQxks5uaV6xgaJpZM4B2e11> .

[CaptainThrowback]

@kdrag0n did you look at the Wiki link and the screenshots? That's what his solution does...

[kdrag0n]

@CaptainThrowback Just looked at the wiki link, it was showing up blank in my client. The point still remains that it shouldn't be under a Settings option because it's already an optional checkbox in the backup partition list.

@steadfasterX another comment: /data/media shouldn't be forced as vfat because it's not all the same owner and group. /data/media/obb contains media_obb:media_obb, and /data/media/0/Android/Data contains media_rw:30013 (example, group is determined by app).

[steadfasterX]

@kdrag0n
Thanks for sharing your thoughts.

I have removed the hardcoding to vfat already (I just forgot to update that wiki part) even when it really doesn't matter as my solution will never format anything. Anyways that's reverted and gets auto detected as before.

The decision to enable that setting or not as a default can be taken by the TWRP maintainer with a build flag. I believe that this is the most flexible way to handle it. Yes ofc the behavior what happens when the build flag is not set is to NOT enable that setting and this can be discussed.

For the moment it will stay like this and maybe one day when all have been thoroughly tested on several devices we just switch the default to enable it.

[kdrag0n]

@steadfasterX I see, thanks for explaining. Good luck on getting it merged into upstream.

[androidacy-user]

I'm thinking maybe to address the issues presented with Internal storage a couple points need addressed:

• Firstly, unlike system, we should consider a recursive merging strategy for restore instead of erasing first. Unlike system, this is significantly less likely to cause catastrophic issues when restored. In addition, we would never overwrite newer files (needs consideration).
• A recursive merging strategy should be also considered in all /data backups. Perhaps a UI setting to "Merge userdata instead of format".
• Currently some devices use persist for settings instead of data. Provided the persist partition is big enough, maybe it should be default
• Don't exclude /data/TWRP but /data/TWRP/backups. Rationale: it's actually nice to have a backup of TWRP settings and/or theme
• Bare minimum data backups should include /sdcard/Android - some apps store important data there
• Backups of data should have a check mark box under data for internal storage perhaps [✓ Internal storage - includes Downloads, Pictures, etc]

Quite honestly I've yet to find a convenient way to backup everything internal storage - MTP and most apps don't see the contents of /data/media/0/Android/*/. And heaven knows a NANDROID backup is probably the safest and most complete backup method - complete except for the one unpleasant surprise a lot of users will get when restoring backup.

As for @Dees-Troy suggestion that most users don't even know what /data/media is, you are EXACTLY right - which is precisely why this should be included by default in data backups. Most people don't read data as "/data excluding /sdcard" but "my data". Although UI has been improved to reflect this, there will still be some misunderstanding.

Personally I have everything I care about backed up to my servers but many don't. Many if I wiped their phone today several years of history would be wiped alongside it.

Just thought I'd put my two cents in an important issue that's been dead for over a year now. Hope it means something to someone

[Manwithpants812]

Complete clownery that users aren't AT LEAST given the option to do this, still 10 years later, and the misleading nature of its omission leading to people losing their internal storage/media. If you're so apprehensive about "what about the new /data/media already there", simply add an additional confirm screen that spells that it will delete your existing internal storage in case of restore. Having it be backed up as an option is invaluable. Users use TWRP specifically to backup their device, that means the whole thing and they assume so.

Backing up to OTG is my only option and I don't have an intuitive easy way do this for my internal storage via TWRP.

[CaptainThrowback]

Complete clownery that users aren't AT LEAST given the option to do this, still 10 years later, and the misleading nature of its omission leading to people losing their internal storage/media. If you're so apprehensive about "what about the new /data/media already there", simply add an additional confirm screen that spells that it will delete your existing internal storage in case of restore. Having it be backed up as an option is invaluable. Users use TWRP specifically to backup their device, that means the whole thing and they assume so.

Backing up to OTG is my only option and I don't have an intuitive easy way do this for my internal storage via TWRP.

Awaiting your patch on Gerrit to enable this, thanks for your concern!

[jxu]

Wow what a throwback from that notification. I'm pretty sure this project is long dead.