Media Not Exporting to XML

[alexh3791]

I have noticed that when exporting a plaintext backup in the current Play Store version of the app, all media attached to messages (and perhaps even the text from messages containing media) is not included in the XML backup, even though 'SMSBackup And Restore' backs up the media with all messages properly when creating an XML backup, and can restore the same data.

Since I use both TextSecure and SMSBackup And Restore in my system to back up a record of my messaging activity, I need media to be included in my plaintext backup alongside text messages.

[moxie0]

Didn't realize SMS B&R supported media now. Maybe @McLoo can take this on.

[alexh3791]

@moxie0 @McLoo Awesome! I'd love to see it implemented. I'd help if I could!

[McLoo]

@moxie not until next weekend. Got to figure out how media is exported. I guess Base64.
Would also need a short description how to access the MMS fields to safe some research time.

[alexh3791]

Has this been resolved?

[McLoo]

Not yet unfortunately. But I do eager researching already.
Though I still hope for some hints on getting the media data @moxie0 or @mcginty maybe?!

[mcginty]

@McLoo check out the setMediaMmsAttributes(MediaMmsMessageRecord) method in ConversationItem to see how you can check for and retrieve media "parts" to a message.

[diego0815]

@McLoo im trying not to nag... but is there any news regarding this? I'm waiting for that feature for some reason ;-)

[McLoo]

@diego0815 just go on nagging. I always need some trigger ;)

I was waiting for @mcginty to finish the makeover of the export process to go on. I guess I could at least go for the import part already...

[diego0815]

@McLoo that sounds great, yepp I also sometimes need this kick in the ass, just feeling bit bad nagging guys who are doing such great things in their spare time ;-) but as long as you appreciate an occasional nag, I'll gladly do so B-)

I'm waiting for media export, so I can get my messages exported, then all wiped ( to fix 814 for me) and then re-imported...

[alexlance]

I put a simple dump file here that reliably fails. There are three messages in it, 2 sms, 1 mms. The mms doesn't import.

http://alexlance.com/TextSecurePlaintextBackup.xml

Would love to get this issue fixed so I can bring all my existing messages into textsecure.

Let me know if I can provide any info or assist in any way. I also commented on #1410 which is probably a dupe of this issue.

[McLoo]

@alexlance MMS are totally ignored right now. The import for them has to be implemented first. So it's more a missing feature than a bug ;-)

[alexlance]

@McLoo may I suggest that it is very surprising behaviour for the
application to behave as though the import was successful, but to have
silently not imported all of the image/mms messages.

And unfortunately after an import into textsecure the user is very
likely to go and delete their messages from the android system sms
database - thus permanently losing the image data.

Are there any plans/timeline to implement this functionality?

[diego0815]

@alexlance as i understand it's being worked on, and its depending on some other prerequisites.
So i'd say its done when its done, but we got a right to nag from time to time ;-)

[alexlance]

@diego0815 Frankly - the user that is getting software for free does not have a right to nag :) but this particular issue stops me from using textsecure as my default messager.

And it might be a missing feature that it doesn't import mms, but it is a bug that it doesn't report an error to the user, saying: Hey everything was imported just fine, except these messages...

Presumably the textsecure dev team have some sort of roadmap of priorities - it'd be lovely to get an indication of when (or if) textsecure will support importing mms/images.

[diego0815]

@alexlance totally agree lets see when we will get what we are craving for (hint @McLoo ;-)

[alexlance]

@McLoo any chance of getting an idea of whether this functionality will be added?

[McLoo]

@alexlance I wish I knew. Work is pushing me quite hard these days. To much to work on such a change right now. 😢

@mcginty what about the rewritten import/export? Dropped?

[alexlance]

@McLoo ok thanks for letting me know. It's such a bugger, because I would love to switch over to TextSecure and delete the system message store, but I can't just discard all the old mmses - a picture is worth a thousand words :)

[Eric1911]

I just installed TextSecure (v 2.2.0) and love it, but was disappointed that my MMS images didn't import along with the SMS text from the Android (v 4.4.4) Messaging (v 4.4.4-24) app. I think this is the right topic to chime in and vote/beg for additional MMS functionality in TextSecure. Thanks for a great app!

[MarcRdC]

Any progress on this issue?

[McLoo]

sry. no more but a growing feel of shame for not touching this.

[MarcRdC]

@McLoo Nothing to be ashamed of; it is understandable that other things have more priority. Nonetheless, the import‐export feature would ease the switch from other SMS apps and would supply a nice way to comprehensively backup all messages (no loss in either case).

[mrdanbrooks]

+1 for this feature. I've been watching this issue for over a year, it is the only reason I have not switched to TextSecure yet.

Does anyone know how SMSBackup And Restore serializes their image data?

[p2004a]

Very similar situation as above. I'm waiting for this feature for more than a year too. I even have two phones. I don't use the second one but I still can't wipe out all the data from it because there is a TextSecure with all my messages. (of course I could do it manually but it would take a long time I'm not willing to waste so much of it..). Possibility of backing up this type of data is very, very important and I don't really care if the backup is encrypted or not. If it is not I will just encrypt it with other tool.

[alexlance]

+1

[alexlance]

Hi @McLoo and @mcginty and @moxie0,

Australia is moving over to mandatory data retention laws soon, which will include text
messages. And I suspect there will be a rise in the number of people wanting to migrate
to a secure messaging platform.

As this issue directly impacts users who are looking to migrate to a new messaging platform,
may I request that we get the ball rolling and discuss what would be required to get this
functionality implemented?

[spectas]

+1

I would love to see this feature!

[Kazpers]

I honestly do not know why this does not have priority. Personally I've introduced close to 10 people to Signal.

They all loved it.

But not a single one of them are using it anymore because after the first time they get a new phone and realize all their mms are lost this app is dead to them. So I'm now the only one using it making its encryption pointless.

Honestly I'd do this myself if I could, however I have exactly zero experience coding for Android. I cannot stress enough how important this issue is, however!

[Grunthos]

Is there an official status on the @rmgk version? Would love to see this feature and am happy to help with coding, testing, etc etc.

Any chance of a road map of what is needed so those of us who find the lack of backup one of the most appalling mis-features of Signal can do something about it?

[breznak]

After reading through the thread and seeing the bug is 3yrs old, I'd like to suggest a minimalist approach:

• not encrypted backup (current export is also plaintext?), users can encrypt themselves
• not necessarily SMS B&R compatible
• just offer to dump all media in a conversation to a folder
  • and for import append those media to the conversation (timestamps don't even have to be solved, just make them all appear as of the date of the import/export)

[fdw]

I'm not sure if I understand your suggestion correctly, @breznak : Should the media be part of the plaintext encryption (base64-encrypted)? Because then, the separate export of all media is not necessary.

Regarding point 3, the export of all media of a conversation was once supported, but is currently not (see #7097)

[breznak]

Should the media be part of the plaintext encryption (base64-encrypted)? Because then, the separate export of all media is not necessary.

Actually that is a good idea, if it's easier! I wasn't thinking of a plaintext export of media encoded as bas64-encoding.

Ad point 3: yes, all my activity was about the feature of "Save/Export All media".

[eipiminus1]

The Signal-Desktop app just transitioned from Chrome App to standalone. In the process of moving from one to another, you have to create a backup (unencrypted) that contains all messages, media and keys.
Maybe this format could also be adapted for the mobile client backups. This might even allow data exchange between mobile and desktop at some point. Unfortunately the export button is only available in the old Chrome app but not in the new standalone version which suggests that the Desktop team sees it purely as an intermediate solution for transition.

Maybe @moxie0 or someone from the OWS team could give us a hint if this is worth investigating further. I do not mean to unnecessarily ping for attention but I believe that some clear guidance on what is expected for a pull request to be considered, would be a huge motivator.
Currently i understand that

the OWS team wants:

• full compatibility with SMS B&R
• encryption

The community (only my impression based on what I read):

• does not fully understand the need for SMS B&R compatibility
• prefers simple, unencrypted backup over no (media) backup at all
• struggles finding good documentation on the SMS B&R format for media and thus to write a robust backup system based on it.
• is motivated to contribute but lacks time (who doesn't?) and guidance to produce merge-ready code

I personally think:

• Reusing a format that already has wide adoption (like SMS B&R) is generally a better idea than inventing some own, poorly documented format
• Even better would be a well documented format (maybe this could be a requirement for the pull request?)
• regarding the encryption of the backup: By default the message database is unencrypted for a while now, since those users that care about security could simply enable the full disk encryption provided by android. [1] The same logic would apply to the backup, right?
• I strongly believe that a fully working backup is a lot more important than a fully featured backup. My phone is slowly dying but as of right now I see (almost) no way to save the conversations. (Trying to compile one of the modified versions above is probably the best shot right now.) That's frustrating

I hope this helps to advance the discussions between community and OWS.
It's a great product that I love to use and promote of course! Thanks for that :)

[1] signalapp/Signal-Desktop#452 (comment)

[Grunthos]

Sounds excellent to me; I am happy to donate time and effort (coding) to this, but only if we have a genuine likelihood of whatever patch being accepted. There have been too many genuine efforts lost and discarded in the past to give me great confidence in the process.

Plain text export of text and images is vastly superior to the current offering.

[xyzzyxw]

No useful backup functionality is such a major drawback of Signal. Currently I don't recommend it to anyone anymore because of this.

I don't understand why compatibility to SMS B&R is required, as it seems to be a huge problem. I have never heard of that app before and also don't know anyone who has.
Just drop that requirement and implement a working backup. I don't care about the format as long as it works.

[fengshaun]

Is there a reason why Signal needs to export its data in a manner that is compatible with an undocumented xml format used by a proprietary application? I understand if the authors use that app and won't work on alternatives, but it seems like other solutions are rejected specifically due to incompatibility with this undocumented format.

[SolidTux]

I'd really like to help with this, but I don't see what has to be done next. Would it be possible to summarize the steps to archive the full backup?

[FeuRenard]

Here is a community forum thread with some recent discussion:
https://whispersystems.discoursehosting.net/t/encrypted-backup/1227/27
Further discussion should happen on the community forum, not here on Github.

[stefan-schiffer]

text backup to xml-file does not work either. it exports only messages from others (those from myself are not included) and includes characters incompatible with xml-parser (see chrome's error message below).

[strugee]

@stefan-schiffer those seem like separate bugs unrelated to this one. You should search for duplicates and file separate issues for each bug if you can't find any already existing.

[breznak]

@Roghetti Thank you so much for your contribution! 👏 Qucikly reviewed and it looks great. Hope the PR gets the attention it deserves and will be merged soon.

[Fmstrat]

@Roghetti So happy to see someone contribute on this one. Out of curiosity, can your method be used for archiving? I.E. keeping the backup file offsite, and then using the export in a viewer outside of the Signal App?

A use case example would be that I've created a Web based message viewer that allows me to go back and view old messages from Signal and Google Voice exports.

[stefan-schiffer]

@Fmstrat Does your solution work with the current text-backup? Maybe similar like this: https://mattj.io/sms-backup-reader-2/main

Could you please post a link to your viewer? I'm very interested on an offline solution too.

[Fmstrat]

@stefan-schiffer Yes it does, but it parses the files for both SMS backup (Signal) and Hangouts (need to update for Voice) into a database. It has a lot of personal info in the code so it's been just for me. Perhaps I could debrand it for you. Please start up a conversation on https://whispersystems.discoursehosting.net vs here, though, and I'll look into it.

[stefan-schiffer]

@Fmstrat Thx for the info.

[jgamage91]

I made an effort to move from WhatsApp to the signal along with my family. After realising lack of media back up, I had to revert back. Did I miss how it's done?

[ckujau]

Full backup/restore is coming to Signal with commit 24e573e ("Support for full backup/restore to sdcard"), which landed in the Beta releases already - yay! That should also solve issue #6539 I suppose.

[Fmstrat]

@ckujau Great news, thanks! Any idea when it will come out of beta? Unfortunately I'm not in the beta program and would need the export to switch over. ;) Picking up an S9 tomorrow so won't have the S7 much longer to export.

[automated-signal]

GitHub Issue Cleanup:
See #7598 for more information.