Improve plaintext export and import

[AsamK]

This PR implements the export of MMS messages (without attachments, only text), fixes the import of group messages to the correct group thread instead of the sender's thread and fixes the export of the delivered state of Signal messages.
Previously messages sent in group threads were not exported, because they are always stored in MmsDatabase even if they only contain text, I’m not sure if this is intended or a bug.

The assignment to the correct group thread is implemented by adding an additional xml attribute "thread_address" that contains the thread recipient if that differs from the message recipient. For groups this is an id like this: "__textsecure_group__!1b92cba8…". The app SMS Backup & Restore doesn’t know this tag but simply ignores it when opening the Signal backup.
When importing into a new Signal instance the group thread will be called "Unnamed group" and you can’t send messages to the group until another member sends a group update message.
Fixes #3024

The last commit removes some attributes from the xml items that are always stored as "null" or 0. These attributes are marked as optional in the schema file of SMS Backup & Restore, see http://android.riteshsahu.com/wp-content/uploads/2014/03/sms.xsd_.txt

I’ll have a look at exporting attachments later, but that will be for another PR.

[merkste]

@AsamK Are you already working on the im/export of attachments? I am asking because I considered to look into this during the holidays.

[AsamK]

@merkste I haven’t started working on that. If you want to implement it, you should have a look at my changes in PlaintextBackupExporter.java. You’ll need to add code there that exports messages with attachments using the tag. And you’ll need to figure out how the mms xml format works.

[McLoo]

@AsamK awesome!
Maybe you can use the MMS XML Tags like SMS B&R does it. That way we would keep compatibility and adding media, with any upcoming chang, wouldn't break the changes you made heRe.

AFAIK the MMS tags are not really documented, but can be examined when creating a SMS B&R backup including MMS.

The additional group tag should workout fine. Didn't check, but can't we also store and recover the (last known) group name?

And, did you try importing into a non empty database. (a group backup with messages, that were deleted but the group still exists (handling duplicates should be another PR which is likely to include a new index))

[AsamK]

That was the plan to use the same MMS XML tag. The XML schema file I linked in the first comment specifies all the necessary tags and also which ones are required or optional. Unfortunately the cryptic names are not explained.

I thought about saving the group name, too, but as the group is not usable without someone else updating it, I didn’t implement it here. I see two ways to do this, either specify the group name in every message (lots of duplication) or make a new tag for each group (probably preferable).

I also imported to a non empty instance, and the messages ended up in the correct, already existing, group thread.

[McLoo]

Currently you're putting the MMS text into SMS tags, were they should go into MMS Tags. IMO.

A XML file with 1 SMS and 1 MMS would like sth. like this:

<?xml version='1.0' encoding='UTF-8' standalone='yes' ?>
<!--File Created By SMS Backup & Restore v400537.07 on 67/22/3053 37:17:59-->
<?xml-stylesheet type="text/xsl" href="sms.xsl"?>
<smses count="2">
    <sms protocol="0" address="+012345678901" date="15767564758212" type="1" subject="null" body="j27vIgEKIQVnVK[...what is that smell!?...]X1TH4LCuWowJcZIgmOhe2Ce1qhz[...ahh fish...]b7WrwSA" toa="null" sc_toa="null" service_center="+987654321000" read="1" status="-1" locked="0" date_sent="1396699987700" readable_date="05.04.2000 12:00:48" contact_name="Tinkerbell" />

    <mms text_only="0" sub="DSC00223" retr_st="128" date="1397567567000" ct_cls="null" sub_cs="106" read="1" ct_l="null" tr_id="NOHGKJFXDGAO27ng4E" st="null" msg_box="1" address="+00 111 2222222" m_cls="personal" d_tm="null" read_status="null" ct_t="application/vnd.wap.multipart.related" sub_id="0" retr_txt_cs="null" d_rpt="128" m_id="NOKASDF0000900002" date_sent="0" seen="1" m_type="132" v="17" exp="null" pri="129" rr="129" resp_txt="null" rpt_a="null" locked="0" retr_txt="null" resp_st="null" m_size="null" readable_date="08.03.3588 12:54:30" contact_name="Hook">
        <parts>
            <part seq="-1" ct="application/smil" name="main.smil" chset="106" cd="null" fn="null" cid="&lt;AAAA&gt;" cl="null" ctt_s="null" ctt_t="null" text='&lt;smil&gt;&lt;head&gt;&lt;layout&gt;&lt;root-layout backgroundColor="#FFFFFF" background-color="#FFFFFF" height="480px" width="640px"/&gt;&#13;&#10;&lt;region id="Image" top="0" left="0" height="50%" width="100%" fit="meet"/&gt;&#13;&#10;&lt;region id="Text" top="50%" left="0" height="50%" width="100%" fit="scroll"/&gt;&#13;&#10;&lt;/layout&gt;&#13;&#10;&lt;/head&gt;&#13;&#10;&lt;body&gt;&lt;par dur="4000ms"&gt;&lt;img src="DSC00223.JPG" region="Image"&gt;&lt;/img&gt;&#13;&#10;&lt;/par&gt;&#13;&#10;&lt;/body&gt;&#13;&#10;&lt;/smil&gt;&#13;&#10;' />
            <part seq="0" ct="image/jpeg" name="DSC00223.JPG" chset="null" cd="null" fn="null" cid="null" cl="DSC00223.JPG" ctt_s="null" ctt_t="null" text="null" data="/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAgGBgcGBQgHBwcJC[...censored...]EIAeACgAMBIgACEQEDEQH/xAGiAAABBQ[... I M WEASEL...] gZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2[...there's nokia like your kia...]m1of/9k=" />
            <part seq="0" ct="text/plain" name="text_0.txt" chset="106" cd="null" fn="null" cid="&lt;text_0.txt&gt;" cl="text_0.txt" ctt_s="null" ctt_t="null" text="In canis corpore transmuto... or was it something with 7 gnomes, 3 eleves and some mortals?" />
        </parts>
    </mms>
</smses>

[AsamK]

I agree that MMS should go in the mms tag. But implementing that needs more work.
So for now I put the text of MMS message in the SMS tag which is better than not exporting them at all.

[moxie0]

@AsamK I'm reluctant to merge this if it's not a step in the direction we want to go. If we want to export/import MMS, let's use the MMS schema.

[AsamK]

@moxie0 Ok, then I’ll try to implement that.

What about messages in the MmsDatabase that have no attachments, e.g. outgoing group messages. IMHO there’s no advantage of storing those with the mms schema.

[moxie0]

@AsamK Those either need to be shoehorned into the MMS schema, or need a new schema of their own. They'll need to include info from the group database.

[moxie0]

Closing this for now, please reopen when you've moved to MMS land. Thanks for taking this on!

[merkste]

@AsamK Did you progress on this matter? If not, I am going to look into the MMS part first, based on your commits.

[AsamK]

@merkste No, I haven't come far at all. MMS are rather complex. You're welcome to take this on.

[merkste]

@AsamK It took some time, but I am finally getting started.
@McLoo and others: Since MMS are not supported properly by my current provider, I'd be happy if someone could help me out with another export from SMS Backup & Restore, containing two or more MMS with and without attachments. It'd be nice to have those messages and their props (sent/rscv times, ...) in plain text as well, in order to allow me double checking the meaning of the xml attributes.

[merkste]

Additionally, shall we discuss technical details here in a separate issue thread?

[fmikker]

Possibility to import MMS attachments would make me switch to Signal as my main communications app, so +1! :)

[moxie0]

@fmikker Please do not +1 issues: https://github.com/WhisperSystems/Signal-Android/blob/master/CONTRIBUTING.md#dont-bump-issues