Message#encoded breaks existing gpg signatures

[ulfurinn]

I'm using mail in an MDA filter script. It reads the message from stdin, parses it with Mail.new, sets some extra headers, and prints the output of Message#to_s. What I've discovered is that by doing this any gpg signatures present in the original message will be rendered invalid. I've narrowed it down to additional headers being injected into the multipart chunk containing the cleartext, and Content-Type being broken into several lines, i.e. this

--h31gzZEtNLTqOjlF
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

test

becomes this:

--h31gzZEtNLTqOjlF
Content-Type: text/plain;
 charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Content-ID: <xxxxxxxx>

test

So it would appear that the entire cleartext chunk must be preserved verbatim for the signature to be valid.

My current workaround is to print #header and #body directly, which bypasses normalisation, but since this behaviour might change in future releases (edit: not usable with more complex messages), I'd prefer an explicit option to suppress normalisation, especially when dealing with externally created messages.

The problem is reproducible in Mail.app/GPGTools (using gpg 1.4.19) and mutt (using gpg 1.4.16)

[wolfedale]

I'm having the same problem with Mutt/1.5.23.

When I will read an e-mail using Mail.new(STDIN.read) or Mail.read(mail_source) like:

require 'mail'
@mail = Mail.read('my_mail')
puts @mail

I'm having different source mail:

Original mime part:

--k4f25fnPtRuIRUb3
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Received mime part:

--k4f25fnPtRuIRUb3
Content-Type: text/plain;
 charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Content-ID: <xxxxxxxx>

[kjg]

I'm not sure if to_s is intended to be able to round trip an email byte for byte, but if it is, it's still a long way off from that I think.

However, the raw_source method allows you to get the exact input that read or read_from_string parsed, so I think @mail.raw_source will give you what you need.

[ulfurinn]

If you read from a string, you already have the original string anyway.

The point was rather that if a message was constructed from source (as opposed to from scratch through the gem API) with the purpose of making specific changes, no other implicit alterations should be done, ideally.

[ulfurinn]

Or is it possible to retrieve the raw source of individual multipart sub-messages?

[kjg]

Unfortunately I don't think it is possible currently to use the Mail gem to make only specific modifications, and then have it not also modify other parts of the message. Maybe the maintainers would accept a pull request that would keep track of which sections have been modified and which haven't so that it would only perform any logic that adds defaults to parts that have been modified.

To answer your question, most parts and sub objects have some sort of API to get the raw source of just that part. The method to get the raw source of a Mail::Message or a Mail::Part (which inherits from Mail::Message) is that same raw_source method.

mail/lib/mail/message.rb

Line 386 in 8130b9c

def raw_source

[wolfedale]

I'm not sure if I understand how raw_source works with @mail. What it will give me since I cannot simply send an e-mail using original source?

So basically if I even change it to @mail.raw_source I cannot call on it .bcc or deliver to send it as it is without changing headers. Not sure if this is intended for some reason or I'm missing something? :-)

[kjg]

Maybe it would help me understand better if you explained exactly what you are trying to do.

If you are trying to take an email, parse it with this gem, modify the email, then send it, I don't think it will be possible to maintain the gpg signature. Part of the purpose of that gpg signature is to verify that the email hasn't been changed. So it is completely expected for the gpg signature to be invalid after making modifications to the email

[wolfedale]

The point is I don't want to modify anything. I'm simply expecting to read an e-mail from STDIN and send it back to the postfix queue. So if I will open it like:

Mail.new(STDIN.read)

or if it's the file like:

Mail.read(mail_source_file)

so now, when I will call deliver method on this @mail I'm expecting to get the same source as it was before, since nothing has been changed.

[wolfedale]

The problem is that now I need to go for IO.popen(sendmail) to send my e-mail from STDIN.read. Since Mail.new is changing it's headers. It might be nice to have some sort of option in this gem to avoid it.

[kjg]

Ah okay, that makes sense. I'm not sure ruby is necessarily the most effective way to take a STDIN or a file and feed it to posfix. Regardless, that's not really what this Mail gem is for. The gem is mostly for two things, 1. parsing emails and providing a nice OO interface for accessing the data, and 2. constructing new emails from a nice OO DSL.

For your use case you may want to just interface with Net::SMTP or sendmail directly. As far as delivery goes, the mail gem provides just a very very small wrapper around those which does no more than some envelop checking and turning the mail object into a string.

If you don't care at all about modifying the message, and you're definitely going to be using ruby, your best bet is probably calling IO.popen(sendmail) or Net::SMTP.new yourself which is basically all the gem is doing anyway for delivery.

mail/lib/mail/network/delivery_methods/smtp.rb

Line 98 in df48a05

def deliver!(mail)

mail/lib/mail/network/delivery_methods/sendmail.rb

Line 72 in df48a05

def self.popen(command, &block)

[wolfedale]

Ok, in that case it's clear and reasonable :-) So I will just stay with IO.popen(sendmail).

Thanks!

[ulfurinn]

Part of the purpose of that gpg signature is to verify that the email hasn't been changed. So it is completely expected for the gpg signature to be invalid after making modifications to the email

Signatures use multipart. You can change headers of the top-level message without invalidating the signature, as long as sub-messages are left intact.

[ulfurinn]

My use case is that I do want to modify top-level headers, but then the gem mangles the signature because it also automatically normalizes headers of the signed sub-message.

[kjg]

@ulfurinn So you want to modify top level headers, but not headers with in the multipart/signed part at all.

Yeah, that's definitely an issue. Maybe logic can be added specifically for Multipart/signed parts or more generically maybe logic can be added to not normalize headers for parts that were read in from somewhere and not modified at all.

Is that something you'd be willing to try to tackle?

[jeremy]

I had a similar case when implementing a mailing list. I needed a remailer that changed message headers but kept the message body verbatim.

Here's a starting point for exploration…

# Use the original mail body, verbatim.
message.body = mail.body.encoded

# Manually split its parts (mail lib quirk, no API for this).
message.instance_variable_set '@separate_parts', message.multipart?
message.send(:process_body_raw)