Ensure Content-Transfer-Encoding binary as the default for MDN's

[open-gdsn]

I'm currently investigating if our home build AS2 software (written in C, not well maintained but which used to be Drummond Certified) can be replaced by some other piece of java based software. AS2-lib being one of the options (running/changing OpenAS2 into running on a servlet container instead of monolithic being the other)

I tried to setup a partnership with our existing AS2 environment to see if at least that would work. Exchanging messages without MDN's works, with non-signed MDN's too, but when sending signed MDN's from AS2-lib to our server, I got "Integrity check failed" errors, after due investigation, I came to the conclusion that there is a very close relation to #13.

The signed MDN contains a Content-Transfer-Encoding of "7bit" which is a difference with the MDN's I got from our production environment, where in most cases the CTE is just absent.

------=_Part_2_1271687676.1532374720099
Content-Type: multipart/report; report-type=disposition-notification;
        boundary="----=_Part_0_900626805.1532374720037"

------=_Part_0_900626805.1532374720037
Content-Type: text/plain
Content-Transfer-Encoding: 7bit

The message sent to Recipient KukelsAS2ID on Mon, 23 Jul 2018 19:38:35 GMT with Subject Goodmorning New AS2 has been received, the EDI Interchange was successfully decrypted and it's integrity was verified. In addition, the sender of the message, Sender ealxs00149 at Location 10.19.26.51 was authenticated as the originator of the message.  An error occurred while storing the data to the file system.
Processor 'DefaultMessageProcessor' threw exception:
com.helger.peppol.sbdh.read.PeppolSBDHDocumentReadException: [invalid-sender-authority] The "Sender/Identifier/Authority" attribute has an invalid value: EAN.UCC
...
  [34 elements omitted -- com.helger.as2lib.processor.AbstractMessageProcessor.executeAction(AbstractMessageProcessor.java:160)]

------=_Part_0_900626805.1532374720037
Content-Type: message/disposition-notification
Content-Transfer-Encoding: 7bit

Reporting-UA: ph-OpenAS2 4.1.1-SNAPSHOT@10.1.224.89:8080
Original-Recipient: rfc822; KukelsAS2ID
Final-Recipient: rfc822; KukelsAS2ID
Original-Message-ID: <rvankuijk_40@descartes.com>
Disposition: automatic-action/MDN-sent-automatically; processed/Error:unexpected-processing-error
Received-Content-MIC: 3o35W1rw8RtitY+FH1uLVC2alh8=, sha1


------=_Part_0_900626805.1532374720037--

------=_Part_2_1271687676.1532374720099
Content-Type: application/pkcs7-signature; name=smime.p7s; smime-type=signed-data
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIAwggOsMIIC
...
Py7xs78B6moSz0E/UvASr7mc9QAAAAAAAA==
------=_Part_2_1271687676.1532374720099--

Since the specs only menion a CTE being absent or set to 8bit or binary, I tried finding where the 7bit was set. The bodyparts are created and added to the parent MimePart in

• https://github.com/phax/as2-lib/blob/master/as2-lib/src/main/java/com/helger/as2lib/util/AS2Helper.java#L142
• https://github.com/phax/as2-lib/blob/master/as2-lib/src/main/java/com/helger/as2lib/util/AS2Helper.java#L166

Both do not have a Content-Transfer-Encoding set at that time, but in the resulting signed bodypart, they do, with a value of "7bit". Debugging where this came from I found that java-mail is adding the CTE based on the actual content

In javax.mail.internet.MimeBodyPart the DataHandler is passed on and used to retrieve the enconding from it.

		if (part.getHeader("Content-Transfer-Encoding") == null)
	    setEncoding(part, MimeUtility.getEncoding(dh));

And in MimeUtility.getEncoding(...)

	/*
 * Try to pick the most efficient means of determining the
 * encoding.  If this DataHandler was created using a DataSource,
 * the getEncoding(DataSource) method is typically faster.  If
 * the DataHandler was created with an object, this method is
 * much faster.  To distinguish the two cases, we use a heuristic.
 * A DataHandler created with an object will always have a null name.
 * A DataHandler created with a DataSource will usually have a
 * non-null name.
 *
 * XXX - This is actually quite a disgusting hack, but it makes
 *	 a common case run over twice as fast.
 */
if (dh.getName() != null)
    return getEncoding(dh.getDataSource());

I expected the DispositionDataContentHandler to be used here but it is not a DataHandler (the default java mail DataHandler is used) and then "7bit" is set due to the content all being ASCII characters.

The only way I got things quickly working is by explicitly setting

aTextPart.setHeader(CHttpHeader.CONTENT_TRANSFER_ENCODING, "binary");

and

aReportPart.setHeader(CHttpHeader.CONTENT_TRANSFER_ENCODING, "binary");

in AS2Helper.java

Yes, using the 'AS2 default' would be better and maybe even retrieve it from the partnership if declared there, but I want to be sure it is something that needs to be 'fixed' on hte AS2-Lib side or that it is a bug in our software that was never noticed since everybody participating in the expesive (both in cost and time) Drummond certifications did not send MDN in this format.

[phax]

Thanks for the very detailed analysis. Let me try to sum it up:
you want a possibility to customize the Content-Transfer-Encoding of MDNs and let binary be the default.
Is this correct?

[open-gdsn]

Thanks for the very detailed analysis.

You are welcome. I help out a lot on Stackoverflow and need to put my foot where my mouth is when stating to people there that 'debugging' with opensource is not that hard (and I'm fairly into protocols so that helps too)

you want a possibility to customize the Content-Transfer-Encoding of MDNs and let binary be the default.

Not specifically. If the current behaviour of AS2-Lib is good according to the specs (cannot find it myself, already checked), I'm fine with that. Just need a reference to the spec then (or the Drummond interoperability tests, since they sometimes (often?) deviate from the spec if all agree) just to 'convince' others current behaviour is officially good. But since the default is, for outgoing messages, already 'binary' (explicitly leaving the CTE out would be fine too!!!) and can be configured on the partnership, I'd say that having the same on outgoing MDN's would not be wrong.

[phax]

I cross-checked my code and found, that I already have the possibility to define a "sending Content-Transfer-Encoding" and a "receiving Content-Transfer-Encoding".
Currently that is only used for "Messages" but not for MDNs.
Is it okay to use these settings for MDNs as well, or do we need separate configuration options?

[kukel]

Re-using these is fine, I already noticed them. But having a global one would be great as well.

Or a feature to NOT have them set by javax.mail if not set by the application.

[phax]

I was investigating and "NOT have them set" in javax.mail is not an option. I found neither a system property nor a really reliable way to disable this. So setting the default CTE should do the trick.

[phax]

Theoretically it should now be impossible to have no CTE set, because the default (binary) should always be set, except if overwritten in the partnership (using the content_transfer_encoding partnership attribute).
Please confirm that this works for you.
@zharpaz I would like to get this released as 4.1.1 before I merge your PR.

[kukel]

Confirmed that this change makes signed MDN's work with our 'old' software. Now let's see how I can try to enable async MND's in the 'as2-peppol-server'

[kukel]

Remarkable, you only set the CTE on the outer headers and the inner body parts are still 7bit. I (thought I) tried this too and did not get it to work, hence setting them on the inner bodyparts. Weird but it works

[phax]

AS2MDNReceiverModule should do the trick :) Port 10081!

[kukel]

async MDN outgoing ;-), traffic the other way around is the next step. Where do you 'discuss' things btw? Cannot imagine you use github for this?

[phax]

I have a Slack for discussion, but it's not used too heavily... will send you and invite.
Most of the things are resolved in the issues - and the rest is just working ;-)

Outgoing async MDN: AS2ClientSettings.setAsyncMDNUrl (String)

[phax]

@kukel as I couldn't find your email address, I sent you an invitation on LinkedIn...

[phax]

This is fixed for 4.1.1 - new issues as new issues pls. Feel free to reference this one. Thx