RSSBUS partner test

[mhofland]

Hello Philip,

first of, great work on this project!

I hope you can help me with a problem I encountered.
I have been testing your library against another AS2 product called RSSBUS. This product comes default with 4 test files of which 1 causes a "org.bouncycastle.cms.CMSSignerDigestMismatchException: message-digest attribute value does not match calculated value" authentication error every time. The other 3 default test files are processed fine.
I reproduced the issue in your latest standalone server version (2.2.3-SNAPSHOT) and used the default (certs.p12) certificates for the test. It seems that any uploaded test file that has linefeeds without carriage returns causes this exception.
I am running as2-server in an IDE on Windows.

Any help would be appreciated.

Kind regards,
Martijn

[phax]

You are talking about sending or receiving an AS2 message?
Is the error when verifying the signature of the message or the MDN?
Can you provide the test files here, or are they private? If they are private, can you send them to me by mail?

[mhofland]

Wow fast response 👍

I sent the message from RSSBUS and the exception is thrown when receiving it (verifying the signature on the message).
Sending the message (from RSSBUS) unsigned and requesting a signed MDN results in a MIC mismatch check on the RSSBUS side:
FAILURE: The Message Integrity Check returned by the server is incorrect. Expected: 4LN4laBGpM63lKYBmsCxJFSbLlA=; Received : XJJZ+0wVwKViL7c9sc+j4zHMJe8=

The test file is not private. It is created by RSSBUS, and I have uploaded it here with extension PNG as that is required by GitHub apparently:

And the AS2 message that is created by RSSBUS for this content is:

[mhofland]

I could be more clear here I think. When sending a signed message from RSSBUS to openas2 server, the problem occurs when verifying signature. For this one I attached the test file.

When sending an unsigned message from RSSBUS to openas2 server and requesting a signed MDN, the problem occurs when RSSBUS verifies the signature on the response MDN.

So I was describing two different scenarios in my previous post.

[phax]

So this is the test request - just for improved clarity:

AS2-To: OpenAS2A
AS2-From: OpenAS2B
AS2-Version: 1.2
EDIINT-Features: multiple-attachments, AS2-Reliability
Date: Tue, 20 Oct 2015 13:27:21 GMT
Message-Id: <0dbb26a8046f4887_-14540955_14ebb365888_-7e99@0dbb26a8046f4887_-14540955_14ebb365888_-7e98>
Disposition-Notification-To: testas2@freeas2.com
Disposition-Notification-Options: signed-receipt-protocol=optional, pkcs7-signature; signed-receipt-micalg=optional, sha1
MIME-Version: 1.0
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="boundaryEHSgAQ=="

--boundaryEHSgAQ==
Content-Type: application/edi-x12
Content-Disposition: Attachment; filename="test_data_2.edi"

ISA*00*ssssssssss*00*rrrrrrrrrr*ZZ*testas2     *zz*testas2      *961007*2013*U*00200*754320000*0*T*:
GS*PO*S1S1S1S1S1S1S1S*R1R1R1R1R1R1R1R*961007*2013*000000004*X*003050
ST*850*000040001
BEG*00*BE*2a*43324234v5523*961007*23tc4vy24v2h3vh3vh*ZZ*IEL*09*RE*09
CUR*11*TRN*5656*65*566*IMF*006*961007
REF*6A*433r1c3r34r34c3312qctgc54*Reference Number
PER*AA*Hans Gutten*CP*1.322.323.4444*****rgg4egv4t4
TAX*4tgtbt4tr4tr*GL*ghgh*********G*C
FOB*TP*CA*USA*02*DOM*CC*Regular Locations per Terms
CTP*DE*C04*453*25000*D9*SEL*23214*23432423423*ES*42243423
SAC*A*B000*AE*3545*3442300
CUR*11*767*7767*65
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
PO1*111-aaa*1000000*AS*90.00*BD*AK*234235v3534q6f3534v4353453vq3q32***********A1*ytrh
CTT*1
SE*32*000040001
GE*1*000000004
IEA*1*000000001

--boundaryEHSgAQ==
Content-Type: application/pkcs7-signature;      name="smime.p7s"
Content-Disposition: attachment; filename="smime.p7s"
Content-Transfer-Encoding: base64

MIIEowYJKoZIhvcNAQcCoIIElDCCBJACAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHAaCCAr0wggK5MIIBoQIEVAgwHjANBgkqhkiG9w0BAQUFADAgMQswCQYDVQQGEwJBVDERMA8GA1UEAwwIT3BlbkFTMkIwIBcNMTQwOTA0MDkyNTUwWhgPMjExNDA4MTEwOTI1NTBaMCAxCzAJBgNVBAYTAkFUMREwDwYDVQQDDAhPcGVuQVMyQjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJE+izonf1SVOday/KqIECp0Vq+eHB1MLQ9VzylQXSpfBRQz1425TzDIHAG94NqywsBg3J31DJJ32Ve7F4ps2kRQIlTjPt340QMR9OJiAdsdHdPH1i9brns354f775ul1gF4ikWpoYmjwR5tnintH8NFr/v3UO/phQR4X3GxJwmNV50H6VbbQerreW3yBYFQ/unNkFgjam4UustHnaGW2RFZXIGd9+14ivUJZTjagcC1b/hVI/hbaz8Ikd/u5ifsRl47jaoQN590epAh2SWyx0nfCUuLLaHBpNw/1Hbug5O6o2ytxeKbE0ySa6wDjabgjz5j5bLME79eYDNzVAE7qJsCAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAdwN9YIO7AaKkcaS63Iiatl+kr3alV7AIRPFzTZCUU0S37XSNLEvOJRvUfKWp6GcCYjRHoTVvXdtvlQJR2pWeVYmimI1opG7eJRmoNM/M8jeeax0KaQQdszcqx7gNyFXEdTpWpxihJSI7cJ5g9VjsX4VsENn9C/B4UJozP+Y2296HAOY+Z4ryd7p73fKij7vcUsTAcu1zhQI/2ZcQTQQjR7y3ZgyM6fO3bkyujf8Ngruco9j9lCjPTak3tUG+jw+UY8suDtCjXYR7hWX1hv2mE941vcNdD6gpcH/pvbcE9G8DfLtUqTirnuJ6pCWr7p0DTckHw5GQ8E4NRM7/MC0D7jGCAa4wggGqAgEBMCgwIDELMAkGA1UEBhMCQVQxETAPBgNVBAMMCE9wZW5BUzJCAgRUCDAeMAkGBSsOAwIaBQCgXTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNTEwMjAxMzI3MjFaMCMGCSqGSIb3DQEJBDEWBBRq3/2/+PQG2HopME3GRSa/XEeE6jANBgkqhkiG9w0BAQEFAASCAQBw7rsjUkahafIBNMEFZBnaO2Nzj1Ybr5e/5RmgdGv32/rgI81QN5GIAnijCEzqSGpDkfK5QLfVWC5wjQISuaOrEsJAM3x2av/9VjzE4ZLQFzvD9yHfafPwulbIkn9QyTROjVhgr574ETg+ZQ2FEB+VN2N4NJMNeNJxCpBoCTBq2alt8kS9sHoYq3vH4ic96B3VhgZmsV+YbSx3ItTZoW2fNUSspcufIJdSfSdGX1TkSakuVIcsT7Y9a7WlJpZ3PL77ZIJ8ow1lQLVVl8WqP1WeoVSpyQ/iAC1HOHF9tvFZQyEoRpuRo7ONizQ25+nDLvjFwXLNMl+IDP5H3Q5Plzjc

--boundaryEHSgAQ==--

[mhofland]

Yes and please note that the test request (content) uses LF to end a line. The problem was reproducible with a simple 1 line test file that had one LF in it (instead of CRLF) when sent as as2 message with signature from RSSBUS.

[phax]

I think the problem is something else, but I'm just investigating. Give me 2 more minutes :)

[phax]

My thought was, that if the signature contains a certificate, that this certificate is potentially the wrong one and therefore the verification fails. But that was not the case.

[phax]

No matter whether the certificate from the MIME body part or the certificate from the partnership is used, I get an error. If you

1. update your as2-server to the latest
2. and add CryptoVerifyUseCertificateInBodyPart="false" into your config.xml (in the openas2 root element)
3. please try again.

[phax]

And for sending your may include CryptoSignIncludeCertificateInBodyPart="false" in the same place.

[mhofland]

I get the same exception with both the attributes enabled.

[phax]

Now it gets difficult. Do you know what kind of certificate RSSBUS is using to send signed messages? Is an encrypted test message contained? Does the decryption work?
A common point of differentiation is, whether to include the header lines in the digest or not. The algorithm to determine this: if encryption, signing or compression is enabled, than the header lines are included, otherwise not. So we need to find the possibility to disable this a try again.
Did you use the provided test certificates or are you using your own certificate suite?

[mhofland]

I have configured RSSBUS as partner OpenAS2B and uploaded certs.p12 as private key and uploaded OpenAS_B.cer as public certificate. Encryption and decryption works for all test messages.
I first encountered this problem with my own certificates, but I have been solely using the certs.p12 certificates with today's tests.

I am not so sure the headers are the issue. Uploading the same test file but with carriage returns added, results in a matching MIC for that particular test file. However taking one of the test files that processes correctly and changing the CRLFs in LFs result in the same MIC mismatch exception. (all content changes done before RSSBUS creates as2 message and calculating digest of course).
But I am willing to try any tests you require.

I will be going home soonish, so I will be happy to read any new comments tomorrow.

[phax]

This is my spare time too :) Don't expect too much ;-)

[phax]

So do I understand you correctly: RSSBUS seems to create the signature and MIC with "CRLF" (\r\n) characters, but the transmission happens wit "LF" (\n) characters only and that's why the mismatch occurs? Is that what you are trying to say?

[mhofland]

It seems that RSSBUS calculates a different hash than as2lib when the content has LF (\n) opposed to CRLF (\r\n).
I do not know why the mismatch occurs, but it may have to do with canonicalization as mentioned in 7.3.1 of RFC4130

From: Philip Helger [mailto:notifications@github.com]
Sent: 21 October 2015 10:43
To: phax/as2-lib
Cc: Martijn Hofland
Subject: Re: [as2-lib] RSSBUS partner test (#12)

So do I understand you correctly: RSSBUS seems to create the signature and MIC with "CRLF" (\r\n) characters, but the transmission happens wit "LF" (\n) characters only and that's why the mismatch occurs? Is that what you are trying to say?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/12#issuecomment-149822527.

[phax]

I stumbled upon this issue which I think relates to the same problem:
http://stackoverflow.com/questions/25562613/smime-cant-be-validated-by-receiver-when-using-newer-version-of-bouncy-castle

[phax]

As the example message does not contain a Content-Transfer-Encoding I suggest you add the attribute content_transfer_encoding_receive="8bit" to the respective partnership and try again.
This means that the transfer encoding is "8bit" if not specified differently in the message.
Forget this. I simply don't perform canonicalization. Give me a few minutes...

[mhofland]

Take all the time you need ☺ I am very glad you are so invested in your product!

I created a few junit tests concerning the problematic test messages in attached zip.

Perhaps they can be of use.

Kind regards,
Martijn

[phax]

Okay so here's my update: the MIC calculation should now honor the line ending canonicalization (if the content transfer encoding is not binary).
I scanned through the BC 1.53 code and found the canonicalization to be used with the same rules (only if CTE != "binary")
So I'm a little bit lost here, because I can't find another (obvious reason) why this wouldn't work.

@mhofland: ZIP? what ZIP?

[phax]

Please check the example mentioned in the previous commit. I managed to get the verification done by handling the line endings correctly - see the overlong string. Requires both components to be 2.2.3-SNAPSHOT

[mhofland]

I tried your test, but it fails for me on identifying the sender. For some reason it does not get the first header across (AS2-To), so it responds with processed/Error:authentication-failed

If I duplicate that first header it continues, but ends up with the verification error again: processed/Error:integrity-check-failed

Did I do something wrong and did you get it verified?

[phax]

Yep sorry - my mistake. I forgot the HTTP status line. Current version has it, and validation still fails.
Is the example message signed with the private key of OpenAS2A that corresponds to the example keystore???

[mhofland]

I configured RSSBUS as partner OpenAS2B and uploaded certs.p12 as private certificate.
I assume it uses the private key of OpenAS2B.

From: Philip Helger [mailto:notifications@github.com]
Sent: 21 October 2015 17:10
To: phax/as2-lib
Cc: Martijn Hofland
Subject: Re: [as2-lib] RSSBUS partner test (#12)

Yep sorry - my mistake. I forgot the HTTP status line. Current version has it, and validation still fails.
Is the example message signed with the private key of OpenAS2A that corresponds to the example keystore???

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/12#issuecomment-149927033.

[phax]

I tried several variants in the example, but can't get it working :( It's strange because the decryption works. So potentially there is some othe kind of canonicalization going on???

[kattelus]

Hi.

I haven't followed every part of discussion. Anyway I have before struggled against one implementation which didn't write content encoding for actual message. In bouncy castle (smime) this kind of message is not understood as binary by default, but in as2 specification it is defined that if encoding is missing default encoding should be assumed as binary.

In BCCryptoHelper I had to make following change to get signature verification to work:
SMIMESignedParser aSignedParser = new SMIMESignedParser(new JcaDigestCalculatorProviderBuilder().build(), aMainPart, "binary");

So if this doesn't make sense in this case just omit it. Just wanted to share if it helps.

[phax]

You are my hero! Thanks for pointing out that difference!!! It solved the problem!

[mhofland]

The junit tests are now successful, so it succeeds in verifying the message. So many thanks to both of you! 👍
However for the end-to-end test I now get complaints from RSSBUS that it cannot match its own MIC with the Received-Content-MIC that is read from the MDN:

FAILURE: The Message Integrity Check returned by the server is incorrect. Expected: at/9v/j0Bth6KTBNxkUmv1xHhOo=; Received : nNEzTj34B9XDL07aBLQV9lHHZmQ=

Is it possible that the fix is not applied to calculating MIC for MDN creation or is there still a difference in message digest calculation?

[mhofland]

Just to confirm that this is not a RSSBUS issue, I configured RSSBUS to send the same message to another Drummond certified product my company has access to, called XFB Gateway. And it calculates the same MIC as RSSBUS: (Received-content-MIC: at/9v/j0Bth6KTBNxkUmv1xHhOo=, sha1)

So I am assuming that that is the correct MIC.

[mhofland]

I got it to match MIC by disabling the code that creates a CRLFOutputStream in BCCryptoHelper.

I am not sure of the implications, but the default test files from RSSBUS are processing correctly now. (calculate the same MIC)
What do you think, is that a possible fix?

[phax]

Thanks for testing. This was one thing I added yesterday because I though the stuff for the MIC must also be canonicalized, but it must not. So basically your suggested fixed is exactly what I did. Can we therefore close the issue?

[mhofland]

Thank you for all the hard work!
I will close the issue.

kind regards,
Martijn

[phax]

Thanks for confirming that it works. Seems like I can create a release 2.2.3 now :)

[sheebapadma]

I see this issue fixed some time back. But I still get this error when running against RSSBus
I use as2-lib version - 4.2.0
this is the error I get on RSSBus, when data is received from RSSBus
"The Message Integrity Check returned by the server is incorrect. Expected: /kaHyzhfEPwww0JRDi1OfFcM7k4=; Received : slhWAt5h8ep9V1yNRm6auNDBS1A="

[phax]

Hi. Please try the latest 4.4.0 version. I finally found the issue that caused the MIC mismatch when using Base64 encoding.

[sheebapadma]

Thanks.. I will test it with 4.4.0 and will let you know..

[sheebapadma]

this is what I was doing to read the http payload
// read payload
final String sReceivedContentType = AS2HttpHelper.getCleanContentType (as2Message.getHeader (CHttpHeader.CONTENT_TYPE));
byte[] bytes = HTTPHelper.readHttpPayload(data, responseSink, as2Message);
final DataSource aPayload = new ByteArrayDataSource(bytes, sReceivedContentType, null);

I see this method 'isHTTPHelper.readHttpPayload()' no longer exists in the new version. May I know what is the alternative for getting the DataSource.

I tried this:
final DataSource aPayload = HTTPHelper.readHttpRequest(new CustomAS2InputStreamProvider(data), responseSink, as2Message);

But that gives java.io.IOException: Invalid HTTP Request (0� *�H�

[sheebapadma]

Unfortunately upgrading to 4.4.0 didn't work..
I still get the error on RSSBus
The Message Integrity Check returned by the server is incorrect. Expected: /kaHyzhfEPwww0JRDi1OfFcM7k4=; Received : slhWAt5h8ep9V1yNRm6auNDBS1A=

[phax]

Okay. Thanks for checking. Than I need the following information:

• is there a public test endpoint from RSSBUS to test against?
• What Content-Transfer-Encoding are you using?
• Are you signing and if yes with what algorithm?
• Are you encrypting - if yes with what algorithm?
• Are you performing the encryption manually in your code or are you relying on AS2Client class?

[sheebapadma]

Thanks for looking into it. Please let me know if there is any more info I can provide you on this.

• is there a public test endpoint from RSSBUS to test against? Unfortunately No.
• What Content-Transfer-Encoding are you using? I can see in the code, its using BASE64
• Are you signing and if yes with what algorithm? Yes,sha1
  Are you encrypting - if yes with what algorithm? No
  Are you performing the encryption manually in your code or are you relying on AS2Client class?
  Nothing is manually done on our code.. I just call
  AS2ReceiverHandler#handleIncomingMessage()

[phax]

Are these X-VaultAPI-... headers eventually added by some proxy server inbetween? If this is not to be shared with the public, you also email me (philip at helger.com)

[sheebapadma]

Thanks Philip.
I think I found the issue.
The issue is at calculating MIC. MIC expects header hash, which is determined by the flag bIncludeHeadersInMIC.
aMIC = getCryptoHelper ().calculateMIC (aMsg.getData (),
aDispositionOptions.getFirstMICAlg (),
bIncludeHeadersInMIC);
final boolean bIncludeHeadersInMIC = aMsg.partnership ().getSigningAlgorithm () != null ||
aMsg.partnership ().getEncryptAlgorithm () != null ||
aMsg.partnership ().getCompressionType () != null;

In AS2ReceiverHandler#handleIncomingMessage(), aMsg gets some value populated, but the above aMsg.partnership algorithm values are not set. So bIncludeHeadersInMIC = false. If I explicitly set this value to true, its working fine..

I can see AS2ReceiverHandler.verify() method setting
aMsg.attrs ().putIn (AS2Message.ATTRIBUTE_RECEIVED_SIGNED, true); but the algorithm details are not set in aMsg.partnership ().

Is this something that can be fixed in the library itself?

[phax]

Congratulations on finding it.
In this case the message is signed, so yes, the headers must be included in the MDN. The partnership is the "point to point" configuration - this depends on your scenario, if this is static or dynamic.
With msg.setPartnership (...) (see class AbstractBaseMessage) you can set the information explicitly.

I will see, if I find a way to determine the used algorithms. This is hidden somewhere in BouncyCastle atm.

[sheebapadma]

I understand that if message is signed, headers must be included in MIC calculation. However, the verification on if the message is signed or not is happening down in the code AS2ReceiverHandler#handleIncomingMessage().verify(). and aMsg attr() is updated with
aMsg.attrs ().putIn (AS2Message.ATTRIBUTE_RECEIVED_SIGNED, true);

would it be safe to extend bIncludeHeadersInMIC to the following (one last check on isSigned) before calling calculateMIC()

final boolean bIncludeHeadersInMIC = aMsg.partnership ().getSigningAlgorithm () != null ||
aMsg.partnership ().getEncryptAlgorithm () != null ||
aMsg.partnership ().getCompressionType () != null ||
aMsg.attrs ().get (AS2Message.ATTRIBUTE_RECEIVED_SIGNED);

[sheebapadma]

Thanks Philip for all the help. Really appreciate your prompt replies.
I could pass along the signingAlgorithm in partnership, which fixed the issue. In fact it worked with version 4.2.0 itself. Do you see any reason for me to upgrade to 4.4.0

[phax]

Glad it worked. Nevertheless I recommend you to switch to v4.4.0 because it fixes an issue with MIC calculation if Base64 CTE is used.