SSL problem on Intel Celeron N3350

[vejo]

• Version: 7.9.0, 6.10.2, 4.8.2
• Platform: Windows on Intel Celeron N3350
• Subsystem: crypto

We ran into a SSL problem on an Intel Skylake (even with the latest nodejs versions), which might be caused by an invalid mac calculation. The error occurs on different systems using this processor.
Here one sample error message:

101057795:error:140943FC:
SSL routines:ssl3_read_bytes:
sslv3 alert bad record mac:
openssl\ssl\s3_pkt.c:1493:
SSL alert number 20 101057795:error:1409E0E5:
SSL routines:ssl3_write_bytes:
ssl handshake failure:openssl\ssl\s3_pkt.c:659:

Attached are programs server.js and client.js, with which the problem can be reproduced, if either server or client runs on an Intel Celeron N3350: test.zip

The problem might be in conjunction with "RSA, DH, ECDH computation failures due to CVE-2016-7055 on the Intel CPU of Broadwell or later #9594", but this issue was closed as solved.

[mscdex]

/cc @nodejs/crypto ?

[vejo]

Can I transfer the issue without duplicating?
(sorry, I'm a greenhorn...)

[tniessen]

@vejo He did not suggest a transfer to another repository, he just requested feedback from the crypto team 😃

[shigeki]

I can't reproduce this issue with the attached programs. I tested it with Node-v7.9.0 on Linux and Windows with i7-6700K (Skylake). The issue of #9594 was confirmed on that machine.

#9594 is a rare bug and caused by a specific type of private keys.

You can generate several new certificate secret keys and try to test with them. If the error is still occurs even if another certificate key is used, the issue would be caused by another reason.

[vejo]

Maybe I'm wrong with the generalization 'Skylake' and the assumption 'CVE-2016-7055'.
The error actually occurred on two Acer Spin SP111-31 (Celeron N3350).

• OK: ECDHE-RSA-AES256-GCM-SHA384,ECDHE-RSA-AES256-SHA384
• 'sslv3 alert bad record mac': ECDHE-RSA-AES128-GCM-SHA256,ECDHE-RSA-AES128-SHA256
• 'sslv3 alert handshake failure': ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-ECDSA-AES256-GCM-SHA384,DHE-RSA-AES128-GCM-SHA256,DHE-RSA-AES128-SHA256,DHE-RSA-AES256-SHA256
• ('SSL23_CLIENT_HELLO:no ciphers available': DHE-RSA-AES256-SHA384,ECDHE-RSA-AES256-SHA256) = no valid proposition

I will go on testing with different client certificates.

[vejo]

The problem does not depend on key values.

I have exchanged the server key and tested with different client keys, the problem is stable with the Acer Spin SP111-31 (Celeron N3350) test system. Modified test code: test2.zip

[vejo]

Additional test results for SSL (as client with client certificate) on the Celeron N3350 system, which might depend on different SSL implementation:

• C# and .NET framework: OK
• Microsoft Edge: OK
• Mozilla Firefox: OK
• Google Chrome: did not work, possibly caused by crypto error
• nodejs: ECONNRESET during connection establishment

[shigeki]

Is a test without client cert also failed?
Does a http test (not https) between server and client work fine?

[vejo]

Test without client cert: also fails with the same error message, as long as either server or client runs on the Celeron. But it works, when cipher is restricted to ECDHE-RSA-AES256-SHA384.

Test with http: no problem

[shigeki]

So your test results suggest the issue is caused by a cipher suite from client.

They show that

• ECDHE-RSA-AES128-SHA256 NG
• ECDHE-RSA-AES256-SHA256 OK
• ECDHE-RSA-AES256-SHA384 OK
• ECDHE-RSA-AES256-GCM-SHA384 OK

How is other following ciphers? They are all included in a default cipher list in Node.

• ECDHE-RSA-AES128-GCM-SHA256
• AES256-GCM-SHA384
• AES256-SHA256
• AES256-SHA
• AES128-GCM-SHA256
• AES128-SHA256
• AES128-SHA

[Edit: two more ciphers was added to OK]

[shigeki]

And please remove client cert auth for test simplicity.

[vejo]

Here is the result:

ECDHE-RSA-AES128-SHA256 Error: write EPROTO 101057795:error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:openssl\ssl\s3_pkt.c:1493:SSL alert number 20
101057795:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:openssl\ssl\s3_pkt.c:659:

ECDHE-RSA-AES256-SHA256 Error: This socket is closed
ECDHE-RSA-AES256-SHA256 Error: 101057795:error:140740B5:SSL routines:SSL23_CLIENT_HELLO:no ciphers available:openssl\ssl\s23_clnt.c:508:

ECDHE-RSA-AES128-GCM-SHA256 Error: write EPROTO 101057795:error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:openssl\ssl\s3_pkt.c:1493:SSL alert number 20
101057795:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:openssl\ssl\s3_pkt.c:659:

ECDHE-RSA-AES256-GCM-SHA384 OK
ECDHE-RSA-AES256-SHA384 OK
AES256-GCM-SHA384 OK
AES256-SHA Error: write EPROTO 101057795:error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:openssl\ssl\s3_pkt.c:1493:SSL alert number 20
101057795:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:openssl\ssl\s3_pkt.c:659:

AES256-SHA256 Error: write EPROTO 101057795:error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:openssl\ssl\s3_pkt.c:1493:SSL alert number 20
101057795:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:openssl\ssl\s3_pkt.c:659:

AES128-GCM-SHA256 Error: write EPROTO 101057795:error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:openssl\ssl\s3_pkt.c:1493:SSL alert number 20
101057795:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:openssl\ssl\s3_pkt.c:659:

AES128-SHA256 Error: write EPROTO 101057795:error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:openssl\ssl\s3_pkt.c:1493:SSL alert number 20
101057795:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:openssl\ssl\s3_pkt.c:659:

AES128-SHA Error: write EPROTO 101057795:error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:openssl\ssl\s3_pkt.c:1493:SSL alert number 20
101057795:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:openssl\ssl\s3_pkt.c:659:

Test code: test3.zip
node -v: v6.10.2
Celeron as client

[shigeki]

Your tests shows that there are issues in SHA-1 and SHA256 computation but not SHA-384.
In order to conform if this is right, I would like to ask you to run the following script and get hash results of sha1,256 and 384.

const crypto = require('crypto');
const sha1 = crypto.createHash('sha1');
const sha256 = crypto.createHash('sha256');
const sha384 = crypto.createHash('sha384');

const text = 'hello world';

sha1.update(text);
sha256.update(text);
sha384.update(text);

// sha1:  2aae6c35c94fcfb415dbe95f408b9ce91ee846ed
console.log('sha1: ', sha1.digest('hex'));
// sha256:  b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9
console.log('sha256: ', sha256.digest('hex'));
//sha384:  fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd
console.log('sha384: ', sha384.digest('hex'));

Looking at the spec of https://ark.intel.com/products/95598/Intel-Celeron-Processor-N3350-2M-Cache-up-to-2_4-GHz, it has AES-NI feature but AVX does not seems to be supported.

There is a possibility that openssl assembler build in node would leads this issue.

I'm now building node binary of Windows without using openssl asm.
I would like to ask you to run the code with that binary to see if asm causes the issue.

Is your windows 64bit version? If not, please tell me it.

[shigeki]

Here is a node binary of Windows 64bit without asm support at the latest master branch(8.0.0-pre). A openssl command (openssl-cli.exe) is also included for a tool of the further investigation. Please test with this node.exe.

node_noasm_bin.zip

[vejo]

Result of hasing script:

sha1:  2aae6c35c94fcfb415dbe95f408b9ce91ee846ed
sha256:  7a5380ee7a5380eee2efcde9e2efcde9c484efe37a5380ee9088f7ace2efcde9
sha384:  fdbd8e75a67f29f701a4e040385e2e23986303ea10239211af907fcbb83578b3e417cb71ce646efd0819dd8c088de1bd

SHA256 is wrong

[vejo]

Result with node v8.0.0-pre is fine!

[vejo]

The initial hint to processor-openssl incompatibility came from an antivirus expert. He listed following processors:

• Intel Pentium J4205
• Intel Celeron J3455
• Intel Celeron J3355
• Intel Pentium N4200
• Intel Celeron N3350
• Intel Celeron N3450
• Intel Atom x7-E3950
• Intel Atom x5-E3940
• Intel Atom x5-E3930

Possibly these processors also are affected, but I don't know how to check the specs.

[shigeki]

Could you try to run the test on node-7.9.0 with setting the following environment?

OPENSSL_ia32cap="~0x1000000000000000:~0"

It would disable avx functionalities.

[vejo]

After C:\Windows\System32> SET OPENSSL_ia32cap=~0x1000000000000000:~0 SHA256 is fine with node v7.9.0.

[shigeki]

Thanks. It proves that the issue occurs on the cpu that has no avx support.

It needs to investigate the issues comes from Node or OpenSSL but unfortunately I do not have such cpu now to test. Probably it takes some time.

That environment is a tentative workaround. Please use it until the issue is resolved.

[vejo]

Shigeki, thanks a lot for your expert aid!
In fact the SHA256 malfunction could have a severe impact, we have to investigate how far systems in the field are affected.
But for now we have a exact analysis and a workaround, that's a lot more than one day before.
Assistance from your side could not have been better.

[vejo]

One more information.

The reason why we went for the Intel Celeron was, that on one computer in the field (the Acer Spin) nodejs crashed at launch - but the latest nodejs version could start.

So there must have been a change between v6.6.0 and v6.7.0, that fixed the startup problem on the Celeron, but also could have opened up the SHA256 malfunction.
An analyzis of this change might help to improve future processor compatibility checks.

[shigeki]

@vejo We've made several tests on other Intel cpus but the issue have not been reproduced yet. See the current summary in nodejs/build#704 (comment).

In order to identify if the issue is caused by OpenSSL or Node, I've just made openssl command binary from the original openssl sources in the here attached openssl.zip .

Please test the following command to see if sha256 hash is the same value.

C:\Users\ohtsu\Desktop>echo|set /p="hello world" | openssl dgst -sha256
WARNING: can't open config file: /usr/local/ssl/openssl.cnf
(stdin)= b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9

If it returns the right value, please test it with the openssl-cli.exe command included in zip file of #12691 (comment) which is built with the openssl sources included in Node. If this value is wrong, the issue is caused by Node.

[vejo]

The SHA256 is correct with both

• the new openssl.exe (I had to add ssleay32.dll and libeay32.dll from another installation) and
• openssl-cli.exe (out of node_noasm_bin.zip 3 days ago)

[shigeki]

I had to add ssleay32.dll and libeay32.dll from another installation

Sorry, I will make new binaries for both of them later and ask you to test them again.

[shigeki]

I made two binaries again.
One is openssl102k\openssl.exe and the others is node\openssl-cli.exe. openssl_binaries.zip
Please test sha256 hash result above.

[vejo]

The result (openssl.exe with its dlls in subdir \o):

C:\Test>echo|set /p="hello world" | openssl-cli dgst -sha256
WARNING: can't open config file: /usr/local/ssl/openssl.cnf
(stdin)= 7a5380ee7a5380eee2efcde9e2efcde9c484efe37a5380ee9088f7ace2efcde9
C:\Test>cd o
C:\Test\o>echo|set /p="hello world" | openssl dgst -sha256
WARNING: can't open config file: /usr/local/ssl/openssl.cnf
(stdin)= b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9

[shigeki]

Thanks. The result shows that the issue is caused by Node. I will investigate this further.

[MylesBorins]

@vejo this has not landed on v6 or v4. With version 4 being in maintenance mode we only do releases for severe bugs and security vulnerabilities.

/cc @nodejs/crypto would you consider this a severe bug?

[tniessen]

@MylesBorins Yes, I think this should be backported. It effectively makes a part of our API useless on certain processors so I consider this a severe bug. If we decide not to fix it, we should add a note about the problem to the website.

[vejo]

@MylesBorins

Thank you for calling backporting into question - let me argue:

Nowadays computing a hash value is used as basic mathematics.
Everybody expects, that the result of 1 + 1 is 2, on every computer system.
That is also true for hash functions, except that this cannot be simply verified by mental calculation.
So, just delivering a wrong result is really, really dangerous.
More even, as it concerns CPU architecture, that is hitting the market just now.

Until v6.6.0 node.exe just crashed on this Intel Celeron platform - this was bad, but it's got much worse:
Since v6.7.0 node is starting, running, but computing wrong results.
Within our software solution it means, that a small percentage of the systems in the field are storing invalid chained data onto disk.
Every day, undetected, in a legal relevant function.

I do understand, that not every bugfix has to be made in every version.
But what is "LTS" actually worth, if it not even guarantees correct SHA256 values?
For Boron and Argon in my opinion a fix is inevitable.
Or - at least - let node crash, like it did up to v6.6.0.

Not for us - knowing the reason we now will be able to do a workaround, even without an official fix.
But for those developers already having chased an unexplainable SSL problem in their program the last three long nights.
Please, let me know your decision.

[MylesBorins]

This seems very reasonable. We should plan a v4 release in the near future and do an audit for similar experience breaking bugs. Thanks for taking the time to respond.

…

On Wed, Jun 14, 2017, 5:13 PM vejo ***@***.***> wrote: @MylesBorins <https://github.com/mylesborins> Thank you for calling backporting into question - let me argue: Nowadays computing a hash value is used as basic mathematics. Everybody expects, that the result of 1 + 1 is 2, on every computer system. That is also true for hash functions, except that this cannot be simply verified by mental calculation. So, just delivering a wrong result is really, really dangerous. More even, as it concerns CPU architecture, that is hitting the market just now. Until v6.6.0 node.exe just crashed on this Intel Celeron platform - this was bad, but it's got much worse: Since v6.7.0 node is starting, running, but computing wrong results. Within our software solution it means, that a small percentage of the systems in the field are storing invalid chained data onto disk. Every day, undetected, in a legal relevant function. I do understand, that not every bugfix has to be made in every version. But what is "LTS" actually worth, if it not even guarantees correct SHA256 values? For Boron and Argon in my opinion a fix is inevitable. Or - at least - let node crash, like it did up to v6.6.0. Not for us - knowing the reason we now will be able to do a workaround, even without an official fix. But for those developers already having chased an unexplainable SSL problem in their program the last three long nights. Please, let me know your decision. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12691 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAecV7hPurNBUBLIBfIp9oRe4xoVdFkkks5sEE0CgaJpZM4NJ3Id> .

[vejo]

Thank you!

[gibfahn]

@vejo just wanted to add that that's a really informative answer, exactly what we need to make a decision about whether it makes sense to include this in a v4.x release. Thanks a lot!

[shigeki]

@MylesBorins @gibfahn This can be fixed by upgrading openssl-1.0.2l. I labelled LTS-watch in #13233 but is not yet backported. Did I missed something?

[MylesBorins]

@shigeki primarily because we haven't done an audit for either branch in the last 13 days (since it landed).

Upgrading openssl on v4.x sounds like a GREAT idea

does #12913 need to land if we upgrade openssl?

Would you be willing to open a pr against v6 and v4 so we can land the change?

[shigeki]

does #12913 need to land if we upgrade openssl?

No, it was fixed in the newer openssl.

Would you be willing to open a pr against v6 and v4 so we can land the change?

I've just submitted #13695 for Node-v6 and #13696 for Node-v4. Please take a look at them.

[tniessen]

If we backport this to v4 and v6, we might want to add a prominent warning about this issue. As far as I know, the new releases will compute different checksums depending on the processor model, which can cause severe problems in a variety of applications.

[vejo]

Here a quick and dirty workaround for SHA256 for node 4.x.
As we already use sjcl.js (Stanford Javascript Crypto Library) as module, it's not a big overhead.
But this will not solve problems with other built in modules (TLS, HTTPS) and is also not useful for hashing streams.

var Crypto = require('crypto')
compatibility()

...

function compatibility() {
    if (Crypto.createHash('SHA256').update('').digest('hex') == 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855') return
    console.log('Crypto', 'processor incompatible, try SET OPENSSL_ia32cap=~0x1000000000000000:~0')
    var ori_createHash = Crypto.createHash
    Crypto.createHash = function (algorithm) {
        return algorithm.toUpperCase() == 'SHA256' ? new compatibility_SHA256() : ori_createHash(algorithm)
    }
}
function compatibility_SHA256() {
    this.buf = new Buffer(0)
    this.update = function (data, inputEncoding) {
        this.buf = Buffer.concat([this.buf, new Buffer(data, inputEncoding)])
        return this
    }
    this.digest = function (encoding) {
        var sjcl = require('./sjcl.js')
        var ret = sjcl.codec.hex.toBits(this.buf.toString('hex'))
        ret = sjcl.hash.sha256.hash(ret)
        ret = new Buffer(sjcl.codec.hex.fromBits(ret), 'hex')
        return encoding ? ret.toString(encoding) : ret
    }

}

[ghost]

Could you try to run the test on node-7.9.0 with setting the following environment?

OPENSSL_ia32cap="~0x1000000000000000:~0"

It would disable avx functionalities.

Works's fine af! Thanks for the genius solution!