Verification of GCM tag failed: MAC check failed

[krizhanovsky]

Current ScaPy-TLS handshake ran against Tempesta FW produces the warning:

/usr/local/lib/python2.7/dist-packages/scapy_ssl_tls/ssl_tls_crypto.py:1088: UserWarning: Verification of GCM tag failed: MAC check failed
  warnings.warn("Verification of GCM tag failed: %s" % why)

However, if execute a test with following patch against ak-737 branch as of data of the issue:

iff --git a/tls/handshake.py b/tls/handshake.py
index ea588cd..36a2177 100644
--- a/tls/handshake.py
+++ b/tls/handshake.py
@@ -136,34 +136,6 @@ class TlsHandshake:
             self.exts += self.elliptic_curves
         else:
             self.exts += [tls.TLSExtension() / tls.TLSExtSupportedGroups()]
-        # We're must be good with standard, but unsupported options.
-        self.exts += [
-            tls.TLSExtension(type=0x3), # TrustedCA, RFC 6066 6.
-            tls.TLSExtension(type=0x5), # StatusRequest, RFC 6066 8.
-            tls.TLSExtension(type=0xf0), # Bad extension, just skipped
-
-            tls.TLSExtension() /
-            tls.TLSExtALPN(protocol_name_list=[
-                tls.TLSALPNProtocol(data="http/1.1"),
-                tls.TLSALPNProtocol(data="http/2.0")]),
-
-            tls.TLSExtension() /
-            tls.TLSExtMaxFragmentLength(fragment_length=0x01), # 512 bytes
-
-            tls.TLSExtension() /
-            tls.TLSExtCertificateURL(certificate_urls=[
-                tls.TLSURLAndOptionalHash(url="http://www.tempesta-tech.com")]),
-
-            tls.TLSExtension() /
-            tls.TLSExtHeartbeat(
-                mode=tls.TLSHeartbeatMode.PEER_NOT_ALLOWED_TO_SEND),
-
-            tls.TLSExtension() /
-            tls.TLSExtSessionTicketTLS(data="myticket"),
-
-            tls.TLSExtension() /
-            tls.TLSExtRenegotiationInfo(data="myreneginfo")
-        ]
         return self.exts
 
     def do_12(self):
diff --git a/tls/test_tls_handshake.py b/tls/test_tls_handshake.py
index 7f2fc6c..828a35c 100755
--- a/tls/test_tls_handshake.py
+++ b/tls/test_tls_handshake.py
@@ -58,7 +58,7 @@ class TlsHandshakeTest(tester.TempestaTest):
 
     def test_tls12_synthetic(self):
         self.start_all()
-        res = TlsHandshake(addr='127.0.0.1', port=443).do_12()
+        res = TlsHandshake(addr='127.0.0.1', port=9443).do_12()
         self.assertTrue(res, "Wrong handshake result: %s" % res)
 
     def test_many_ciphers(self):

agains Nginx with self signed certificates (just generated as described in the Wiki) with following config:

	listen 9443 ssl backlog=131072 deferred reuseport fastopen=4096;
	ssl_certificate /root/keys_selfsigned/tfw-root.crt;
	ssl_certificate_key /root/keys_selfsigned/tfw-root.key;
	resolver 192.168.1.1 valid=300s;
	resolver_timeout 5s;
	ssl_session_cache   shared:SSL:10m;
	ssl_session_timeout 10m;

the test doesn't show the warning:

est_tls12_synthetic (tls.test_tls_handshake.TlsHandshakeTest) ... 	Init test case...
Running deproxy
Running deproxy server manager
	Deproxy: Server: Start on 127.0.0.1:8000.
	Starting TempestaFW on localhost
	Teardown
	Stopping TempestaFW on localhost
	Deproxy: Server: Stop on 127.0.0.1:8000.
Stopping deproxy
Finished deproxy manager
ok

----------------------------------------------------------------------
Ran 1 test in 2.764s

[krizhanovsky]

The new warnings in current master (1b4f36d):

# ./run_tests.py -n tls.test_tls_basic
....
----------------------------------------------------------------------
Running functional tests...
----------------------------------------------------------------------

test_bad_request (tls.test_tls_basic.TlsBasic) ... 	Init test case...
Running deproxy
Running deproxy server manager
	Deproxy: Server: Start on 127.0.0.1:8000.
	Starting TempestaFW on localhost
	Starting deproxy client
Error while polling: 	Deproxy: Client: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1864) (SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1864))
Finished deproxy manager
	Teardown
	Stopping TempestaFW on localhost
	Deproxy: Server: Stop on 127.0.0.1:8000.
Stopping deproxy
ok

----------------------------------------------------------------------
Ran 1 test in 0.750s

OK

[krizhanovsky]

With #1375 sporadic tls.test_tls_handshake.TlsHandshakeTest test failures appeared (the same on current master as of a4660c6):

# ./run_tests.py -n tls
....
======================================================================
FAIL: test_10byte_transfer (tls.test_tls_handshake.TlsHandshakeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta-test/tls/test_tls_handshake.py", line 80, in test_10byte_transfer
    self.assertTrue(res, "Wrong handshake result: %s" % res)
AssertionError: Wrong handshake result: False

======================================================================
FAIL: test_old_handshakes (tls.test_tls_handshake.TlsHandshakeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta-test/tls/test_tls_handshake.py", line 259, in test_old_handshakes
    self.assertTrue(res, "Wrong old handshake result: %s" % res)
AssertionError: Wrong old handshake result: False

----------------------------------------------------------------------

Sometimes single tls.test_tls_handshake.TlsHandshakeTest passes, sometimes not:

# ./run_tests.py -n tls.test_tls_handshake.TlsHandshakeTest
======================================================================
FAIL: test_10byte_transfer (tls.test_tls_handshake.TlsHandshakeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta-test/tls/test_tls_handshake.py", line 80, in test_10byte_transfer
    self.assertTrue(res, "Wrong handshake result: %s" % res)
AssertionError: Wrong handshake result: False

----------------------------------------------------------------------
Ran 16 tests in 99.681s

FAILED (failures=1)

[vankoven]

Also caught the bag in reverse scenario: checksum calculated by scapy_tls_ssl on abbreviated handshake is rejected by Tempesta, but accepted by Nginx+OpenSSL. Reproduces on tempesta-tech/tempesta-test#175 :

ERROR: test_empty_ticket (tls.test_tls_tickets.TlsTicketTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/tempesta-test/tls/test_tls_tickets.py", line 100, in test_empty_ticket
    res = hs_abb.do_12_resume(master_secret, ticket)
  File "/home/user/tempesta-test/tls/handshake.py", line 439, in do_12_resume
    if not self._do_12_hs_resume(master_secret, ticket, fuzzer):
  File "/home/user/tempesta-test/tls/handshake.py", line 402, in _do_12_hs_resume
    self.send_recv(tls.TLS.from_records([msg]))
  File "/home/user/tempesta-test/tls/handshake.py", line 200, in send_recv
    pkt, resp)
TLSProtocolError: UNKNOWN alert returned by server: UNKNOWN DESCRIPTION

scapy_tls_ssl doesn't decrypt alerts, thus exact alert is not shown here. I also noticed two things: with the patch similar to one in the first one message (replace Tempesta with Nginx) everything is fine; due to the bug in fsm, Tempesta can read application data even if client Finished is not received and scapy_tls_ssl can receive valid responses from the server, i.e. key exchange is ok.