OpenSSL: Improve non-blocking eof test

[valga]

Follow up on #3752.

/cc @bukka

[nikic]

The AppVeyor failures here look legit for once. Fails on this test with:

========DIFF========
003+ string(0) ""
004+ string(0) ""
005+ string(0) ""
========DONE========

[valga]

I tried to work around that failure, but had no luck. I think it is related with OpenSSL/TLS version: somehow it marks socket as ready to read even when no new data is available in there. Will look into it next weekend.

[valga]

Those additional reads were caused by TLS 1.3 handshake. It now uses application data packets, so there is no reliable way for proxy (and client) to tell whether handshake is completed.

I worked it around by similar way as it was before.

[nikic]

Merged as 74888be into 7.2+. Thanks for following up on this!

[nikic]

After turning on parallel tests on AppVeyor, I'm seeing a lot of failures of this test. I wonder if this might be due to the timeout of 1s being insufficient in that scenario.

[nikic]

Test disabled for the time being, as it is too flaky on AppVeyor.

[bukka]

@nikic It should be disabled on Windows only if Travis is still ok...

@valga Has this been tested with OpenSSL 1.1.1?

[bukka]

It would be great if we could change one of the Travis builds (probably non ZTS one) to run different version of OpenSSL - ideally 1.1.1 as it's a new LTS so we have got this covered by Linux build as well...

[valga]

@bukka I checked it against latest Windows stable build (that has OpenSSL 1.1.1a).

It is possible that 1s timeout is not enough for parallel testing on AppVeyor.

Anyway, the test relies on TLS protocol: it works best with TLS 1.2, TLS 1.3 has unpredictable behavior (sometimes handshake packet leaks into stream_select() loop causing additional empty reads), TLS 1.4 could bring some more incompatibility. So it should be either locked to TLS 1.2 or disabled at all, because it makes no sense to run the test only against one version.

[bukka]

On Sat, 23 Feb 2019, 05:45 Abyr Valg, ***@***.***> wrote: Anyway, the test relies on TLS protocol: it works best with TLS 1.2, TLS 1.3 has unpredictable behavior (sometimes handshake packet leaks into stream_select() loop causing additional empty reads), TLS 1.4 could bring some more incompatibility. So it should be either locked to TLS 1.2 or disabled at all, because it makes no sense to run the test only against one version.

There is already set max version to TLS 1.2 by default so that should not be the issue.

[valga]

@bukka Windows builds (and AppVeyor) use TLS 1.3 by default.

[bukka]

On Sat, 23 Feb 2019, 18:07 Abyr Valg, ***@***.***> wrote: @bukka <https://github.com/bukka> Windows builds (and AppVeyor) use TLS 1.3 by default.

Why do you think that? There is a max version set as you can see at https://github.com/php/php-src/blob/20addf88e4363d58deeebfa24c591049fe900657/ext/openssl/xp_ssl.c#L1721 You can find the flags at https://github.com/php/php-src/blob/20addf88e4363d58deeebfa24c591049fe900657/main/streams/php_stream_transport.h#L189 Basicaly TLS 1.3 will need to be explicitly enabled. There is actually PR to add that support but it's not merged yet. So not sure how it could be enabled?

[valga]

@bukka I dumped all traffic via proxy (from the test) while debugging additional empty reads on Windows builds:

client => server
1603010200010001fc03034215e0c04b98557c4c5484afa2eb9de5d40d12ab9e054663773040205f8b90aa204c3b3bfbe69793e4f66ca24f004b8609b05b360fe8ada243b501eeb7a3b3c5a80096130213031301c02fc02bc030c02c009e00a200a3009fc027c023c013c009c028c024c014c00a006700330040006b00380039009c009dc0aec0acc0a2c09e0032c0a0c09c003c002fc0afc0adc0a3c09f006ac0a1c09d003d0035cca9cca8ccaac05dc061c057c053c05cc060c056c052c073c07700c400c3c072c07600be00bd0088008700450044c051c05000c000ba0084004100ff0100011d0000000d000b0000086275673737333930000b000403000102000a000c000a001d0017001e00190018002300000016000000170000000d0030002e040305030603080708080809080a080b080408050806040105010601030302030301020103020202040205020602002b0009080304030303020301002d00020101003300260024001d002064e0ad41deadcc311facc80c93760cb09b5308b5ebde9359453f34a5998d7f0d0015007300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

server => client
160303007a020000760303d4d0b71a0cc38c9b1ec1bc779651bd812858865c958a6fa1cd3a3b27bcce7533204c3b3bfbe69793e4f66ca24f004b8609b05b360fe8ada243b501eeb7a3b3c5a8130200002e002b0002030400330024001d00202737d93d93de391804aa294c08f981104382767cb0f4563ae9d4162f0f6e7770

140303000101

1703030017317744ac9e27202799f5454a6693baf8d4d56b955b9287

17030300421a5372d5b4435060564afe8ab43a95efacbedbfdd3e90e55dab59d061c7367da49767c763edf3ff95774793eecd13ff95bd62ff89682e542a099abce18b0a63313cf

170303043d89b9b5c41130b537b436446a82ca29c278463f087ca52ff1f283d3af3c9e7432d5c784bafe02dc5c693767358fc36a92a7800afedd70f8a26e071910d507abafda36537eba22c8851f31a616ab135d7ab468f2f7a59648dc8abbb604c934883e509554e5bc9086cd3caa9ce6c14a208370d341f1b6aaf87de070470198a391fc8a68df8ac225666560e805cc896c4bcfec158e61ebde96e14d6e157e5d58a5854afcb1a89919a4e0fa725dfa5d46a873a8079312758dca13649743ec964c2a6289ceb9c3c682bb1c47acef61b7dfaeb40eac36783aeb6f5d2ee1b779aba18a2fd405b8d3fb82a61846320573f041bcf8ee1181e64514c63507455d885305acf1a197a95ef93a634e55730e6b09ffc8a2a716f87e96c6203921d23da91644605d76d164b187e564870361da7bffe6cba7eedacdb9cd87b4dd97275831987a3c5de0bf8088cf432bcf6ca9f9ebb4fa82595406c237194a93888fe0f2d02761c8d7b6d8cf47f7974413a7c26bcc45646fdda92a2b171936dd7e36dfc387228baeff9713c0dd04c4f77c12415630f57514bdf148711e578b01030ddb83f2911ca51486cd997c07a8cbd76aca2f63d292772e98ce1bb707c4c410d4260d6aeb74306b974634597266ce616ed8ed36197117c737c1f0844396d7e117d381566813a127fe63322881deb5b05918134c76321a91b1bc925d44593664d4fdfc12223403618e4132d15c8456625f3d8a47a20e04dc81b57642e1b1b75f26cf4618e967c8c3cba7da65290d2c03f58e78b24c7115c5006b28a370dba7a94b64d359ec7141239a3ace7fd4a0a1ed9164127efd334126bad0653e5f99711d71b809b47c052a8f34cf41b0debbbc0c72e327d51eccf707bd3edfec2fb9b7c00be64c4ee26a2b818b1034b93c82e364666e9ccb1d969dd2903ee35b7521cdae1b14e29db4d20fbd9b7ad0ec64446d977844cbd22d56598072a9d9a9db104000f412d41e02dcf9a82360c06dfbccbc40c764ad90254c0ddb5c655387043ff796b2c925bb4a5ccacad5fe713270fb9ea6560beb8616c77df729ec59c56f087b6710a955eadc014bb13c9b582fb84623d7d4df7b00f86a560ffe018f55d7884bf1bc7846276944b5e494e7bac3fcc5d1fc1ba2bc355ed05c77d99121b49e1b6b7d3274393d0837b7c2043bec0d156a4218f840972ba6a4ae816fa0fa2407cd317d6fd002bc3a274f1f395ca4f601c39e2c89d515787e4a8eec8e135e577a6b5c52f524fb10baad4ac9f24c0f6801e011d8893b0758f777936842186f8a60b90aaf6cbd689061484e8bee2f0dee88584f98322e950f002c4c5a52560b4c92790ec285b3380b9d90785c185b7ec3489101da765deae758797f1630409af1efbf3bbb5f31dabc7ed8c96b1a3df7046c6696d0824d43f3ad9c0c1059970712e4160dca5dc0b4286df011e7b5450936ba7180902ab689e11a24d077773e9c493b0dc2ccd333c74947242a2dd776ecc1ab700a4a8ad7a6471d08d9ae1cbd43a1ea70205894a45ebde1

17030301194a7c4c0682c918a80a80036d62425474e6780682a36ab0a4ef00bb0f6f36e68fdff8b02cb41a8e2d01a2502736b6bbdb6bb44f65d37c521bdfc76c5f8689068e9a60e5f36d7424b938a6c83108be8b021ed4d34c2624232e1e27e17103dfd619f202732ad89b8a369d7ecdedc864c60c4338d68d3b53ac3455ff04da6cef2828e386b0468dbf86462e552c63198037b686f1d0816a30a8c66d213a6ce11801c1ac3e482489512c26ae52cf1a53439f1e824dc7f3282814c1aab1ea3009f20c3a7da977361e7405b7192c8ccb575c84308bf9a546440ce84f1c844b358627bd6e828fadcb2aab79bfd5f06a9e43cb75d50a545c556bc8e9dc89ecb325bc64367f6bfd246daf6362086812610585e219fd21918d2b82d3808360

17030300458e36658179f54889b1470007aa1b6b2f87d938a8af0cb0fcd495b502f3d6711bcfdcb8e37772ebc48c3e375f96de68c7692467e26079189975f7ddbb9921036d3770c04806

client => server
140303000101

1703030019d62c703523de7538c37fc776b5ccaab82f35ec9b48668b8bfe

17030300457230495487c311a6622e4b8a5f15570e8b47def17cb3c8de97499a9644cf21a6350e6f3372795fee4c36358e6a6d259293b2934a368e8975d218aa890bd28c69e2def29f54

server => client
17030300eaaf188f447fbb4148c124116f53d892c7290b0f33eeea7f283095cbbac79ff2df79329633fd38bba145ae2f3d68c9c2f109957f2c9a67510755e06a871168fdcf1a04fd37e4abe798d08af261ce4c3e041683ba53e6bcbb906143cab840afd1cd5c800183e8ff967073b3eba2f649376cc158ce56c4cf3c67a62f9839149b07d6724ac2092a5b515038d8c6115dd3be71f7587b0e787ce119d6e094265d02bc647a4bc830fa39cb46ce6d1a2ff2fca3a808dfff66dba7a17a2a1902e4f608e0b3f9ef0c24c94d166d51ac50329351bd22ec9cea3f8f11f40914d87217a96394ff7130cce4d942791152b6

server => client
17030300ea00bc4eb16c82884cfaa9be465b876092050923cbbebef1d432022ff07ecbbdf3443a30af4cbc2f381f9b1f3b7a3bc50b4c1616f42a7b3c32b9300de33f718ba09a90efac69cb01957a4a3ffa1088aebf5d6fb13fed0f8bca3fbb66f4342a4caaf601f0c63d87906ba39233e888a4b66203e3d543f9a919eb8d5da6c31ab2dc67766228756ca3473b9cc63aace7d9b52dd353c7b082baaf19f1867971efa16e46c5bfd5468f13feda37bccfee6f1710ef03ee1c1998807ef3ba271a06d62417b8465d04eecbf94ee8bd2043554cbd450212ed907d2a1f10e6c5dc221d5945cb2689edecb39fbdfe0b2330

server => client
170303001d8aaac883d915a271c620b2fbe01990bf5d23baa9eb72f2572b0fdde3f0

server => client
170303001324f53e54dcff3c80ec6ba90af8713afcbaf516

It's TLS 1.3. And the reason for additional empty reads is everything is now application data (0x17).

[bukka]

Not sure where you see that. I see everywhere 0303 which means TLS 1.2. TLS 1.3 is 0304 as you can see for example at https://tools.ietf.org/html/rfc8446#section-4.2.1 .

[valga]

@bukka It's called middlebox compatibility mode, that is on by default. For the reference, same session with enforced TLS 1.2:

client => server
1603010200010001fc03032a9d9974b28d152a60ad9abd40012138c2a47a28b580857a32c5500bde7cea24000082c02fc02bc030c02c009e00a200a3009fc027c023c013c009c028c024c014c00a006700330040006b00380039009c009d00a400a0003f003e003200310030c031c02dc029c025c00ec004003c002f00a500a1006a0069006800370036c032c02ec02ac026c00fc005003d0035008800870086008500840045004400430042004100ff010001510000000d000b0000086275673737333930000b000403000102000a000a0008001700190018001600230000000d0020001e060106020603050105020503040104020403030103020303020102020203000f000101001500f9000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

server => client
16030300420200003e0303da38f6cfe0e33ba0a5ad776be594f3d9398e6fc5f4c36283376931c30470435f00c02f000016ff01000100000b00040300010200230000000f000101

16030303a50b0003a100039e00039b308203973082027fa003020102020100300d06092a864886f70d01010b05003076310b300906035504061302474231123010060355040813094265726b73686972653110300e060355040713074e65776275727931263024060355040a131d4578616d706c6520436572746966696361746520417574686f726974793119301706035504031310434120666f7220504850205465737473301e170d3139303231373130313631305a170d3139303231393130313631305a3056310b3009060355040613024259310e300c06035504080c054d696e736b310e300c06035504070c054d696e736b31143012060355040a0c0b4578616d706c65204f72673111300f06035504030c08627567373733393030820122300d06092a864886f70d01010105000382010f003082010a0282010100d533113c855e3e65b42e3f24d13d8b552c351673c169ac284c3856e27bb2e81553859cc60e4dc5d95f32c7fab797bbbd4061e899dddd78c4fd3247a446a56ff986a3e3d469d766ef10fdaef78d527e260a3e7c7998998b9a2bf0f7c193f467746875d733c15b3d1d4f5ed597f6d1b2336d9de9d941eaef1fdd9ebb76d407e091a107795ebc13216d2201bcf23a269610ff54cff7d71d52ba0a34f9e78e5225296c7f5804c1c5ba2c85e3d2d70da5249b63a25408c5c4fb62a6743529e70e933102b5de1291237577f43202852083335fc84b6c25cce3661ea4b750c49ec5c904d00bbe1902cd52703dc76fc897dcde4bbeccc5bd5f6c7c556f00127e4e8315390203010001a350304e301d0603551d0e04160414d7c5ea266cc2f116e56baa079f0ff978fc373c7b301f0603551d230418301680142a30e810611afde9d35f1bdc680e85456a0d400c300c0603551d13040530030101ff300d06092a864886f70d01010b05000382010100427b11329d0b68a9fa5160fe63d1951cd8801ba976b7471480adbe3928adbcd7ca6ad1d2137a97135d1cc4371968a2355a526b972bd8b37392e0f0e77022e1533e5e13e4cb0e6a5e262bec0b11e01a7ed983ea36b6df795ac095d1a4a67c22c0d8cfb6848c163d899c463cf86439376e7dc826b191d8220e32eaca596e41f6aead018ce8e3cbd32dd85b07c82ebd29fb799876caad826c444e142ea0ce2b40d7cc3efaf2bff50865763edf818a9df9e37250094632f1201e2de1f69429519a21f85189b23e2b66741e429b4cd2f72c43ca3dd62fa85bc5d02e1ca992e97adc4d07ef1e8883729689127917f30b9b1f2d7c559ccb617f61f611894592176b7f9a

160303014d0c0001490300174104a4f8eee92a56cd3e8a6aa2a081dfc7cef2eb81124e79602bbf2da169b5b9287884931113df884d5b5a3cea851ed32f332655c912432bc51352232df8b164a82c060101006fe39a152a806668fee8ce1d40566e8b4abf89306275255d37617df5e37ff892682bea533f177e4a49339ab5d90eaa0e22cd12c9dc5dd3a21758c6ea2b08dd3b27af68961c760e88e5921b295476e155c8c7caab86a26ae3fab57e83b1877f1e4d977f322840389671c403579ae1befb50270a55bbb9a48d0e779a923a58db4e6184a2ab7f5f8bab7c43a7a818f41be3b22c4260d3ba1e6c0a7a419c316710f5f20e24e5256e05dd937b772a54ee5eac6ac2d8ac260b3baa646be99b7a555a2ab9b30d87c07483713c539f48c78937207f8347e581c3e7fdd4a16b891f25bf5f71c5274b6597bb659250e07778879ed47bf3ee8b8aaa8b63f2ad0ad89b284b2a

160303002e0d00002603010240001e06010602060305010502050304010402040303010302030302010202020300000e000000

client => server
16030300070b000003000000

1603030046100000424104d35328f76b16f54652dd72dcd5c6d4fbf0e0e57e9b117dcf5f868fa327076329ca64614485cb9f3b8c8973ec973961385408fe95aecc3520b047f2db08d5180c

140303000101

16030300287500e62d702b94db64f5e4305038b21ae78d08c478846ed03c4a6349fdcbc66076c6fdb295529733

server => client
16030300aa040000a60000012c00a01b4d7a773f663e0ea7ec7f735e06e3f925687bca08686cd37ba096aa69a6e989d3f5c377684f7b55a23261484d90c35bad04fd11f0691f4c38307fe789c4c7bcdf1ed380e5232599a583479ea89e01dd08247a369ccda15dfe471ba16fd21879f720cda7027f8d5c64b192d2063d2e226fe6d216def3bd436c887605ac532f27795db4f6b83ebd044eacfe07f043710f82dc6c545554416a9c1a33e243e46f19

140303000101

16030300281a4fcf324f30256e70a15f1a32197d905b68c014fa1e0fe1da9196ed302fa9d24b6fe7963d514805

server => client
17030300241a4fcf324f30256fb51c43cbc1fa83433471d86f3d0286ae8425186b0afa47aaecf6b8ed

server => client
150303001a1a4fcf324f302570fd1f769f69c0669d9817e33216079f1f3316

[bukka]

Ah yeah you right, it's defined by Supported Version extension in TLS 1.3 which is 002b00020304. I just tested locally with OpenSSL 1.1.1 and getting that too with 7.2. That functionality that I linked above is in 7.3+ so 7.2 will negotiate TLS 1.3.

I tested 7.3 and it uses TLS 1.2 as expected. The thing is that the test was failing in 7.4 and master when the AppVeyor parallel build got enabled so the version is not the actual issue IMO.

I think that if we used similar hack that I did before (printing empty string just once), then it might be fine. It doesn't print all the expected behavior but it still tests the actual bug (it basically still tests that it doesn't get stuck in while(1) loop)...

[bukka]

I pushed it in 01c0095 . We will see if it helps... If not we should disable it on AppVeyor till there is a better solution.