DATA frame does not have end of stream flag set after sending file

[ketulsheth2]

I am using h2load to test my apache2 server and see when I send 100k requests using 100 clients, for some streams apache server doesn't set the end of stream flag in DATA frame. In the example below, response HEADER contains content-length as 11321, however, in the mod_http2 debug logs even after sending DATA frame with length 11321, no end of stream flag is seen. One in maybe 100k streams have this issue. Also, I don't see any empty DATA frame after this with the end of stream flag.

[stream_id=15] :status: 200
[stream_id=15] date: Thu, 21 Jun 2018 19:10:37 GMT
[stream_id=15] server: Apache/2.4.33 (Ubuntu)
[stream_id=15] strict-transport-security: max-age=63072000; includeSubdomains
[stream_id=15] x-frame-options: DENY
[stream_id=15] x-content-type-options: nosniff
[stream_id=15] last-modified: Tue, 12 Jun 2018 22:33:46 GMT
[stream_id=15] etag: "2c39-56e79759112f9"
[stream_id=15] accept-ranges: bytes
[stream_id=15] content-length: 11321
[stream_id=15] vary: Accept-Encoding
[stream_id=15] content-type: text/html

[Thu Jun 21 11:49:40.549970 2018] [http2:debug] [pid 10060:tid 140103116588800] h2_session.c(544): [client 192.168.40.6:47936] AH03068: h2_session(146,BUSY,4): sent FRAME[DATA[length=11321, flags=0, stream=865, padlen=0]], frames=567/1716 (r/s)

[icing]

Having the complete DATA without EOS flag is normal, however - as you noted - there should then follow a last DATA frame with that flag and length 0.

If the client decides it has all it needs, it may initiate a connection shutdown before the server has sent this last frame. That would be legal behaviour as well. Do you see this missing EOS somewhere randomly or at the end of connections?

[ketulsheth2]

I don't see the the DATA frame with EOS flag anywhere within the connection. Client opens up 100k streams, out of which 2-3 streams do not have EOS DATA frame, however all data has been transferred of these streams. After that, as you mentioned, client initiates a connection shutdown and server also closes the connection gracefully.

The problem is h2load reports these as HTTP request failures. If this is a valid scenario, it is probably a h2load issue.

[icing]

@tatsuhiro-t san, is there a race in h2load when having all data for a stream that it issues connection close before all EOS frames have arrived? And then report them missing?

mod-h2 will, when client GOAWAYs shut the connection probably without sending the EOS (sometimes, timing dependent). I might have to improve that but wondered if you had any insight of what the going ons in h2load at this stage are.

[tatsuhiro-t]

I think h2load has no race here because it normally closes connections in on_close_stream callback which has to be called after seeing end of stream flag.

[icing]

@ketulsheth2 could you check release v1.11.3? I think this should solve your issue.