Fix waiting for response body on HTTP HEAD requests with Content-Length set

[tisba]

We recently noticed, that tsung will wait on HTTP HEAD requests when Content-Length header is set to non-zero value.

According to RFC2616 section 14.13:

The Content-Length entity-header field indicates the size of the entity-body, in decimal number of OCTETs, sent to the recipient or, in the case of the HEAD method, the size of the entity-body that would have been sent had the request been a GET.

So the Content-Length header - if present - will contain the expected response size for a non-HEAD request. Actually it is recommended that Content-Length should be present (also RFC2616 section 14.13):

Applications SHOULD use this field to indicate the transfer-length of the message-body, unless this is prohibited by the rules in section 4.4.

So it is considered "normal" that a HTTP HEAD requests contains a Content-Length header.

This is the same with HTTP 1.0, according to RFC1945 Sec 10.4.

Issue

The problem in tsung was with ts_http_common:check_resp_size/4. If headers are received completely and the content length is non-zero, we always expect more data. This resulted in tsung waiting indefinitely (until server closes connection).

Solution

In case we have completely read the HTTP headers and in case of the HTTP verb is HEAD, we are done reading the response, regardless what Content-Length states. In this case we can set ack_done in #state_rcv to true.

The problem can be reproduced using

<?xml version="1.0"?>
<!DOCTYPE tsung SYSTEM "/tsung/install/share/tsung/tsung-1.0.dtd" []>
<tsung loglevel="debug" dumptraffic="true">
  <clients>
    <client host="localhost" use_controller_vm="true"/>
  </clients>
  <servers>
    <server host="testapp.loadtest.party" port="80" type="tcp"/>
  </servers>
  <load duration="300" unit="second">
    <arrivalphase phase="0" duration="300" unit="second">
      <users maxnumber="1" arrivalrate="1" unit="second" />
    </arrivalphase>
  </load>
  <sessions>
    <session name="http_head" weight="1" type="ts_http">
      <request subst="true">
        <http url="/status/200" method="HEAD" version="1.1">
        </http>
      </request>
    </session>
  </sessions>
</tsung>

Note that this host will respond with Content-Length:

$ curl --head testapp.loadtest.party
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 112
Connection: keep-alive
Status: 200 OK
Date: Sun, 16 Sep 2018 18:08:59 GMT
X-Powered-By: Phusion Passenger 5.3.4
Server: nginx/1.14.0 + Phusion Passenger 5.3.4

[tisba]

Hey @nniclausse. Could you take a quick look at this?

It took me a while to figure out what was going on here but I think the solution now is okay. But since this looks like it was so fundamentally broken, I'm a bit irritated and want to make sure I'm not overlooking something obvious here :)

[tisba]

Sorry for pining again, but could you maybe take a quick look, @nniclausse?

I've tested this quite a bit, but as I already said, this is just so strange because this was broken basically forever...

[nniclausse]

Sorry for the delay
Since this bug is here like forever, it means that almost nobody use HEAD requests ... :-/

[tisba]

We had actually some tests that used HEAD requests, but the responses were dynamic. The application did not include a Content-Length header which is totally okay from the RFC perspective. But recent tests hitting static resources uncovered the issue, as most web servers DO include Content-Length in this case.

Anyway, thanks for taking a look! 👍