[Packetbeat] HTTP: Improve support for 100-continue

[adriansr]

Packetbeat lacks support for 100-continue request/response, which looks like:

1. Client: sends request headers including Expect: 100-continue.
2. Server: responds with 100 status code (or an error, which terminates the request).
3. Client: sends the request body.
4. Server: Answers with a full response.

Currently this is causing Packetbeat to:

1. Output an error document with "unmatched response" for the 100-continue response(2).
2. Output a correct document for the rest (1,3,4).

Example with Packetbeat monitoring port 9200 for http:

curl -H 'Expect: 100-continue' -H 'Content-Type: application/json' -XPOST 'http://localhost:9200/filebeat/_doc/mydoc' --data '{}'

Produces:

{
  "error": {
    "message": "Unmatched response"
  },
  "status": "Error",
  "type": "http",
  "http": {
    "response": {
      "status_code": 100,
      "bytes": 25,
      "headers": {
        "content-length": 0
      },
      "status_phrase": "continue"
    }
  },
  [...]
}

and

{
  "type": "http",
  "query": "POST /filebeat/_doc/mydoc",
  "status": "OK",
  "user_agent": {
    "original": "curl/7.54.0"
  },
  "method": "post",
  "http": {
    "version": "1.1",
    "request": {
      "method": "post",
      "bytes": 173,
      "body": {
        "bytes": 2
      },
      "headers": {
        "content-type": "application/json",
        "content-length": 2
      }
    },
    "response": {
      "headers": {
        "content-type": "application/json; charset=UTF-8",
        "content-length": 160
      },
      "status_phrase": "ok",
      "status_code": 200,
      "bytes": 247,
      "body": {
        "bytes": 160
      }
    }
  },
  "url": {
    "port": 9200,
    "path": "/filebeat/_doc/mydoc",
    "full": "http://localhost:9200/filebeat/_doc/mydoc",
    "scheme": "http",
    "domain": "localhost"
  }
  [...]
}

---

A simple workaround is to drop the events which contain this error:

processors:
  - drop_event.when:
     and:
     - equals.http.response.status_code: 100
     - equals.error.message: 'Unmatched response'

[OhBonsai]

Hi, I would like to do some work on this. Could you @adriansr get me up to speed quickly on this? :)

[adriansr]

Hi @OhBonsai, sure.

So I take it you know more or less how it works, but let me summarize:

Packetbeat expects a normal HTTP request/response transaction:

client --> request(headers, body)  --> server
client <-- response(headers, body) <-- server

From this Packetbeat generates a document for the request/response transaction.

but when the client request includes a Expect: 100-continue header, the transaction is a bit different:

client --> request(headers)        --> server
client <-- response(100-continue)  <-- server
client --> request(body)           --> server
client <-- response(headers, body) <-- server

From this currently Packetbeat generates two documents:

• one with the 100-continue response and an error.message: "Unmatched response"
• one with the client request and the final response from the server.

We want just the last document, while making sure that configuration options like send_headers/include_body_for/etc still work and return data from the client request and final response from the server, and not from the intermediate 100-continue response from the server.

Code for this is in packetbeat/protos/http.

Thanks for contributing!

[OhBonsai]

@adriansr PTAL

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[marc-gr]

Closed in #19349