Service clients should timeout HTTP requests if no response is received

[tsandall]

Steps to reproduce

This problem can reproduced simply with nc. For example:

$ nc -l localhost 8000
GET /foo HTTP/1.1

In a separate terminal start opa and point it at the server:

$ opa run -s --set services.test.url=http://localhost:8000 --set bundles.test.resource=foo
{"addrs":[":8181"],"insecure_addr":"","level":"info","msg":"Initializing server.","time":"2020-01-17T17:17:10-05:00"}
{"level":"info","msg":"Starting bundle downloader.","name":"test","plugin":"bundle","time":"2020-01-17T17:17:10-05:00"}

netcat will print the request but not send any response. OPA will continue waiting indefinitely because there is no timeout set on the HTTP request.

Solution

This page does a good job of explaining how the various client timeouts can be set. OPA should be updated to set the Client.Do timeout (which I think means cancelling the context). Note, we do not want to set default timeouts on reading the full response since that'll cause problems for users with large bundles. It should be safe to assume that OPA starts receiving HTTP headers on the order of seconds (e.g., 10 seconds is probably a safe default.)

Initially, users should be able to configure the timeout on the service object. For example:

services:
   test:
     url: https://example.com/v1
     response_header_timeout_seconds: 10