The networking code is hard to test

[jdm]

We can write full integration tests in the form of pages that run in the WPT harness and Python scripts that control the server responses, but it's not particularly efficient. At some point we're going to implement an HTTP cache, and being able to test precisely how the networking stack treats a series of HTTP requests/responses will be important. Furthermore, the improvements for testing can lay the groundwork for an implementation of request interception for a future ServiceWorker implementation!

I propose the following:

• we create a new Response struct that contains all of the data used by Servo's networking stack from hyper's Response
• we create a trait called HTTPResponseProvider with a method that takes in all the data associated with an HTTP request and returns the new Response type
• we create an implementation called RemoteHTTPResponse that extracts all of the code from http_loader::load that initiates a network connection and waits for a response
• we figure out what else makes http_loader::load difficult to test

[jdm]

cc @samfoo does this address your concerns from #6703?

[samfoo]

Yep, it's a great start.

[samfoo]

Hey guys, I'll pick this up over the weekend.

[jdm]

Fantastic!

[samfoo]

I've done some proof-of-concept work on this over the past couple of days. I'm still debating whether the approach makes sense. The current approach:

• Abstract load into two different functions, one that is responsible for performing the request and managing errors, and one that handles IPC/inter-task comms. This, at least, makes testing less complicated.
• Inject the hyper::net::NetworkConnector dependency, and then mock that out (as necessary) in tests. For the proof-of-concept, I've simply copied all the HTTP1 code from hyper::mock. It's probably better to wrap this in something like what @jdm suggested, like a servo HttpRequester or something, but the current code is pretty tightly coupled to hyper at the moment.
• Written two tests just to show some progress 🎁.

I've also made some simplifying assumptions for the moment, that may need to change before finishing (e.g. is nossl still an option that we care about? are all unsupported scheme errors are the same? etc)

[jdm]

Exciting! I am having trouble imagining how the split into two functions was managed, but perhaps I'm over-complicating matters in my head.