Intermittent WPT web server failures

[jdm]

From http://build.servo.org/builders/mac/builds/168/steps/shell_3/logs/stdio:

 5:02.32 TEST_START: Thread-TestrunnerManager-4 /html/semantics/embedded-content/the-canvas-element/initial.reset.path.html
 5:02.36 PROCESS_OUTPUT: Thread-TestrunnerManager-4 "ERROR:glfw: GLFW Error: Failed to retrieve display name" (pid:70080 command:/Users/servo/buildbot/slave/mac/build/target/servo --cpu --hard-fail http://localhost:8000/html/semantics/embedded-content/the-canvas-element/initial.reset.path.html)
 5:02.39 PROCESS_OUTPUT: Thread-TestrunnerManager-4 "ERROR:net::resource_task: error receiving bytes: Server returned malformed HTTP response" (pid:70080 command:/Users/servo/buildbot/slave/mac/build/target/servo --cpu --hard-fail http://localhost:8000/html/semantics/embedded-content/the-canvas-element/initial.reset.path.html)
 5:02.39 PROCESS_OUTPUT: Thread-TestrunnerManager-4 "ERROR:script::html::hubbub_html_parser: error loading script http://localhost:8000/resources/testharness.js" (pid:70080 command:/Users/servo/buildbot/slave/mac/build/target/servo --cpu --hard-fail http://localhost:8000/html/semantics/embedded-content/the-canvas-element/initial.reset.path.html)
 5:02.41 PROCESS_OUTPUT: Thread-TestrunnerManager-4 "ERROR:js::rust: Error at http://localhost:8000/resources/testharnessreport.js:5: ReferenceError: add_completion_callback is not defined" (pid:70080 command:/Users/servo/buildbot/slave/mac/build/target/servo --cpu --hard-fail http://localhost:8000/html/semantics/embedded-content/the-canvas-element/initial.reset.path.html)
 5:02.46 PROCESS_OUTPUT: Thread-TestrunnerManager-4 "evaluate_script failed" (pid:70080 command:/Users/servo/buildbot/slave/mac/build/target/servo --cpu --hard-fail http://localhost:8000/html/semantics/embedded-content/the-canvas-element/initial.reset.path.html)
 5:02.46 PROCESS_OUTPUT: Thread-TestrunnerManager-4 "ERROR:js::rust: Error at http://localhost:8000/html/semantics/embedded-content/the-canvas-element/initial.reset.path.html:2: ReferenceError: async_test is not defined" (pid:70080 command:/Users/servo/buildbot/slave/mac/build/target/servo --cpu --hard-fail http://localhost:8000/html/semantics/embedded-content/the-canvas-element/initial.reset.path.html)
 5:02.46 PROCESS_OUTPUT: Thread-TestrunnerManager-4 "evaluate_script failed" (pid:70080 command:/Users/servo/buildbot/slave/mac/build/target/servo --cpu --hard-fail http://localhost:8000/html/semantics/embedded-content/the-canvas-element/initial.reset.path.html)
 5:17.34 TEST_END: Thread-TestrunnerManager-4 TIMEOUT, expected OK

@jgraham do you know what to make of this?

[jdm]

I propose modifying https://github.com/servo/rust-http/blob/servo/src/http/client/response.rs#L28 to take an additional error message and passing in descriptive strings in each of the call sites to help narrow down what rust-http is choking on.

[jdm]

At least one run with servo/rust-http#34 is reporting an invalid http version number. I'm guessing it's getting null or EOF or something.

[jdm]

@jgraham: http://build.servo.org/builders/mac/builds/350/steps/shell_4/logs/stdio shows a test run on Mac with --processes=2 that still hit the web server problem. /XMLHttpRequest/status-async.htm timed out with a bunch of the invalid HTTP version errors on display, while /XMLHttpRequest/status-error.htm failed to load the test page due to a broken pipe.

[jdm]

Next debugging strategy I'll try: run serve.py locally and write some test programs that connect over and over and yield more information if it doesn't work right.

[jdm]

diff --git a/wptserve/server.py b/wptserve/server.py
index d5bb7b3..1e9d2f7 100644
--- a/wptserve/server.py
+++ b/wptserve/server.py
@@ -106,6 +106,7 @@ class RequestRewriter(object):
 class WebTestServer(ThreadingMixIn, BaseHTTPServer.HTTPServer):
     allow_reuse_address = True
     acceptable_errors = (errno.EPIPE, errno.ECONNABORTED)
+    request_queue_size = 2000

     # Ensure that we don't hang on shutdown waiting for requests
     daemon_threads = True

Tada. @jgraham?