net/http: change Error to generate an HTML page

[rsc]

There is constant background noise about this web client or that web client mistakenly treating http.Error's responses as HTML and therefore being subject to scripting attacks. This is awful, and depressing, and generally disgusting.

One way to eliminate the noise would be to change Error from sending back (approximately)

Content-Type: text/plain

<ERROR HERE>

to

Content-Type: text/html

<pre>
&lt;ERROR HERE&gt;

That is, if everyone is going to interpret the result as HTML, okay fine, let's send (and correctly Content-Type) an actual HTML response with proper escaping of the message.

Anyone see any reasons not to do this? The only one I can think of is that it makes clients of API services that send back http.Error errors have to deal with the HTML, but as a writer of API service clients myself, most of the errors I see come back in HTML anyway, because they're generated by some box in front of the API service.

[bradfitz]

Yeah, sure. This is the least offensive option.

[dsymonds]

It does make a mess for people trying to hit HTTP servers with wget/curl and then would not be able to read non-trivial error messages. It also makes a mess if an HTTP client receives such an error, and escapes it again for embedding in its own error message (e.g. frontend reporting a backend error).

Do we have a list of user agents that don't respect the existing Content-Type+X-Content-Type-Options headers? Are we deciding to support them at the expense of other use cases? Will they still behave reasonably with the proposed change?

[bradfitz]

Yes, I would like to wait on this change until we can get a public statement with real data here about which user agents mess this up, and publish that information on this bug.

I don't want to just perpetuate the idea that all browsers always suck. Certainly it's not all of them.

[adg]

I'm concerned about breaking clients of Go services that use the current implementation. They expect error messages in plain text; swapping it out for HTML could cause them to do the wrong thing.

I agree that I'd like to see some hard data on affected browsers before taking further steps.

[slrz]

I concur with Andrew. This certainly has the potential to break some of our clients. In case this gets done, the change should be made very well known, so servers can remove the calls to http.Error if deemed necessary.

[ianlancetaylor]

I don't have any hard data about affected browsers, and I don't have anything public I can point to. That said, I have been told that the Adobe Reader browser plugin, among others, is vulnerable to attacks when the user can control parts of the error message (which happens when the Go error string quotes part of the user's input). This is true even with the Content-Type and X-Content-Type-Options headers, as the browser plugin apparently ignores those headers.

[ailg]

I talked with some colleagues to get you more data:

1. text/plain triggers content sniffing in IE. Microsoft recommendation (https://msdn.microsoft.com/en-us/library/ms775147(v=vs.85).aspx) is to use something more specific (e.g. application/go-error-message). Other browsers are OK (they will respect text/plain and not interpret it as HTML).
2. As Ian just said the main problem is plugins, on all browsers, as they ignore headers (Content-Type, X-Content-Type-Options), read the content, sniff, execute then can execute javascript (XSS).
   • Adobe Reader (PDF) will execute a PDF if it's found anywhere in the page
   • Adobe Flash (SWF) is a bit more restrictive, it'll only execute if the flash file is at the beginning. Often errors are prefixed with a program string (e.g. error: %v) in which case this attack would not work, but that's not a guarantee.

I hope this helps so you can make a decision, let me know if you need more info.

[bradfitz]

I think only old MSIE is a problem. MSIE 8+ gets this right, I believe: https://blogs.msdn.microsoft.com/ie/2008/07/02/ie8-security-part-v-comprehensive-protection/

If the problem is only Adobe, perhaps we can vary our behavior based on the User-Agent and only HTML escape for Adobe and ancient MSIE.

[rogpeppe]

If the problem is only Adobe, perhaps we can vary our behavior based
on the User-Agent and only HTML escape for Adobe and ancient MSIE.

Given that Error only takes a ResponseWriter, is it actually possible for it
to determine the user agent without undue dirtiness?