mnot / redbot

Empty date headers aren't handled correctly currently; to investigate.

This is a well-known security issue; RED should notice this condition.

RED should be able to test and report on multiple URLs. This includes:

• taking multiple URLs in URL input
• formatting report as a table, etc.

RED should have the option to output messages in JSON, to support integration with tools, etc. May
need to be disabled on a per-deployment basis (e.g., we probably don't want this on redbot.org).

RED should support compression of HTML output, e.g., using cgi_buffer.

RED should offer some mechanism to allow webmasters to tell it 'no'. Straw-man is to support
robots.txt, but not honour *; i.e., you'd need to target RED specifically.

Right now we have a connect() timeout, but no read() timeout, so a connection could in theory stall
RED for some time, consuming resources.

When a response has cookies, RED's UI should have the option to resubmit a new request with
those cookies.

RED should allow users to specify HTTP authentication credentials (Basic, Digest, and maybe
OAuth).

RED hasn't even come near IE yet; it's been tested mostly in Safari and Mozilla.

Look at cookies and note interaction with caching, etc.

parse HTML forms and submit them in a side panel

Allow contains a list of allowed methods; where possible, allow them to be used (perhaps uploading a POST or PUT body?)

There are a lot of features / problems in the code, indicated by FIXME or TODO.

RED should support HTTPS URLs as well.

This means changes in nbhttp as well as some modification of the caching logic.

Some servers behave differently using HTTP/1.0, either because chunked encoding is not used (length delimitation), or because they think that they need to send back different headers (e.g., Cache-Control).

RED should send a 1.0 request and see if there's any appreciable difference.

http://groups.google.com/group/firebug-working-group/web/http-tracing---export-format

Initially for input; maybe output too?

Currently, RED makes a request that sets Accept-Encoding: gzip first, and then shows those response headers for the 'main' (detail) view.

This is done to optimise; if the resource doesn't support negotiation, RED doesn't need to make two requests to figure this out.

However, now that we're showing the body, which is always uncompressed, showing the headers from the compressed response is confusing and not the right thing to do.

Fixing this means either introducing some logic to find the uncompressed response and promote it to the one that's shown, or to always make two requests.

Also, some wording in the messages will need to change.

Sometimes there may be a good reason to serve data with a Content-Type that seems not to match. For example, serving HTML as text/plain, because the idea is showing the source code. But some mismatches are very clearly wrong, like serving a PNG image as image/jpeg.

RED should compare the Content-Type with file(1) output (or similar) and warn about mismatches.

A real-world instance of this problem (unless it's fixed since I wrote this) may be found in the OGG links here:
http://www.mazedude.com/aa/themusic.htm
They are served as text/html despite being audio/ogg, causing browsers to show junk text.

On 4 Jan 2010 @ 9:57pm EST a request generated this Date header:

Date: Tue, 05 Jan 2010 02:57:21 GMT

RED flagged it as an error:

The server's clock is 2 weeks 3 days ahead.

The RED code is current. It's running on OS X 10.6.

This request:

http://127.0.0.1:8080/

Results in a 302.

Curl shows the Location header as:

Location: http://127.0.0.1:8080/topics/en/CA/

RED shows the Location header as:

Location: http://127.0.0.1/topics/en/CA/

RED dropped the port number.

If a server includes a response header like:

Vary: Accept-Encoding,User-Agent

RED says:

"Sending Vary: User-Agent requires caches to store a separate copy of the response for every User-Agent request header they see.

Since there are so many different User-Agents, this can "bloat" caches with many copies of the same thing, or cause them to give up on storing these responses at all."

Section 13.6 of RFC 2616 says:

"A server SHOULD use the Vary header field to inform a cache of what request-header fields were used to select among multiple representations of a cacheable response subject to server-driven negotiation."

http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.6

Following the standard, the Apache docs (for example) say:

"The mod_deflate module sends a Vary: Accept-Encoding HTTP response header to alert proxies that a cached response should be sent only to clients that send the appropriate Accept-Encoding request header. This prevents compressed content from being sent to a client that will not understand it.

If you use some special exclusions dependent on, for example, the User-Agent header, you must manually configure an addition to the Vary header to alert proxies of the additional restrictions."

http://httpd.apache.org/docs/2.2/mod/mod_deflate.html#proxies

While I understand the nature of RED's recommendation, it would lead to User Agents that can't deal with cached content getting cached content (and vice-versa, depending on which kind of response made it into a given proxy cache first).

Of course, this is just a suggestion: it would be easier to quickly and easily try out RED if it came bundled with--or easily made use of--a Python web server. Configuring something like Apache in order to run a development tool that examines other web servers seems... well, unusual.

both input from the form, and from HTML that's parsed when descending.

This may actually be a nbhttp enhancement...