mnot / redbot

Source
Commits
Network (4)
Issues (23)
Graphs
Branch: master

RED checks HTTP resources to see how they use HTTP, makes suggestions, and finds common protocol mistakes. — Read more

Sort by: Priority | Votes | Last Updated

Loading…

28.

0 votes

Unable to submit request when using custom headers
2 comments Created about 1 month ago by solarissmoke

Title

Body
Hi, This issue is with the public instance of RED. I'm finding that if I try to use a custom header (either a completely new header or a customised string for one of the existing header options) with the "add headers" feature, then hitting return doesn't submit the form (and there doesn't appear to be any other way to do it). It works fine with the pre-defined headers. This is on Firefox 3.5.4 on WinXP. Incidentally, I tried using IE8 to see if it works there, and the entire "add headers" feature is broken in IE! The second input field (where you choose the specific header) never appears.
or cancel

Hi,

This issue is with the public instance of RED.

I'm finding that if I try to use a custom header (either a completely new header or a customised string for one of the existing header options) with the "add headers" feature, then hitting return doesn't submit the form (and there doesn't appear to be any other way to do it). It works fine with the pre-defined headers. This is on Firefox 3.5.4 on WinXP.

Incidentally, I tried using IE8 to see if it works there, and the entire "add headers" feature is broken in IE! The second input field (where you choose the specific header) never appears.

Comments

mnot Sun Dec 06 22:55:09 -0800 2009 | link

Both should be fixed. Some browsers won't submit a form upon 'enter' if there's more than one field on the screen; I've fixed this by listening for a keypress (although I may add an actual 'submit' button in the future, once I figure out how it should look).

The IE8 issue was because of lack of support for 'change' in jQuery live events; while it worked on other browsers, it didn't work for IE (as advertised in the jQuery docs).

Pleae reopen if you find any more problems, and thanks for reporting the issue!

Comment
Both should be fixed. Some browsers won't submit a form upon 'enter' if there's more than one field on the screen; I've fixed this by listening for a keypress (although I may add an actual 'submit' button in the future, once I figure out how it should look). The IE8 issue was because of lack of support for 'change' in jQuery live events; while it worked on other browsers, it didn't work for IE (as advertised in the jQuery docs). Pleae reopen if you find any more problems, and thanks for reporting the issue!
or cancel

solarissmoke Tue Dec 08 03:22:19 -0800 2009 | link

Works fine now, thanks!

Comment
Works fine now, thanks!
or cancel

Parsed with GitHub Flavored Markdown
25.

0 votes

new tooltip style pop scrolbar; difficult to view
2 comments Created 2 months ago by cburroughs

Title

Body
The new tooltip messages will pop the scrollbar if they are long, but their is no reasonable way to scroll down without moving the mouse... which causes the tooltip to go away. http://img73.imageshack.us/img73/8169/tooltipscroll.png
or cancel

The new tooltip messages will pop the scrollbar if they are long, but their is no reasonable way to scroll down without moving the mouse... which causes the tooltip to go away.

http://img73.imageshack.us/img73/8169/tooltipscroll.png

Comments

mnot Thu Oct 15 14:41:45 -0700 2009 | link

Ack; will fix asap. Thanks.

Comment
Ack; will fix asap. Thanks.
or cancel

mnot Thu Oct 15 20:59:16 -0700 2009 | link

make sure popups at the bottom of the page aren't obscured; closed by f6da1fc

Comment
make sure popups at the bottom of the page aren't obscured; closed by f6da1fc56846bf478d8a598912dd6dce8e42caed
or cancel

Parsed with GitHub Flavored Markdown
24.

1 vote

Nested comments in headers are valid
2 comments Created 2 months ago by daybarr

Title

Body
Comments in headers can be nested according to [the RFC2616 definition](http://www.apps.ietf.org/rfc/rfc2616.html#page-17). RED does not recognise such nested comments and can therefore give spurious warnings for headers that are in fact valid. For example, [the RED results](http://redbot.org/?uri=http%3A%2F%2F%77%77%77.velocix.com) for www.velocix.com show a warning *"The Via header's syntax isn't valid"* for the following header which is actually valid: Via: 1.1 velocix.com:80 (pcd/26.1.10.31395 (2009-09-11 08:50:08 UTC)) RED's inability to cope with nested comments appears to be a known issue (without an actual issue to record it until now). See the comment in the [RED source](http://github.com/mnot/redbot/blob/master/src/response_analyse.py#L74). Probably not fixed yet due to the difficulty of coping with parenthesis nesting in a regex.
or cancel
Comments in headers can be nested according to the RFC2616 definition. RED does not recognise such nested comments and can therefore give spurious warnings for headers that are in fact valid.

For example, the RED results for www.velocix.com show a warning "The Via header's syntax isn't valid" for the following header which is actually valid:
```
Via: 1.1 velocix.com:80 (pcd/26.1.10.31395 (2009-09-11 08:50:08 UTC))
```
RED's inability to cope with nested comments appears to be a known issue (without an actual issue to record it until now). See the comment in the RED source. Probably not fixed yet due to the difficulty of coping with parenthesis nesting in a regex.
Comments

mnot Thu Oct 15 18:09:07 -0700 2009 | link

Yes. The fix is probably to come up with a regex that allows a limited level of recursion in comments...

Comment
Yes. The fix is probably to come up with a regex that allows a limited level of recursion in comments...
or cancel

mnot Thu Oct 15 21:17:21 -0700 2009 | link

allow two levels of nested comments; closed by dbe24ae (for now)

Comment
allow two levels of nested comments; closed by dbe24ae70f5932119ed5ed244a295ed9d9455d12 (for now)
or cancel

Parsed with GitHub Flavored Markdown
23.

0 votes

TypeError Exception thrown on timeout
1 comment Created 2 months ago by cburroughs

Title

Body
<pre> Traceback (most recent call last): File "/tmp/cool/cool-stuff/red/webui.py", line 652, in RedWebUi(test_uri, base_uri, output, descend) File "/tmp/cool/cool-stuff/red/webui.py", line 106, in __init__ body_procs=[link_parser.feed], File "/tmp/cool/cool-stuff/red/red.py", line 74, in __init__ status_cb, body_procs, req_type=method) File "/tmp/cool/cool-stuff/red/red_fetcher.py", line 117, in __init__ self._makeRequest() File "/tmp/cool/cool-stuff/red/red_fetcher.py", line 147, in _makeRequest nbhttp.run() File "/tmp/cool/cool-stuff/nbhttp/push_tcp.py", line 476, in run what() File "/tmp/cool/cool-stuff/nbhttp/push_tcp.py", line 495, in cb callback(*args) File "/tmp/cool/cool-stuff/red/webui.py", line 203, in timeoutError self.output("***", conn.uri.encode('utf-8', 'replace')) TypeError: output() takes exactly 1 argument (2 given) </pre>
or cancel
```
Traceback (most recent call last):
  File "/tmp/cool/cool-stuff/red/webui.py", line 652, in 
    RedWebUi(test_uri, base_uri, output, descend)
  File "/tmp/cool/cool-stuff/red/webui.py", line 106, in __init__
    body_procs=[link_parser.feed],
  File "/tmp/cool/cool-stuff/red/red.py", line 74, in __init__
    status_cb, body_procs, req_type=method)
  File "/tmp/cool/cool-stuff/red/red_fetcher.py", line 117, in __init__
    self._makeRequest()
  File "/tmp/cool/cool-stuff/red/red_fetcher.py", line 147, in _makeRequest
    nbhttp.run()
  File "/tmp/cool/cool-stuff/nbhttp/push_tcp.py", line 476, in run
    what()
  File "/tmp/cool/cool-stuff/nbhttp/push_tcp.py", line 495, in cb
    callback(*args)
  File "/tmp/cool/cool-stuff/red/webui.py", line 203, in timeoutError
    self.output("***", conn.uri.encode('utf-8', 'replace'))
TypeError: output() takes exactly 1 argument (2 given)
```
Comments

mnot Thu Oct 15 00:24:52 -0700 2009 | link

Should be fixed in
http://github.com/mnot/redbot/commit/9bf99d4ebc7f2d4fc562d4294f7f0b021b5aeb5b

thanks!

Comment
Should be fixed in http://github.com/mnot/redbot/commit/9bf99d4ebc7f2d4fc562d4294f7f0b021b5aeb5b thanks!
or cancel

Parsed with GitHub Flavored Markdown
22.

1 vote

Potentially incorrect detection of HTTP compression if server-side UA filtering is used
5 comments Created 4 months ago by MortyEU

Title

Body
It is debatable whether this is a bug and whether it should be fixed or not... While playing around with redbot.org I noticed that it listed jQuery hosted on Google's CDN as not using HTTP compression. This made me wonder, as I'm pretty sure this CDN uses compression. Upon examination with http://www.rexswain.com/httpview.html it seems that Googles CDN uses the "User-Agent" request string. If Google recognizes the browser as a 'whitelisted' browser, say Firefox or MSIE 7, then they'll return a compressed response. If I enter some garbage in the User-Agent string, they'll return un-compressed content. Now, arguably what redbot does now (sending its own user-agent string / a framework default string) is perfectly standards compliant. At the same time, the whitelisting of known-to-work browsers is very common, as there have been many browser bugs around HTTP compression (Netscape 4, MSIE 6, many mobile phone browsers). I noticed this around HTTP compression, but of course the issue is more general, as User-Agent matching could be done to several kinds of HTTP responses / payloads. Perhaps redbot should masquerade as a Firefox 3 or a similar modern browser? Or have it as an option?
or cancel

It is debatable whether this is a bug and whether it should be fixed or not...

While playing around with redbot.org I noticed that it listed jQuery hosted on Google's CDN as not using HTTP compression. This made me wonder, as I'm pretty sure this CDN uses compression.

Upon examination with http://www.rexswain.com/httpview.html it seems that Googles CDN uses the "User-Agent" request string. If Google recognizes the browser as a 'whitelisted' browser, say Firefox or MSIE 7, then they'll return a compressed response. If I enter some garbage in the User-Agent string, they'll return un-compressed content.

Now, arguably what redbot does now (sending its own user-agent string / a framework default string) is perfectly standards compliant. At the same time, the whitelisting of known-to-work browsers is very common, as there have been many browser bugs around HTTP compression (Netscape 4, MSIE 6, many mobile phone browsers).

I noticed this around HTTP compression, but of course the issue is more general, as User-Agent matching could be done to several kinds of HTTP responses / payloads.

Perhaps redbot should masquerade as a Firefox 3 or a similar modern browser? Or have it as an option?

Comments

MortyEU Thu Aug 13 19:42:08 -0700 2009 | link

I should clarify: If I send a request with a valid "accept-encoding: gzip" header AND a nonsensical "User-Agent" header, then Google does not compress the response, despite the presence of a valid "accept-encoding" header.

Comment
I should clarify: If I send a request with a valid "accept-encoding: gzip" header AND a nonsensical "User-Agent" header, then Google does not compress the response, despite the presence of a valid "accept-encoding" header.
or cancel

mnot Wed Aug 19 04:39:11 -0700 2009 | link

I think it's worthwhile considering this as an option, but I'd note that the practice isn't good; to make it kosher you have to Vary: Accept-Encoding, User-Agent, and that's going to pollute caches quite a bit.

Additionally, it cheats UAs that are well-implemented but not on the whitelist out of compressed responses.

All of that said, one of the planned directions for RED is to allow more control over the request; e.g., allowing different methods, custom headers, request bodies. This would definitely fall into that.

Comment
I think it's worthwhile considering this as an option, but I'd note that the practice isn't good; to make it kosher you have to Vary: Accept-Encoding, User-Agent, and that's going to pollute caches quite a bit. Additionally, it cheats UAs that are well-implemented but not on the whitelist out of compressed responses. All of that said, one of the planned directions for RED is to allow more control over the request; e.g., allowing different methods, custom headers, request bodies. This would definitely fall into that.
or cancel

MortyEU Wed Aug 19 07:09:29 -0700 2009 | link

I completely agree with you on the 'badness' of this practice. But as noted, some people are doing this in production on larger sites today.

Thank you for the consideration, and for RED! :-)

Comment
I completely agree with you on the 'badness' of this practice. But as noted, some people are doing this in production on larger sites today. Thank you for the consideration, and for RED! :-)
or cancel

mnot Thu Oct 15 00:24:07 -0700 2009 | link

UA (and other request headers) can now be set.

Comment
UA (and other request headers) can now be set.
or cancel

nicolas17 Wed Dec 09 14:22:54 -0800 2009 | link

They should be using a user-agent blacklist instead of a whitelist. Unknown agents should be assumed to be standards-compliant.

Comment
They should be using a user-agent blacklist instead of a whitelist. Unknown agents should be assumed to be standards-compliant.
or cancel

Parsed with GitHub Flavored Markdown
16.

0 votes

Incorrect Date + Age detection
2 comments Created 7 months ago by rtomayko

Title

Body
I'm serving an incorrect Date header at tomayko.com when the Age is > 0 (refresh the following until you get a Age value > 5 or so to see RED detect this): http://redbot.org/?uri=http%3A%2F%2Ftomayko.com%2F RED detects the skew but displays the message "The server's clock is none." instead of "The server's clock is ahead.". The issue is caused by Nginx overwriting the Date header set by downstream servers. In this case, Rack::Cache is setting the Date header to the time when the response was last validated with the origin and the Age header to the number of seconds since last validation (correct per my reading of RFC 2616) but the Date value is then being overwritten as it passes through Nginx. I'm able to get the message to display "The server's clock is ahead." by commenting out lines 988/999 here: http://github.com/mnot/redbot/blob/8b861f08e327019c45ee10f38edd45159ed779a3/src/red.py#L988 I assume this will cause issues elsewhere, however.
or cancel

I'm serving an incorrect Date header at tomayko.com when the Age is > 0 (refresh the following until you get a Age value > 5 or so to see RED detect this):

http://redbot.org/?uri=http%3A%2F%2Ftomayko.com%2F

RED detects the skew but displays the message "The server's clock is none." instead of "The server's clock is ahead.".

The issue is caused by Nginx overwriting the Date header set by downstream servers. In this case, Rack::Cache is setting the Date header to the time when the response was last validated with the origin and the Age header to the number of seconds since last validation (correct per my reading of RFC 2616) but the Date value is then being overwritten as it passes through Nginx.

I'm able to get the message to display "The server's clock is ahead." by commenting out lines 988/999 here:

http://github.com/mnot/redbot/blob/8b861f08e327019c45ee10f38edd45159ed779a3/src/red.py#L988

I assume this will cause issues elsewhere, however.

Comments

rtomayko Mon May 25 04:44:59 -0700 2009 | link

Okay, I think I may have found it (ageskew branch):

http://github.com/rtomayko/redbot/commit/c957caaaa782ae69dfcf5c4b118ef6b3b9c1242a

We check for skew using the response timestamp plus the value of the age header but report it based on the date header alone.

Also, the more I look at this, the more I wonder if my reading of RFC 2616 is wrong. You're using the locally generated response timestamp plus the Age header to calculate the current server time instead of using the Date header plus the Age header. My understanding is that the Date is when the message was generated (or last validated) at the origin. The Date + Age should be the best approximation of the nearest server/intermediary's current time.

Comment
Okay, I think I *may* have found it (ageskew branch): http://github.com/rtomayko/redbot/commit/c957caaaa782ae69dfcf5c4b118ef6b3b9c1242a We check for skew using the response timestamp plus the value of the age header but report it based on the date header alone. Also, the more I look at this, the more I wonder if my reading of RFC 2616 is wrong. You're using the locally generated response timestamp plus the Age header to calculate the current server time instead of using the Date header plus the Age header. My understanding is that the Date is when the message was generated (or last validated) *at the origin*. The Date + Age should be the best approximation of the nearest server/intermediary's current time.
or cancel

mnot Sun Jun 14 04:10:01 -0700 2009 | link

Hey -- sorry I didn't see this earlier (have to see if I can convince github to send me e-mails...). Incorporated in a slightly different way -- Thanks!

WRT age calculation - please see the current code (red.py line 237) -- is this still an issue?

Comment
Hey -- sorry I didn't see this earlier (have to see if I can convince github to send me e-mails...). Incorporated in a slightly different way -- Thanks! WRT age calculation - please see the current code (red.py line 237) -- is this still an issue?
or cancel

Parsed with GitHub Flavored Markdown
14.

0 votes

enhancement
x

Catch bad chunked encoding
1 comment Created 8 months ago by mnot

Title

Body
Redbot needs to catch bad chunking and display alerts. The nginx wiki appears to be one example of this presently.
or cancel

Redbot needs to catch bad chunking and display alerts. The nginx wiki appears to be one example
of this presently.

Comments

mnot Tue Jun 23 22:41:13 -0700 2009 | link

done

Comment
done
or cancel

Parsed with GitHub Flavored Markdown
12.

0 votes

enhancement
x

Pretty icons
3 comments Created 8 months ago by mnot

Title

Body
RED would like some pretty icons for bullet items (GOOD, INFO, BAD). Maybe a favico too.
or cancel

RED would like some pretty icons for bullet items (GOOD, INFO, BAD). Maybe a favico too.

Comments

mnot Tue Jun 16 16:28:56 -0700 2009 | link

bullet icons done; favico and logo to come.

Comment
bullet icons done; favico and logo to come.
or cancel

mnot Tue Jun 23 21:34:28 -0700 2009 | link

logo on project page

Comment
logo on project page
or cancel

mnot Sun Jun 28 06:13:53 -0700 2009 | link

favico up.

Comment
favico up.
or cancel

Parsed with GitHub Flavored Markdown
5.

0 votes

enhancement
x

Checking referenced assets
1 comment Created 8 months ago by mnot

Title

Body
Cacheability engine had the ability to check page assets; e.g., IMGs, CSS, etc. RED should have an option to do this as well. Requires matrix/table output (see separate issue).
or cancel

Cacheability engine had the ability to check page assets; e.g., IMGs, CSS, etc.

RED should have an option to do this as well. Requires matrix/table output (see separate issue).

Comments

mnot Sun Jul 05 20:38:10 -0700 2009 | link

done.

Comment
done.
or cancel

Parsed with GitHub Flavored Markdown
4.

0 votes

enhancement
x

List links on page
1 comment Created 8 months ago by mnot

Title

Body
One of the body options should show the links in the page, for formats we understand. Clicking on a link will check it.
or cancel

One of the body options should show the links in the page, for formats we understand. Clicking on
a link will check it.

Comments

mnot Thu May 07 01:40:43 -0700 2009 | link

now on master.

Comment
now on master.
or cancel

Parsed with GitHub Flavored Markdown