http tests testing more than the error code are fragile

[karlcow]

BPO	17355
Nosy	@orsenthil, @bitdancer, @karlcow
Superseder	bpo-12921: http.server.BaseHTTPRequestHandler.send_error , ability send a detailed response

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-03-05.13:35:23.514>
created_at = <Date 2013-03-05.12:21:27.691>
labels = ['tests', 'library']
title = 'http tests testing more than the error code are fragile'
updated_at = <Date 2013-03-05.13:35:23.505>
user = 'https://github.com/karlcow'

bugs.python.org fields:

activity = <Date 2013-03-05.13:35:23.505>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2013-03-05.13:35:23.514>
closer = 'r.david.murray'
components = ['Library (Lib)', 'Tests']
creation = <Date 2013-03-05.12:21:27.691>
creator = 'karlcow'
dependencies = []
files = []
hgrepos = []
issue_num = 17355
keywords = []
message_count = 2.0
messages = ['183521', '183526']
nosy_count = 3.0
nosy_names = ['orsenthil', 'r.david.murray', 'karlcow']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '12921'
type = None
url = 'https://bugs.python.org/issue17355'
versions = ['Python 3.3']

[karlcow]

Some of the tests of the HTTP Test suite are checking for the full status-line, instead of just the error code.

Why is it an issue?

1. The only mandatory part in the status-line is the error code. The phrase is optional. For example the response is made of 3 parts.

   HTTP/1.1 400 Bad Request
   ('HTTP/1.1', '400', 'Bad Request')

Only 400 is the mandatory part of the error code and is testing the error.

2. It creates dependencies on how we conceive, modify the construction of the status-line and error messages.

3. Some tests are testing messages with different "optional texts".

http://hg.python.org/cpython/file/3.3/Lib/test/test_httpservers.py#l640
Test for "HTTP/1.1 400 Line Too Long\r\n".
Instead of just testing "400". (A specific message could be put in the body, but that's informational)

http://hg.python.org/cpython/file/3.3/Lib/test/test_httpservers.py#l633
Test for "HTTP/1.1 414 Request-URI Too Long\r\n"
instead of just testing "414", btw the spec says for the optional message "URI Too Long"
http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-22#section-6.5.12

Testing the message, IMHO, should be only, for testing the construction of the message. The tests here are made for testing the right error code, but they try to do more.

Hope it helps :)

Threading: related to http://bugs.python.org/issue12921#msg183520

[bitdancer]

We are testing our http server. It is not obviously inappropriate to test the full string. Especially in the Line to Long case, where we are testing that our specific error message is produced. If you wish to argue that we should emit the standard message and only put the specific info in the body of the response, that would be a design change, and then we should probably test for *that*, since it would be a change from current behavior. Personally I find the more-specific information being in the response code message to be helpful, but I haven't done enough web programming lately for my opinion to count for much :)

So, I do not think this issue needs to be separate from bpo-12921. The tests should be fixed as part of the enhancement being advocated there (and I certainly agree it should be *possible* to control the two messages separately, regardless of what we choose to do by default).