BaseHTTPServer reinventing rfc822 date formatting

[schmiddy]

BPO	7370
Nosy	@devdanzin, @merwok, @bitdancer, @karlcow, @berkerpeksag
Superseder	bpo-747320: rfc2822 formatdate functionality duplication
Files	BaseHTTPServer.patch: patch of BaseHTTPServer.py and rfc822.py typofix BaseHTTPServer.patch: patch of BaseHTTPServer.py making it RFC822 compliant test_httpserver.patch: unit test for BaseHTTPServer.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-03-07.10:32:41.661>
created_at = <Date 2009-11-20.16:22:43.853>
labels = ['type-feature', 'library']
title = 'BaseHTTPServer reinventing rfc822 date formatting'
updated_at = <Date 2016-03-07.10:32:41.649>
user = 'https://bugs.python.org/schmiddy'

bugs.python.org fields:

activity = <Date 2016-03-07.10:32:41.649>
actor = 'berker.peksag'
assignee = 'none'
closed = True
closed_date = <Date 2016-03-07.10:32:41.661>
closer = 'berker.peksag'
components = ['Library (Lib)']
creation = <Date 2009-11-20.16:22:43.853>
creator = 'schmiddy'
dependencies = []
files = ['15369', '17573', '17656']
hgrepos = []
issue_num = 7370
keywords = ['patch']
message_count = 19.0
messages = ['95554', '102664', '102665', '102666', '102914', '102931', '102946', '102947', '102967', '107183', '107200', '107202', '107205', '107740', '107741', '107909', '108017', '182981', '261290']
nosy_count = 8.0
nosy_names = ['techtonik', 'ajaksu2', 'eric.araujo', 'r.david.murray', 'karlcow', 'schmiddy', 'l0nwlf', 'berker.peksag']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '747320'
type = 'enhancement'
url = 'https://bugs.python.org/issue7370'
versions = ['Python 2.7', 'Python 3.2']

[schmiddy]

While digging through Lib/BaseHTTPServer.py, I noticed that the
date_time_string() function duplicates rfc822.formatdate(). Attached is
a patch to eliminate this duplication of code.

[devdanzin]

Thanks for the patch. Per bpo-2849, use of rfc822 should be gone from the stdlib. Please re-open if you disagree.

[merwok]

Following the last link on bpo-2859, I’ve found that “rfc822.formatdate(time)” can be changed to “email.utils.formatdate(time, usegmt=True)”.

I’ll make a real diff in a few days if noone beats me to it.

Regards

[bitdancer]

The issue is not invalid. The code duplication should be removed, but using the email module as Éric suggests. Reopening.

[l0nwlf]

Instead of “email.utils.formatdate(time, usegmt=True)” we can simply use time.strftime() and clean the code in a better way. The duplication is there in date_time_string() as well as log_date_time_string(). Submitting the patch for review.

[merwok]

Nice catch. The patch looks good to me and applies correctly on my trunk copy. There seems to be no test about this in the test suite; do you have a little test script to compare old and new code?

On a sidenote, I find all this business with time.time, time.gmtime, time.localtime and time.strftime always confusing. We have datetime objects now, would it be ok to use them in this module?

Regards

[bitdancer]

There are a couple problems with this patch. The first is that fixing date_time_string by using strftime rather than email.utils.formatdate is suboptimal from a code reuse standpoint. The reason is that the date in HTTP message headers is required to conform to the same RFC standard as that generated by formatdate, so it is better to use the same routine to generate it. That way any bug fixes needed to handle RFC compliance are centralized in once place.

The second problem with the patch is that strftime generates locale-aware week and month names, but per RFC the header timestamps must use English names (see for example msg53731 in bpo-665194; the comment about locale applies to both strftime and strptime).

If bpo-665194 were implemented formatdate could use it, and then BaseHTTPServer could also use it directly. But absent that it should use email.util.formatdate. (That issue should also answer Éric's question about whether we can use DateTime here: not yet.)

Now, the logging routine is a different story. That timestamp isn't required to follow the RFC, and one could argue that it makes sense for its timestamp to use the locale. (One could also ask whether BaseHTTPServer should use the logging module, but that is a whole separate issue.)

We definitely should have a unit test before applying this patch, that makes sure the timestamp gets generated without error. Checking the detailed format of the timestamp can be assumed to be covered by the unit tests for formatdate. (I don't think those tests are completely adequate; for example they don't test that the date remains in English if the locale is different, but again that is a different issue.)

[merwok]

“One could also ask whether BaseHTTPServer should use the logging
module, but that is a whole separate issue.”
Indeed: http://www.python.org/dev/peps/pep-0337/

Cheers

[l0nwlf]

Quoting from the docstring of trunk/Lib/email/utils.py -> formatdate()

"We cannot use strftime() because that honors the locale and RFC 2822 requires that day and month names be the English abbreviations."

So yes, I do agree that email.utils.formatdate() should be used instead of time.strftime() to remove duplicate codes and be compliant with RFC.
Removing previous patch and submitting another.

[l0nwlf]

Seems that earlier patch was incorrect. Rectifying and submitting the correct patch.

[merwok]

You could get a minor speedup by doing “from email.utils import formatdate”.

Do we have tests know to check that the patch does not break anything?

Can this still go into 2.7?

[l0nwlf]

You could get a minor speedup by doing “from email.utils import formatdate”.

I guess I shall do that.
Applied the patch and tested it, it does not break anything IMO and should go in python2.7

[merwok]

Opinions are nice, tests are better! :)

[l0nwlf]

Agreed.
Attaching the unit test.

[merwok]

The skeleton is good but you have to change one thing. Your test should
exercise a function or method of BaseHTTPServer, not the underlying
implementation detail. I failed to explain that earlier:

1a) Write a test that checks that the current code produces right values
for a handful of cases (perhaps the RFC has them, else pick at random);
1b) Check that the test pass with the current code;
2a) Change BaseHTTPServer to use emails.utils.formatdate;
2b) Check that the test still pass.

End of HOWTO write a regression test :)

[techtonik]

I do not like the idea that BaseHTTPServer depends on email package, which in turn may depend on another package etc. Having date formatting function inside of email package breaks "single responsibility" principle that would be nice to have in stdlib.

[bitdancer]

The HTTP RFCs reference the email RFCs for the date format, so the email package is the logical place for this function: email is the correct responsible party. In any case, the function resides in email.utils, which has no dependencies on anything else in the email package. Of course, it does have dependencies on other parts of the python standard library, but I hardly think you'd want every module re-implementing every stdlib function.

[karlcow]

I think it is now fixed by my patch in http://bugs.python.org/issue747320

[berkerpeksag]

There is an up-to-date patch for Python 3 in bpo-747320. Closing this as a duplicate.