Add pure Python implementation of datetime module to CPython

[briancurtin]

BPO	7989
Nosy	@malemburg, @tim-one, @brettcannon, @rhettinger, @amauryfa, @mdickinson, @abalkin, @pitrou, @vstinner, @giampaolo, @merwok, @bitdancer, @briancurtin, @durban
Files	PyPy-2.7.diff datetime-sandbox-pypy.diff issue7989-2.7-3.2.diff issue7989-cmp.diff issue5288-proto.diff issue7989.diff: Patch against py3k issue7989b.diff issue7989c.diff issue7989d.diff datetimetester.py issue7989e.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = <Date 2010-07-23.19:32:38.815>
created_at = <Date 2010-02-22.16:22:07.251>
labels = ['type-feature', 'library']
title = 'Add pure Python implementation of datetime module to CPython'
updated_at = <Date 2010-07-23.19:32:38.814>
user = 'https://github.com/briancurtin'

bugs.python.org fields:

activity = <Date 2010-07-23.19:32:38.814>
actor = 'belopolsky'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2010-07-23.19:32:38.815>
closer = 'belopolsky'
components = ['Library (Lib)']
creation = <Date 2010-02-22.16:22:07.251>
creator = 'brian.curtin'
dependencies = []
files = ['17698', '17701', '17718', '17842', '17845', '17848', '17978', '17992', '18130', '18148', '18155']
hgrepos = []
issue_num = 7989
keywords = ['patch']
message_count = 91.0
messages = ['99774', '99801', '99804', '107171', '107258', '107272', '107275', '107292', '107293', '107295', '107298', '107302', '107303', '107308', '107309', '107312', '107317', '107333', '107451', '107453', '107454', '108019', '108020', '108021', '108023', '108024', '108025', '108026', '108027', '108028', '108029', '108030', '108035', '108037', '108047', '108048', '108051', '108053', '108055', '108056', '108059', '108089', '108090', '108122', '108124', '108126', '108129', '108158', '108179', '108553', '108624', '108643', '108645', '108648', '108649', '108666', '108682', '108687', '108697', '108743', '108747', '109058', '109061', '109063', '109067', '109069', '109128', '109129', '109131', '109134', '109135', '109136', '109140', '109142', '109144', '109146', '109147', '109148', '109149', '109152', '109286', '109662', '110155', '110248', '111097', '111131', '111210', '111211', '111338', '111356', '111375']
nosy_count = 16.0
nosy_names = ['lemburg', 'tim.peters', 'brett.cannon', 'rhettinger', 'amaury.forgeotdarc', 'mark.dickinson', 'davidfraser', 'belopolsky', 'pitrou', 'vstinner', 'techtonik', 'giampaolo.rodola', 'eric.araujo', 'r.david.murray', 'brian.curtin', 'daniel.urban']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue7989'
versions = ['Python 3.2']

[briancurtin]

After discussion on numerous issues, python-dev, and here at the PyCon sprints, it seems to be a good idea to move timemodule.c to _timemodule.c and convert as much as possible into pure Python. The same change seems good for datetime.c as well.

[bitdancer]

By 'convert', I believe you mean 'create a python implementation of'. That is, there's no reason to drop any c code, just to create parallel python versions when possible.

Also note that one of the alternate implementations (IronPython I think) has an implementation of DateTime in Python that they plan to contribute.

[briancurtin]

Correct, your wording is better than mine.

I'll ask around and see where that datetime module may be and what it's state is.

[abalkin]

As far as I remember, the datetime module started as a pure python module and was reimplemented in C around year 2003 or so. One of the important additions at that time was the C API to datetime functionality. I am afraid that with the _timemodule.c/timemodule.py split there will be more an more functionality that is awkward to access from C API.

[malemburg]

Alexander Belopolsky wrote:

Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:

As far as I remember, the datetime module started as a pure python module and was reimplemented in C around year 2003 or so. One of the important additions at that time was the C API to datetime functionality. I am afraid that with the _timemodule.c/timemodule.py split there will be more an more functionality that is awkward to access from C API.

That's correct, though the main reason for rewriting the module in
C was to gain performance - this is essential for basic types like
date/time types.

-1 on undoing the C rewrite.

It would be much better to spell out the problems you mention and
provide patches to implement solutions for them.

[abalkin]

For the datetime module there are also a few more subtle issues that would make it difficult to make a hybrid C/Python implementation. The problem is that unlike the time module where most of the functionality is implemented in module level functions, datetime functionality is entirely in class methods.

One may suggest to use inheritance and derive datetime classes from those implemented in _datetime.c, but this will not work because for example datetime + timedelta will return a _datetime class unless __add__ is overridden in Python. There is another even less obvious issue with inheriting from datetime classes: see issue bpo-5516.

Therefore, it looks like there are only two choices:

1. Replicate all functionality in both _datetime.c and datetime.py and thus double the cost of implementing new features in CPython. (OK, maybe not double because Python development is easier than C, but increase instead of decrease.)

2. Incur a performance penalty in every method because every C call wil have to be wrapped in Python.

Another -1 from me.

[briancurtin]

It seems like this might not be worth it or a good idea, and I have no strong feeling for this being done. Feel free to close/reject this one.

[abalkin]

Brett Cannon wrote in a comment (msg106498) on another issue:

"""
The stated long-term goal of the stdlib is to minimize the C extension modules to only those that have to be written in C (modules can still have performance enhancing extension back-ends). Since datetime does not meet that requirement it's not a matter of "if" but "when" datetime will get a pure Python version and use the extension code only for performance.
"""

I'll keep this open for a while to give Brett and others a chance to respond to opposition.

[abalkin]

Also, my opposition is only to splitting datetime. While I am not against splitting the time module, I believe it should be phased out eventually and posix compatibility portion folded into posix module.

[brettcannon]

So I see a couple of objections here to the idea that I will try to address.

First is MAL's thinking that this will undo any C code, which it won't. The idea is that stdlib modules that do not inherently rely on other C code (e.g. sqlite3 does not fall underneath this) would have a pure Python implementation with possible C enhancements. In the case of datetime that code is done, so it won't go anywhere. In this case it would be bringing in a pure Python implementation like the one PyPy maintains. You can look at heapq if you want an existing example of what it looks like to maintain a pure Python and C version of a module.

Alexander is worried about performance because of Python shims and duplication of work. For the performance issue, there is none. If you look at something like heapq you will see that there is an import * in there to pull in all extension code; there is no Python shim to pass through if you don't engineer the extension that way. So in datetime's case all of the extension code would just get pulled into datetime.py directly and not have any indirection cost.

As for duplication of work, we already have that with datetime in the other Python VMs. IronPython, Jython, and PyPy have to do their own version of datetime because the stdlib doesn't provide a pure Python version for them to use. So while CPython might not be duplicating work, other people are. Besides, people typically prototype in Python anyway (including you, Alexander, for the UTC patch) and then write the C code, so there really isn't any wasted development cycles by having the Python and C version already.

The key thing to remember is that this is so the VMs other than CPython are not treated as second-class. They are legit implementations just like CPython, but because we have this legacy stuff written in C in the stdlib they are put at a disadvantage. It would be better to pool resources and make sure everyone gets to use an equivalent standard library when possible.

And I should also mention I expect the PyPy folks to be the ones to move their code over; I would never expect anyone to re-implement datetime from scratch.