str(datetime_obj) doesn't include microseconds if their value is 0

[ezio-melotti]

BPO	7342
Nosy	@tim-one, @doerwalter, @birkenfeld, @pitrou, @ezio-melotti, @bitdancer
Files	issue7342.patch: Fixed the test to use a datetime obj with a specific number of microseconds.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ezio-melotti'
closed_at = <Date 2009-12-18.15:41:36.920>
created_at = <Date 2009-11-17.22:02:49.412>
labels = ['type-bug', 'tests']
title = "str(datetime_obj) doesn't include microseconds if their value is 0"
updated_at = <Date 2009-12-18.15:41:36.919>
user = 'https://github.com/ezio-melotti'

bugs.python.org fields:

activity = <Date 2009-12-18.15:41:36.919>
actor = 'ezio.melotti'
assignee = 'ezio.melotti'
closed = True
closed_date = <Date 2009-12-18.15:41:36.920>
closer = 'ezio.melotti'
components = ['Tests']
creation = <Date 2009-11-17.22:02:49.412>
creator = 'ezio.melotti'
dependencies = []
files = ['15476']
hgrepos = []
issue_num = 7342
keywords = ['patch']
message_count = 9.0
messages = ['95404', '95405', '95406', '95532', '95536', '95537', '96079', '96344', '96568']
nosy_count = 6.0
nosy_names = ['tim.peters', 'doerwalter', 'georg.brandl', 'pitrou', 'ezio.melotti', 'r.david.murray']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue7342'
versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2']

[ezio-melotti]

Last night, test_strptime failed on one of the buildbots [1] with the
following error:
test test_strptime failed -- Traceback (most recent call last):
  File
"C:\buildslave\3.x.moore-windows\build\lib\test\test_strptime.py", line
279, in test_fraction
    tup, frac = _strptime._strptime(str(now), format="%Y-%m-%d %H:%M:%S.%f")
  File "C:\buildslave\3.x.moore-windows\build\lib\_strptime.py", line
332, in _strptime
    (data_string, format))
ValueError: time data '2009-11-16 17:17:14' does not match format
'%Y-%m-%d %H:%M:%S.%f'

The reason is that the __str__ method of datetime objects
(datetime.now() in the test) adds the trailing .%f only if the
microseconds are != 0. Since the possible values of microseconds are
between 0 and 999999 (both included), there is 1 possibility out of
1000000 that the microseconds of datetime.now() are equal to 0 (and
depending on how the value is incremented it might not include 0 among
the possible values at all). Apparently that was the cause of the
failure in the test.

This can be reproduced easily doing:
>>> from datetime import datetime
>>> str(datetime.now())
'2009-11-17 22:11:23.522951'
>>> str(datetime(2012, 12, 21)) # no microseconds
'2012-12-21 00:00:00'

This behavior is defined in Modules/datetimemodule.c, in the C function
datetime_isoformat (line 4145 on py3k, introduced by tim_one in r30151)
and in isoformat_time (line 1278 on trunk, called by datetime_isoformat,
introduced by walter.doerwald in r55714).

Is there a valid reason to omit the microseconds in case they are equal
to 0 instead of just adding the trailing '.000000'?

If the behavior doesn't change the test can be probably fixed checking
the value of the microseconds before the call to str(). The
documentation and other tests to check this should also be added.

(Thanks to R. David Murray for pointing me in the right direction while
I was investigating the problem.)

[pitrou]

Yes, there is a valid reason. You certainly don't want spurious
microseconds to be displayed when a datetime object was constructed from
a second-precise source (e.g. parsing e-mail headers).

[tim-one]

This behavior is intentional and is documented in the
datetime.isoformat() docs:

"""
Return a string representing the date and time in ISO 8601 format,
YYYY-MM-DDTHH:MM:SS.mmmmmm or, if microsecond is 0, YYYY-MM-DDTHH:MM:SS
...
"""

It was Guido's idea ;-) The point is that __str__ is supposed to
produce "nice" output, and ".0000000" was thought to be more annoying
than useful, since the common case is that datetime objects don't use
microseconds.

[ezio-melotti]

If __str__ is supposed to produce "nice" output, the microsecond shouldn't
be visible at all imho (special cases are not special enough to break the
rules).
If the date/time object is read by a human he probably doesn't care of
the microseconds anyway, if it's parsed by a machine the '0 microseconds'
situation must be special-cased to avoid failures like the one mentioned in
the first message.
The fact that the documentation of datetime.isotime() mentions it is not
enough if the user doesn't know that it's used by str(), so a note should be
added to the doc.
I don't know if/how the situation can be fixed though.

[tim-one]

Ezio, it was Guido's design decision, it was intentional, and it's been
documented from the start(*). So you can disagree with it, but you
won't get anywhere claiming it's "a bug": intentional, documented
behaviors are never "bugs". At best you can make a case for that
they're design errors.

But that won't fly either. It's no more inconsistent than that, e.g.,
most floating-point numbers in practice aren't usually displayed with an
exponent, but if an exponent is needed, one is added to the output:

>>> .25
0.25
>>> .25 * 10**100
2.5e+99

Overly literal readings don't help your case either. Yes, __str__ aims
at producing nice output, but it's pedantic to argue as if that is, or
should be, __str__'s /only/ goal. Python's design just isn't that
simple-minded ;-) If someone uses non-zero microseconds, presumably
they want to see them.

Utterly trivial mechanical parsing of str() output is a non-goal in Python.

Of course the failing test should be repaired, and that was a good
catch. But the chance of changing the language to make the test work
as-is is approximately 0%.

(*) The datetime docs already say:

"""
__str__( )

For a datetime instance d, str(d) is equivalent to d.isoformat(' ').
"""

[ezio-melotti]

Yes, I wrote the previous message from a cellphone and I wasn't able to
check the doc. It is indeed already documented in datetime.__str__ --
sorry for the noise.
I also noticed that the microseconds are not the only thing that can
change in datetime.isoformat(), there are also the separator and the UTC
offset, so if someone is parsing the output of str(d)/d.isoformat(), he
should already include other checks anyway.

I agree that only the test should be fixed (and possibly others tests
should be added to check that str() outputs the right thing when there
is a UTC offset and/or a different separator).

I'll work on a patch for that.