format(1234.5, '.4') gives misleading result

[mdickinson]

BPO	5864
Nosy	@rhettinger, @mdickinson, @ericvsmith
Files	issue5864.patch issue5864_v2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/mdickinson'
closed_at = <Date 2009-07-07.15:28:41.186>
created_at = <Date 2009-04-28.11:38:19.999>
labels = ['interpreter-core', 'type-bug']
title = "format(1234.5, '.4') gives misleading result"
updated_at = <Date 2009-07-07.15:28:41.185>
user = 'https://github.com/mdickinson'

bugs.python.org fields:

activity = <Date 2009-07-07.15:28:41.185>
actor = 'eric.smith'
assignee = 'mark.dickinson'
closed = True
closed_date = <Date 2009-07-07.15:28:41.186>
closer = 'eric.smith'
components = ['Interpreter Core']
creation = <Date 2009-04-28.11:38:19.999>
creator = 'mark.dickinson'
dependencies = []
files = ['13814', '13816']
hgrepos = []
issue_num = 5864
keywords = ['patch']
message_count = 16.0
messages = ['86728', '86742', '86749', '86797', '86804', '86806', '86808', '86811', '86816', '86819', '86825', '87863', '87886', '90238', '90239', '90240']
nosy_count = 3.0
nosy_names = ['rhettinger', 'mark.dickinson', 'eric.smith']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5864'
versions = ['Python 2.6']

[mdickinson]

In all versions of Python from 2.6 up, I get the following behaviour:

>>> format(123.456, '.4')
'123.5'
>>> format(1234.56, '.4')
'1235.0'
>>> format(12345.6, '.4')
'1.235e+04'

The first and third results are as I expect, but the second is somewhat
misleading: it gives 5 significant digits when only 4 were requested,
and moreover the last digit is incorrect.

I propose that Python 2.7 and Python 3.1 be changed so that the output
for the second line above is '1.235e+03'.

Note that in both Python and C, '%.<precision>g' formatting switches to
exponential notation at 1e<precision> precisely to avoid the problem of
producing more significant digits than were requested; I'm proposing
that the same solution be applied for '' formatting, except that since
the empty format code is always required to produce at least one digit
after the decimal point, the switch should happen at 1e<precision-1>
instead of 1e<precision>.

This change should not be backported to 2.6 or 3.0 since there's a small
risk of breakage (particular in doctests).

[ericvsmith]

I agree this should be changed.

This:
>>> format(1234.56, '.4')
'1235.0'
is definitely a problem.

[rhettinger]

I think the change should be deemed a bug and backported. Nothing good
can come from having version differences persist or from having the odd
looking output.

[mdickinson]

I'm working on a patch. The fix for the new-style PyOS_double_to_string
using Gay's dtoa is a one-liner. The fix for the fallback code is going
to take a little while longer...

[mdickinson]

Here's a patch for py3k; backporting to trunk should be straightforward.
Backporting to 2.6 may involve more work.

[mdickinson]

Updated patch; the previous patch messed str() up.

[ericvsmith]

I'm getting an undefined _remove_trailing_zeros using a normal build
(not using the fallback code). I think it's now always called.

[mdickinson]

I think it's now always called.

I think you're right. I've moved remove_trailing_zeros further up, out of
the #ifdef PY_NO_SHORT_FLOAT_REPR.

[ericvsmith]

Using the _v2 patch, I tested both with the fallback code and with Gay's
code, and both work.

It's sort of shocking just how much effort you had to go through! But I
can't see a simpler way to do it.

It looks good to me.

[mdickinson]

Thanks, Eric. Applied to py3k in r72109. I've blocked it from release30-
maint for now, since it will almost certainly not be possible to svnmerge
it cleanly, but that doesn't preclude backporting it at some stage.

I'll backport to trunk soon.

[mdickinson]

Backported to trunk in r72119.

[mdickinson]

Eric, any thoughts on whether this should be backported to 2.6 and 3.0?
It looks like quite a lot of work.