json module won't parse a float that starts with a decimal point

[picomancer]

BPO	19871
Nosy	@tim-one, @terryjreedy, @jcea, @mdickinson, @pitrou, @nedbat, @ezio-melotti, @serhiy-storchaka, @vajrasky
Files	fix_doc_parse_non_valid_json_float.patch parse_non_valid_json_float_with_unit_test.patch: Doc fix and unit test parse_non_valid_json_float_with_unit_test_v2.patch: Doc fix and unit test, updated after Ezio's review.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-12-03.08:21:12.979>
labels = ['easy', 'type-bug', 'tests', 'docs']
title = "json module won't parse a float that starts with a decimal point"
updated_at = <Date 2017-03-07.15:51:01.908>
user = 'https://bugs.python.org/picomancer'

bugs.python.org fields:

activity = <Date 2017-03-07.15:51:01.908>
actor = 'serhiy.storchaka'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Tests']
creation = <Date 2013-12-03.08:21:12.979>
creator = 'picomancer'
dependencies = []
files = ['33159', '33160', '33558']
hgrepos = []
issue_num = 19871
keywords = ['patch', 'easy']
message_count = 17.0
messages = ['205080', '205083', '205091', '205114', '205119', '205242', '205246', '205247', '206280', '206292', '208519', '210807', '210812', '210817', '210819', '210825', '215087']
nosy_count = 13.0
nosy_names = ['tim.peters', 'holdenweb', 'terry.reedy', 'jcea', 'mark.dickinson', 'pitrou', 'nedbat', 'ezio.melotti', 'cvrebert', 'docs@python', 'serhiy.storchaka', 'vajrasky', 'picomancer']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'pending'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19871'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

[picomancer]

Try the following in your favorite Python version:

    import json
    json.loads(".5")

On my Python (2.7.4 and 3.3.1 on Ubuntu Saucy Salamander), I get an exception. However, x = .5 is a valid Python number.

With respect to the parsing of floats by the json module, the docs state:

By default, this is equivalent to ``float(num_str)``.

This statement does not match the behavior I have observed in every version of Python I have tried, and is still in the bleeding-edge (as of this writing) at http://hg.python.org/cpython/file/9283a9c5d0ce/Doc/library/json.rst

I think it's clear that the following changes should definitely be implemented:

(1) The docs and behavior should match
(2) Whatever the desired behavior is, there is a unit test specifically for this corner case

Of course, to implement (1), there are two routes:

(1a) Leading decimal floats should be accepted by the json module; the behavior should be changed to match the docs. Supported by Postel's Law -- "be liberal in what [your program] accept[s]"), see http://en.wikipedia.org/wiki/Postel%27s_law and the slightly relaxed attitude toward standards compliance detailed in the json module documentation.

(1b) Leading decimal floats should be rejected by the json module; the docs should be changed to match the behavior. This fits with a strict standards compliance worldview.

I think (1a) is better. In my particular use case, I was manually writing a json file with several numerical parameters. The backtrace given by json.load(open("whatever.json", "r")) is uninformative and merely says "No JSON object could be decoded"; finding the token the parser considered to be malformed was fairly easy since there were only six or seven keys. It could have been much worse if I was making manual changes to a larger JSON file.

[pitrou]

I think it would be better to adhere to the JSON spec, which doesn't allow numbers to start with a decimal point:
http://json.org/

If we go this way, the documentation should at least be fixed; and, as you say, we could also add a unit test for it.

[serhiy-storchaka]

Agree with Antoine.

[mdickinson]

In context, the doc is correct:

"""
parse_float, if specified, will be called with the string of every JSON float to be decoded. By default, this is equivalent to float(num_str).
"""

IIUC, parse_float only comes into play once the JSON source has already been tokenized, and the tokenization stage has already rejected things like '.5' by that point. (The point of parse_float is that you can choose to turn numeric strings into decimal.Decimal instances instead of floats if you so wish.)

I agree it could use clarification.

[nedbat]

There are other forms of numbers allowed by Python that are not allowed by JSON: "001.1"

Oddly, with all of the strictness in JSON, the exponent-marker "e" can be upper- or lower-case: 1e1 and 1E1 are both valid JSON.

[mdickinson]

Oddly, with all of the strictness in JSON, the exponent-marker "e"
can be upper- or lower-case

I'd guess that the aim is that common floating-point output formats from a variety of languages are valid JSON. That would also explain why both '+' and '-' are allowed on the exponent, but only '-' on the significand, and why leading zeros are permitted on the exponent but not the significand.

[tim-one]

We should adhere to the json spec, but there's no harm (and some real good!) in the docs pointing out notable cases where json and Python syntax differ.

[serhiy-storchaka]

There are too many cases where json and Python syntax differ. Final comma("[1, 2,]"), non-string keys ({1: 2}), tuples ("(1, 2)"), leading zeros ("0001"), hexadecimal integers ("0xaf"), escapes of astral characters('"\U0001d504"'), single quotes ("'spam'"), octal escape codes ("\015"), etc, etc...

[vajrasky]

How about this doc fix?