Log type of unserializable value when raising JSON TypeError

[MadisonMay]

BPO	24641
Nosy	@rhettinger, @terryjreedy, @etrepum, @pitrou, @ezio-melotti, @serhiy-storchaka
PRs	bpo-24641: Improved error message for JSON unserializible keys. #4364

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2017-11-25.15:53:30.343>
created_at = <Date 2015-07-15.20:49:32.887>
labels = ['3.7', 'type-feature', 'library']
title = 'Log type of unserializable value when raising JSON TypeError'
updated_at = <Date 2017-11-25.15:53:30.342>
user = 'https://bugs.python.org/MadisonMay'

bugs.python.org fields:

activity = <Date 2017-11-25.15:53:30.342>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2017-11-25.15:53:30.343>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2015-07-15.20:49:32.887>
creator = 'Madison May'
dependencies = []
files = []
hgrepos = []
issue_num = 24641
keywords = ['patch']
message_count = 7.0
messages = ['246776', '246862', '246865', '246919', '306045', '306046', '306961']
nosy_count = 7.0
nosy_names = ['rhettinger', 'terry.reedy', 'bob.ippolito', 'pitrou', 'ezio.melotti', 'serhiy.storchaka', 'Madison May']
pr_nums = ['4364']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24641'
versions = ['Python 3.7']

[MadisonMay]

Currently the json lib only logs the string representation of the variable, which does not always include type information.

I recently ran into a difficult to debug issue with code similar to the following:

import json
import numpy as np
d = {'data': np.int16(5)}
json.dumps(d)

which produces the following error:

TypeError: 5 is not JSON serializable

It took us quite a while to determine that 5 was actually of type np.int instead of the native type.

A cursory glance at StackOverflow suggests that I'm not alone in running into this issue: http://stackoverflow.com/questions/10872604/json-dump-throwing-typeerror-is-not-json-serializable-on-seemingly-vali

I'd like to consider modifying the error message to be more similar to the following:

TypeError: 5 of type `numpy.int16` is not JSON serializable

[terryjreedy]

A typical TypeError message use the following pattern:
TypeError: 'int' object is not callable
as it is the class, not the value, that is the problem.

If the same is always true for the JSON TypeError, at the point of failure, then the dumps message could follow the same pattern.

However, I don't know if the message *is* value independent. The SO question asked about

TypeError: {'album': [u"Rooney's Lost Album"], 'title': [u'The Kids
After Sunset'], 'artist': [u'Rooney']} is not JSON serializable

and the OP claimed in a comment to the accepted answer that the problem was that the value objects represented as lists were not list() objects, just as '5' here is not an int() object.

The JSON error code, in 3.5 at

File "C:\Programs\Python35\lib\json\encoder.py", line 180, in default
raise TypeError(repr(o) + " is not JSON serializable")

could be expanded to check whether the string representation starts with the class name and if it does not, add "'classname' object" at the front. This would solve Madison's posted issue, but not the SO problem.
(It would, however, have made it clear that the {}s represented a real dict() object and directed attention inward.)

For testing, compile('','','exec') returns an object that cannot be dumped.

To me, this looks more like an enhancement than bugfix, so a change might be limited to future releases.

[serhiy-storchaka]

I think it would be better to change error message to mention the type only. Yet one argument is that the repr of affected object can be very large, while the type name usually is short enough. repr() even can raise an exception (e.g. MemoryError).

[etrepum]

This seems like a very reasonable proposal. It would be great if we could also include a path in the error message (e.g. obj["foo"][1]["bar"]) as well to provide enough context to track down the error quickly.

[serhiy-storchaka]

This issue was partially fixed in bpo-26623. The error message for unsupported types now is "Object of type '%s' is not JSON serializable" % o.__class__.__name__.

But this change is not complete. The error message for non-string keys still contains the repr of a key. And the example for default() in the module docstring contains the repr of an object.

[serhiy-storchaka]

The idea about including a path to non-serializable object looks interesting (and it would be even more useful for pickling), but harder to implement.

[serhiy-storchaka]

New changeset cfa797c by Serhiy Storchaka in branch 'master':
bpo-24641: Improved error message for JSON unserializible keys. (bpo-4364)
cfa797c