Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

clean nan/inf in json_clean #2194

Merged
merged 2 commits into from

4 participants

Min RK Jason Grout Brian E. Granger Bradley M. Froehle
Min RK
Owner

these values are not json-compliant

closes #2187

Min RK minrk clean nan/inf in json_clean
these values are not json-compliant
fbc781d
IPython/utils/jsonutil.py
@@ -30,6 +30,12 @@
ISO8601="%Y-%m-%dT%H:%M:%S.%f"
ISO8601_PAT=re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+$")
+# float constants
+NAN = float('nan')
+INF = float('inf')
+NINF = float('-inf')
+INFS = (INF, NINF)
+
Bradley M. Froehle Collaborator

Is it worth defining all of these if we'll only use INFS in the end?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
IPython/utils/jsonutil.py
((6 lines not shown))
# containers that we need to convert into lists
container_to_list = (tuple, set, types.GeneratorType)
+ if isinstance(obj, float):
+ # cast out-of-range floats to their reprs
+ if obj != obj or obj in INFS:
+ return repr(obj)
Bradley M. Froehle Collaborator

You and I both know obj != obj is the way that you test for a a float being NaN, but I wonder if it wouldn't be a lot clearer to do:

if math.isnan(obj) or math.isinf(obj):
    return repr(obj)
Min RK Owner
minrk added a note

I used the exact same tests in stlib json, but those seem clearer, so I will do that.

Min RK Owner
minrk added a note

done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Jason Grout

Can we add datetime objects to the list of things to clean? I believe json not handling python datetime objects is the sole reason for these lines that cut out some datetime objects from messages: https://github.com/ipython/ipython/blob/master/IPython/frontend/html/notebook/handlers.py#L381

@ellisonbg: am I correct in assuming that the reason for popping those dates is because of JSONing limitations?

Update: never mind; I guess I didn't see the rest of the file. There already are functions there to clean dates.

I'm curious why we don't just embed these sorts of things in our own json dumps/loads functions, so you can just call the custom dumps function and it will automatically take care of things like encoding for inf, encoding dates, etc. Python lets you define custom encoders and decoders: http://docs.python.org/library/json.html

Brian E. Granger
Owner
Min RK
Owner
Jason Grout

So can we just make a custom JSON encoding class that does both the effects of json_clean and default=date_default automatically? Then it's as easy as calling our JSON encoding function with no arguments.

Jason Grout

(see http://stackoverflow.com/questions/455580/json-datetime-between-python-and-javascript for some more ideas how to do datetimes. Especially the suggestion there to use .isoformat() instead of doing your own iso formatting.)

Min RK
Owner

Hm, I wonder how I missed .isoformat(). Other than that, I don't see anything preferable in the SO question.

One advantage of decoupling cleaning / datetime-formatting: If a message gets to the Session object that is not JSON-serializable, that is a bug and should raise. Most datetime objects are actually created inside the Session object as the timestamp in every header.

Bradley M. Froehle bfroehle merged commit 157d99a into from
Bradley M. Froehle
Collaborator

I just committed this NaN/Inf JSON fix. If we want to pursue a custom JSON encoding class that can be done in another issue. (I have NOT opened another issue to track that idea. Perhaps @jasongrout can do it?)

Min RK minrk referenced this pull request from a commit
Min RK minrk Backport PR #2194: clean nan/inf in json_clean
these values are not json-compliant

closes #2187
7048487
Min RK minrk deleted the branch
Yaroslav Halchenko yarikoptic referenced this pull request from a commit in yarikoptic/ipython
Yaroslav Halchenko yarikoptic Merge commit 'rel-0.13-33-gcfc5692' into debian-01X
* commit 'rel-0.13-33-gcfc5692': (33 commits)
  Backport PR #2347: adjust division error message checking to account for Python 3
  Backport PR #2305: RemoteError._render_traceback_ calls self.render_traceback
  Backport PR #2280: fix SSH passwordless check for OpenSSH
  Backport PR #2270: SSHLauncher tweaks
  Backport PR #2261: Fix: longest_substr([]) -> ''
  Backport PR #2250: fix html in notebook example
  Backport PR #2235: remove spurious print statement from setupbase.py
  fixup
  Backport PR #2223: Custom tracebacks
  Backport PR #2214: use KernelApp.exec_lines/files in IPEngineApp
  Backport PR #2212: catch errors in markdown javascript
  Backport PR #2194: clean nan/inf in json_clean
  Backport PR #2177: remove numpy install from travis/tox scripts
  Backport PR #2169: ipdb: pdef, pdoc, pinfo magics all broken
  Backport PR #2186: removed references to h5py dependence in octave magic documentation
  Backport PR #2185: added test for %store, fixed storemagic
  Backport PR #2170: Fix tab completion with IPython.embed_kernel().
  Backport PR #2163: fix 'remote_profie_dir' typo in SSH launchers
  Backport PR #2117: use explicit url in notebook example
  Backport PR #2126: ipcluster broken with any batch (PBS/LSF/SGE)
  ...
fc7c7f5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jul 23, 2012
  1. Min RK

    clean nan/inf in json_clean

    minrk authored
    these values are not json-compliant
  2. Min RK

    use math to check for nan/inf

    minrk authored
This page is out of date. Refresh to see the latest.
Showing with 8 additions and 1 deletion.
  1. +8 −1 IPython/utils/jsonutil.py
9 IPython/utils/jsonutil.py
View
@@ -11,6 +11,7 @@
# Imports
#-----------------------------------------------------------------------------
# stdlib
+import math
import re
import sys
import types
@@ -161,11 +162,17 @@ def json_clean(obj):
"""
# types that are 'atomic' and ok in json as-is. bool doesn't need to be
# listed explicitly because bools pass as int instances
- atomic_ok = (unicode, int, float, types.NoneType)
+ atomic_ok = (unicode, int, types.NoneType)
# containers that we need to convert into lists
container_to_list = (tuple, set, types.GeneratorType)
+ if isinstance(obj, float):
+ # cast out-of-range floats to their reprs
+ if math.isnan(obj) or math.isinf(obj):
+ return repr(obj)
+ return obj
+
if isinstance(obj, atomic_ok):
return obj
Something went wrong with that request. Please try again.