Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP


NaN in object_info_reply JSON causes parse error #2187

tkf opened this Issue · 4 comments

3 participants


When I send object_info_request for pandas.DataFrame.reindex, I've got the following JSON reply.

Please look the list at content.defaults. It contains NaN. This causes parse error in Emacs JSON parser I am using for my Emacs Notebook Client tkf/emacs-ipython-notebook#38.

{ "content" : { "argspec" : { "args" : [ "self",
          "defaults" : [ null,
          "varargs" : null,
          "varkw" : null
      "base_class" : "<type 'instancemethod'>",
      "call_def" : null,
      "call_docstring" : null,
      "class_docstring" : null,
      "definition" : "\u001b[0mpandas\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreindex\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcolumns\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlevel\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfill_value\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mnan\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "docstring" : "Conform DataFrame to new index with optional filling logic, placing\nNA/NaN in locations having no value in the previous index. A new object\nis produced unless the new index is equivalent to the current one and\ncopy=False\n\nParameters\n----------\nindex : array-like, optional\n    New labels / index to conform to. Preferably an Index object to\n    avoid duplicating data\ncolumns : array-like, optional\n    Same usage as index argument\nmethod : {'backfill', 'bfill', 'pad', 'ffill', None}, default None\n    Method to use for filling holes in reindexed DataFrame\n    pad / ffill: propagate last valid observation forward to next valid\n    backfill / bfill: use NEXT valid observation to fill gap\ncopy : boolean, default True\n    Return a new object, even if the passed indexes are the same\nlevel : int or name\n    Broadcast across a level, matching Index values on the\n    passed MultiIndex level\nfill_value : scalar, default np.NaN\n    Value to use for missing values. Defaults to NaN, but can be any\n    \"compatible\" value\n\nExamples\n--------\n>>> df.reindex(index=[date1, date2, date3], columns=['A', 'B', 'C'])\n\nReturns\n-------\nreindexed : same type as calling instance",
      "file" : "/home/takafumi/.virtualenvs/mypy/local/lib/python2.7/site-packages/pandas/core/",
      "found" : true,
      "init_definition" : null,
      "init_docstring" : null,
      "isalias" : false,
      "isclass" : null,
      "ismagic" : false,
      "length" : null,
      "name" : "pandas.DataFrame.reindex",
      "namespace" : "Interactive",
      "source" : null,
      "string_form" : "<unbound method DataFrame.reindex>",
      "type_name" : "instancemethod"
  "header" : { "msg_id" : "6c393862-7412-4a5f-a4b2-2993f5430a6d",
      "msg_type" : "object_info_reply",
      "session" : "ce9c7083-370e-416f-8fe9-111b68410228",
      "username" : "kernel"
  "msg_id" : "6c393862-7412-4a5f-a4b2-2993f5430a6d",
  "msg_type" : "object_info_reply",
  "parent_header" : { "msg_id" : "d83e0578-d143-48b6-a672-f123034ed316",
      "msg_type" : "object_info_request",
      "session" : "55e244de-e83e-496f-805c-883e62c70462",
      "username" : "username"

You can check that indeed it's not a valid JSON using

Parse error on line 17:
...ll,                NaN,               
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['

Well that's awkward.

The error, of course, comes in writing out the call specification for pandas.core.frame.DataFrame.reindex which is:

    def reindex(self, index=None, columns=None, method=None, level=None,
                fill_value=np.nan, limit=None, copy=True):

Apparently there is no NaN or +/-Inf in the JSON spec. There some stack overflow questions about this, like Python: sending NaN in JSON.

I'm not sure what an appropriate resolution for this issue is yet.


Well, reading the stackoverflow it looks like browsers support it so maybe it is fine.
But I guess sending repr() should do the job, as it is only used for UI (not used for something like remote procedure call).


stdlib json has an allow_nan flag, so we can set allow_nan=False to catch this (but not fix it).

We already have a json_clean method (IPython.utils.jsonutil), which is where we handle non-jsonable objects, so that's where the fix should be.

@bfroehle bfroehle closed this

Closed by #2194.

@minrk minrk referenced this issue from a commit
@minrk minrk Backport PR #2194: clean nan/inf in json_clean
these values are not json-compliant

closes #2187
@jasongrout jasongrout referenced this issue in sagemath/sagecell

graphics with NaN throw a json error #433

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.