Skip to content

NaN in object_info_reply JSON causes parse error #2187

tkf opened this Issue Jul 22, 2012 · 4 comments

3 participants

tkf commented Jul 22, 2012

When I send object_info_request for pandas.DataFrame.reindex, I've got the following JSON reply.

Please look the list at content.defaults. It contains NaN. This causes parse error in Emacs JSON parser I am using for my Emacs Notebook Client tkf/emacs-ipython-notebook#38.

{ "content" : { "argspec" : { "args" : [ "self",
          "defaults" : [ null,
          "varargs" : null,
          "varkw" : null
      "base_class" : "<type 'instancemethod'>",
      "call_def" : null,
      "call_docstring" : null,
      "class_docstring" : null,
      "definition" : "\u001b[0mpandas\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreindex\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcolumns\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlevel\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfill_value\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mnan\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "docstring" : "Conform DataFrame to new index with optional filling logic, placing\nNA/NaN in locations having no value in the previous index. A new object\nis produced unless the new index is equivalent to the current one and\ncopy=False\n\nParameters\n----------\nindex : array-like, optional\n    New labels / index to conform to. Preferably an Index object to\n    avoid duplicating data\ncolumns : array-like, optional\n    Same usage as index argument\nmethod : {'backfill', 'bfill', 'pad', 'ffill', None}, default None\n    Method to use for filling holes in reindexed DataFrame\n    pad / ffill: propagate last valid observation forward to next valid\n    backfill / bfill: use NEXT valid observation to fill gap\ncopy : boolean, default True\n    Return a new object, even if the passed indexes are the same\nlevel : int or name\n    Broadcast across a level, matching Index values on the\n    passed MultiIndex level\nfill_value : scalar, default np.NaN\n    Value to use for missing values. Defaults to NaN, but can be any\n    \"compatible\" value\n\nExamples\n--------\n>>> df.reindex(index=[date1, date2, date3], columns=['A', 'B', 'C'])\n\nReturns\n-------\nreindexed : same type as calling instance",
      "file" : "/home/takafumi/.virtualenvs/mypy/local/lib/python2.7/site-packages/pandas/core/",
      "found" : true,
      "init_definition" : null,
      "init_docstring" : null,
      "isalias" : false,
      "isclass" : null,
      "ismagic" : false,
      "length" : null,
      "name" : "pandas.DataFrame.reindex",
      "namespace" : "Interactive",
      "source" : null,
      "string_form" : "<unbound method DataFrame.reindex>",
      "type_name" : "instancemethod"
  "header" : { "msg_id" : "6c393862-7412-4a5f-a4b2-2993f5430a6d",
      "msg_type" : "object_info_reply",
      "session" : "ce9c7083-370e-416f-8fe9-111b68410228",
      "username" : "kernel"
  "msg_id" : "6c393862-7412-4a5f-a4b2-2993f5430a6d",
  "msg_type" : "object_info_reply",
  "parent_header" : { "msg_id" : "d83e0578-d143-48b6-a672-f123034ed316",
      "msg_type" : "object_info_request",
      "session" : "55e244de-e83e-496f-805c-883e62c70462",
      "username" : "username"

You can check that indeed it's not a valid JSON using

Parse error on line 17:
...ll,                NaN,               
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['

Well that's awkward.

The error, of course, comes in writing out the call specification for pandas.core.frame.DataFrame.reindex which is:

    def reindex(self, index=None, columns=None, method=None, level=None,
                fill_value=np.nan, limit=None, copy=True):

Apparently there is no NaN or +/-Inf in the JSON spec. There some stack overflow questions about this, like Python: sending NaN in JSON.

I'm not sure what an appropriate resolution for this issue is yet.

tkf commented Jul 22, 2012

Well, reading the stackoverflow it looks like browsers support it so maybe it is fine.
But I guess sending repr() should do the job, as it is only used for UI (not used for something like remote procedure call).

IPython member
minrk commented Jul 23, 2012

stdlib json has an allow_nan flag, so we can set allow_nan=False to catch this (but not fix it).

We already have a json_clean method (IPython.utils.jsonutil), which is where we handle non-jsonable objects, so that's where the fix should be.

@bfroehle bfroehle closed this Jul 27, 2012

Closed by #2194.

@minrk minrk added a commit that referenced this issue Jul 28, 2012
@minrk minrk Backport PR #2194: clean nan/inf in json_clean
these values are not json-compliant

closes #2187
@jasongrout jasongrout referenced this issue in sagemath/sagecell Dec 12, 2013

graphics with NaN throw a json error #433

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.