NaN in object_info_reply JSON causes parse error

[tkf]

When I send object_info_request for pandas.DataFrame.reindex, I've got the following JSON reply.

Please look the list at content.defaults. It contains NaN. This causes parse error in Emacs JSON parser I am using for my Emacs Notebook Client tkf/emacs-ipython-notebook#38.

{ "content" : { "argspec" : { "args" : [ "self",
              "index",
              "columns",
              "method",
              "level",
              "fill_value",
              "copy"
            ],
          "defaults" : [ null,
              null,
              null,
              null,
              NaN,
              true
            ],
          "varargs" : null,
          "varkw" : null
        },
      "base_class" : "<type 'instancemethod'>",
      "call_def" : null,
      "call_docstring" : null,
      "class_docstring" : null,
      "definition" : "\u001b[0mpandas\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreindex\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcolumns\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlevel\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfill_value\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mnan\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "docstring" : "Conform DataFrame to new index with optional filling logic, placing\nNA/NaN in locations having no value in the previous index. A new object\nis produced unless the new index is equivalent to the current one and\ncopy=False\n\nParameters\n----------\nindex : array-like, optional\n    New labels / index to conform to. Preferably an Index object to\n    avoid duplicating data\ncolumns : array-like, optional\n    Same usage as index argument\nmethod : {'backfill', 'bfill', 'pad', 'ffill', None}, default None\n    Method to use for filling holes in reindexed DataFrame\n    pad / ffill: propagate last valid observation forward to next valid\n    backfill / bfill: use NEXT valid observation to fill gap\ncopy : boolean, default True\n    Return a new object, even if the passed indexes are the same\nlevel : int or name\n    Broadcast across a level, matching Index values on the\n    passed MultiIndex level\nfill_value : scalar, default np.NaN\n    Value to use for missing values. Defaults to NaN, but can be any\n    \"compatible\" value\n\nExamples\n--------\n>>> df.reindex(index=[date1, date2, date3], columns=['A', 'B', 'C'])\n\nReturns\n-------\nreindexed : same type as calling instance",
      "file" : "/home/takafumi/.virtualenvs/mypy/local/lib/python2.7/site-packages/pandas/core/frame.py",
      "found" : true,
      "init_definition" : null,
      "init_docstring" : null,
      "isalias" : false,
      "isclass" : null,
      "ismagic" : false,
      "length" : null,
      "name" : "pandas.DataFrame.reindex",
      "namespace" : "Interactive",
      "source" : null,
      "string_form" : "<unbound method DataFrame.reindex>",
      "type_name" : "instancemethod"
    },
  "header" : { "msg_id" : "6c393862-7412-4a5f-a4b2-2993f5430a6d",
      "msg_type" : "object_info_reply",
      "session" : "ce9c7083-370e-416f-8fe9-111b68410228",
      "username" : "kernel"
    },
  "msg_id" : "6c393862-7412-4a5f-a4b2-2993f5430a6d",
  "msg_type" : "object_info_reply",
  "parent_header" : { "msg_id" : "d83e0578-d143-48b6-a672-f123034ed316",
      "msg_type" : "object_info_request",
      "session" : "55e244de-e83e-496f-805c-883e62c70462",
      "username" : "username"
    }
}

You can check that indeed it's not a valid JSON using http://jsonlint.com/:

Parse error on line 17:
...ll,                NaN,               
----------------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['

[bfroehle]

Well that's awkward.

The error, of course, comes in writing out the call specification for pandas.core.frame.DataFrame.reindex which is:

    def reindex(self, index=None, columns=None, method=None, level=None,
                fill_value=np.nan, limit=None, copy=True):

Apparently there is no NaN or +/-Inf in the JSON spec. There some stack overflow questions about this, like Python: sending NaN in JSON.

I'm not sure what an appropriate resolution for this issue is yet.

[tkf]

Well, reading the stackoverflow it looks like browsers support it so maybe it is fine.
But I guess sending repr() should do the job, as it is only used for UI (not used for something like remote procedure call).

[minrk]

stdlib json has an allow_nan flag, so we can set allow_nan=False to catch this (but not fix it).

We already have a json_clean method (IPython.utils.jsonutil), which is where we handle non-jsonable objects, so that's where the fix should be.

[bfroehle]

Closed by #2194.