JSON serialization problems with ObjectId type (MongoDB) #518

kaazoo opened this Issue Jun 11, 2011 · 3 comments


None yet
2 participants

kaazoo commented Jun 11, 2011

I have a problem with creating new tasks through client.lbview.apply() when I inserted an object as a MongoDB doctument in the database before. IPython seems to fail to convert the _id to JSON. I tried to remove any reference of _id, but that didn't help. IPython still seems to see some reference to an BSON ObjectId type.

$ python2.6 sendjob_ipython.py -s 1 -e 5 -b 1 -r blender -f /usr/local/drqueue/tmp/icetest.blend -n "job042" 
Traceback (most recent call last):
  File "sendjob_ipython.py", line 72, in <module>
  File "sendjob_ipython.py", line 47, in main
  File "/Users/kaazoo/Documents/Entwicklung/drqueue-entwicklung/drqueue-zmq/DrQueue/client.py", line 131, in job_run
    ar = self.lbview.apply(DrQueue.run_script_with_env, render_script, env_dict)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/parallel/client/view.py", line 204, in apply
    return self._really_apply(f, args, kwargs)
  File "<string>", line 2, in _really_apply
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/parallel/client/view.py", line 52, in sync_results
    ret = f(self, *args, **kwargs)
  File "<string>", line 2, in _really_apply
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/parallel/client/view.py", line 41, in save_ids
    ret = f(self, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/parallel/client/view.py", line 973, in _really_apply
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/parallel/client/client.py", line 921, in send_apply_message
    subheader=subheader, track=track)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/parallel/streamsession.py", line 259, in send
    to_send = self.serialize(msg, ident)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/parallel/streamsession.py", line 205, in serialize
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/parallel/streamsession.py", line 51, in <lambda>
    json_packer = lambda obj: jsonapi.dumps(obj, **{_default_key:_date_default})
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/zmq/utils/jsonapi.py", line 74, in dumps
    return _squash_unicode(jsonmod.dumps(o, separators=(',',':'),**kwargs))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/__init__.py", line 237, in dumps
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/encoder.py", line 367, in encode
    chunks = list(self.iterencode(o))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/encoder.py", line 309, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/encoder.py", line 275, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/encoder.py", line 317, in _iterencode
    for chunk in self._iterencode_default(o, markers):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/encoder.py", line 323, in _iterencode_default
    newobj = self.default(o)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/IPython/parallel/streamsession.py", line 48, in _date_default
    raise TypeError("%r is not JSON serializable"%obj)
TypeError: ObjectId('4df3d1e053c27b2598000000') is not JSON serializable

What can I do?


minrk commented Jun 11, 2011

Are you connecting to MongoDB directly (i.e. with a PyMongo Connection as opposed to an IPython.parallel.controller.mongodb.MongoDB object)?
You must be careful, because all direct MongoDB queries include the _id key (just like all IPython db queries include msg_id), so if you are explicitly querying MongoDB you must scrub the _id after your queries (our MongoDB backend object does this).

If you stick a print statement (print msg['header']) in Session.serialize, you will see where the bad _id object is getting inserted. I presume it's getting in the subheader by way of targets, etc. (though possibly the msg_id?).

kaazoo commented Jun 11, 2011

I think my problem was the following:

self.ip_client.session.session = job_id

After I converted the job_id (MongoDB _id), it seems to work.

self.ip_client.session.session = str(job_id)

@kaazoo kaazoo closed this Jun 11, 2011


minrk commented Jun 12, 2011

Yes - everything that goes into the message header/content must be serializable by the Session's packer (JSON by default). You can't put arbitrary binary data in there (to JSON, a MongoDB ObjectId is just a random class).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment