commitWithin gives serializing error

[wichert]

An extract from our errors logs (running under mod_wsgi):

[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]     solr.add([data], commit=None, commitWithin=60 * 1000)
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]   File "/opt/s4uadmin/eggs/pysolr-2.0.15-py2.7.egg/pysolr.py", line 686, in add
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]     m = ET.tostring(message, encoding='utf-8')
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1121, in tostring
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]     ElementTree(element).write(file, encoding, method=method)
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 815, in write
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]     serialize(write, self._root, encoding, qnames, namespaces)
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 927, in _serialize_xml
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]     v = _escape_attrib(v, encoding)
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1087, in _escape_attrib
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]     _raise_serialization_error(text)
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1047, in _raise_serialization_error
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1]     "cannot serialize %r (type %s)" % (text, type(text).__name__)
[Tue Mar 06 10:11:10 2012] [error] [client 127.0.0.1] TypeError: cannot serialize 60000 (type int)

[toastdriven]

Bizarre. No clue what's going wrong here, since etree has no problems serializing integers. Using pdb to inspect the message (pysolr.py, line 686 from the traceback) would probably be informative.

[wichert]

pdb under mod_wsgi in production isn't really an option I'm afraid, and I haven't been able to reproduce this in a test environment.

[toastdriven]

Then perhaps setting up a try/except block that catches the error & logs what was sent? You might also try installing lxml, as it looks like you're falling back on Python's etree, which sometimes has subtle differences from lxml.etree.

[tleach]

We just hit this. Can pysolr not simply do the type conversion of the commitWithin arg from int -> str?

Happy to contribute a PR if so desired.

[acdha]

This is an annoying limitation in etree:

>>> from xml.etree.ElementTree import Element, ElementTree
>>> import sys
>>> ElementTree(Element(42)).write(sys.stdout)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 812, in write
    self._root, encoding, default_namespace
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 881, in _namespaces
    _raise_serialization_error(tag)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1047, in _raise_serialization_error
    "cannot serialize %r (type %s)" % (text, type(text).__name__)
TypeError: cannot serialize 42 (type int)

Since ElementTree is [arguably] braindead I'd vote for a pull request adding an str() call on pysolr.py#L689 and, ideally, a test exercising this path.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.