Doc write exception with unicode characters

[djc]

From hhmw56...@gmail.com on April 28, 2014 23:42:35

What steps will reproduce the problem? The following test script fails with 0.9, but works fine with 0.8:

import json
import couchdb

db = couchdb.Server(u"https://:@")['']
docs = [{u'Type':"TestDoc", u'Test':True, u'val':unichr(40960)}]
ret = db.update( docs )
print ret What is the expected output? What do you see instead? The write should succeed.

The exception is:

File "couch_encoding_test.py", line 10, in
ret = db.update( docs )
File "/usr/local/lib/python2.7/dist-packages/CouchDB-0.9-py2.7.egg/couchdb/client.py", line 764, in update
_, _, data = self.resource.post_json('_bulk_docs', body=content)
File "/usr/local/lib/python2.7/dist-packages/CouchDB-0.9-py2.7.egg/couchdb/http.py", line 527, in post_json
*_params)
File "/usr/local/lib/python2.7/dist-packages/CouchDB-0.9-py2.7.egg/couchdb/http.py", line 546, in _request_json
headers=headers, *_params)
File "/usr/local/lib/python2.7/dist-packages/CouchDB-0.9-py2.7.egg/couchdb/http.py", line 542, in _request
credentials=self.credentials)
File "/usr/local/lib/python2.7/dist-packages/CouchDB-0.9-py2.7.egg/couchdb/http.py", line 327, in request
resp = _try_request_with_retries(iter(self.retry_delays))
File "/usr/local/lib/python2.7/dist-packages/CouchDB-0.9-py2.7.egg/couchdb/http.py", line 285, in _try_request_with_retries
return _try_request()
File "/usr/local/lib/python2.7/dist-packages/CouchDB-0.9-py2.7.egg/couchdb/http.py", line 307, in _try_request
conn.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 812, in _send_output
msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xea in position 52: ordinal not in range(128) What version of the product are you using? On what operating system? couchdb-python 0.9, ubuntu 12.04 Please provide any additional information below. If the parameter to couchdb.Server is a str instead of a unicode object, the script executes correctly.

Original issue: http://code.google.com/p/couchdb-python/issues/detail?id=235

[djc]

From djc.ochtman on July 06, 2014 04:02:00

I tried this test case and couldn't get it to fail on the current tip of the default branch:

diff --git a/couchdb/tests/client.py b/couchdb/tests/client.py
--- a/couchdb/tests/client.py
+++ b/couchdb/tests/client.py
@@ -185,6 +185,10 @@
self.assertEqual(u'Iñtërnâtiônàlizætiøn', self.db[u'føø'][u'bår'])
self.assertEqual(u'ASCII', self.db[u'føø'][u'baz'])

• def test_unicode_235(self):

•    doc = {u'type': 'test', u'val': unichr(40960)}
  

•    self.db.update([doc])
  

  def test_disallow_nan(self):
  try:
  self.db['foo'] = {'number': float('nan')}

This makes me think this has been fixed already. If you can still reproduce it with a more recent changeset, feel free to reopen.

Status: WorksForMe

[djc]

From @mikebell on July 14, 2014 23:26:19

"If the parameter to couchdb.Server is a str instead of a unicode object, the script executes correctly."

In other words, the behaviour changes depending on whether the couchdb.Server object is initialized with a string or a unicode string.

[djc]

From djc.ochtman on July 14, 2014 23:57:16

Right. Have you tried reproducing with current tip? Because I still can't.

[djc]

From hhmw56743 on July 16, 2014 11:13:54

I have reproduced the problem in 0.10 and the latest couchdb-python from github.

Looking at your test code it is not obvious to me that the Server object is being initialized with a unicode string, which is necessary to trigger the behavior I'm describing.

[djc]

Okay, at least I have a reproducible test case now:

diff --git a/couchdb/tests/client.py b/couchdb/tests/client.py
index 1b4cb57..b8c7452 100644
--- a/couchdb/tests/client.py
+++ b/couchdb/tests/client.py
@@ -116,6 +116,15 @@ class ServerTestCase(testutil.TempDatabaseMixin, unittest.TestCase):
         ls = self.server.uuids(count=10)
         assert type(ls) == list and len(ls) == 10

+    def test_235_unicode_server(self):
+        server = client.Server(client.DEFAULT_BASE_URL.decode('utf-8'))
+        dbname = 'couchdb-python/test-235-unicode-server'
+        db = server.create(dbname)
+        try:
+            db.update([{'foo': u'\ua000'}])
+        finally:
+            server.delete(dbname)
+

 class DatabaseTestCase(testutil.TempDatabaseMixin, unittest.TestCase):

I also had a fix that worked on Python 2, but it doesn't work on Python 3. Needs some more research...