Ensure the encoding reported by chardet is unicode

Fixes issue 323. Thanks to Barry Warsaw for reporting this! git-svn-id: http://feedparser.googlecode.com/svn/trunk@680 73d2b349-402e-0410-baf4-070fd12ab5b7
unode · Feb 5, 2012 · 6e2a1f9 · 6e2a1f9
1 parent 4268fab
commit 6e2a1f9
Show file tree

Hide file tree

Showing 2 changed files with 2 additions and 1 deletion.
diff --git a/NEWS b/NEWS
@@ -6,6 +6,7 @@ Changes coming in the next release:
     * Issue 313 (include the compression test files in MANIFEST.in)
     * Issue 315 (HTTP server for unit tests runs on 0.0.0.0)
     * Issue 321 (malformed URIs can cause ValueError to be thrown)
+    * Issue 323 (installing chardet causes 11 unit test failures)
     * Issue 325 (map `description_detail` to `summary_detail`)
 
 5.1 - December 2, 2011

diff --git a/feedparser/feedparser.py b/feedparser/feedparser.py
@@ -3839,7 +3839,7 @@ def parse(url_file_stream_or_string, etag=None, modified=None, agent=None, refer
             break
     # if no luck and we have auto-detection library, try that
     if (not known_encoding) and chardet:
-        proposed_encoding = chardet.detect(data)['encoding']
+        proposed_encoding = unicode(chardet.detect(data)['encoding'], 'ascii', 'ignore')
         if proposed_encoding and (proposed_encoding not in tried_encodings):
             tried_encodings.append(proposed_encoding)
             try: