UTF-8 decode URL + data

README

@@ -1,12 +1,11 @@
 In Python, read the .80 file format, for 80legs web crawl results.
 
+The URL and data are UTF-8 decoded.
+
 From http://80legs.pbworks.com/Results:
     For people interested in deserializing in other languages, the file format this creates and reads is:
         <classID><versionID><URL-SIZE><URL><DATA-SIZE><DATA>
     Note that:
         * The last 4 items (<URL-SIZE><URL><DATA-SIZE><DATA>) repeat for each url/data pair.
         * <classID>, <versionID>, <URL-SIZE>, and <DATA-SIZE> are encoded 32-bit integers.
         * The url is encoded using UTF-8.  
-
-ISSUES:
-    * I don't Unicode decode either the URL or the data.

eightyformat.py

@@ -27,6 +27,7 @@ def read(file):
         (DATASIZE,) = struct.unpack("i", l)
 #        print DATASIZE
         data = str(file.read(DATASIZE))
+        data = data.decode("utf-8")
         yield (url, data)
 #        print data
 #        print data.decode("utf-8")