Mix of encoded and decoded strings in final result

[jorgecarleitao]

Consider the following function to return a string in HTML from a pdf:

from StringIO import StringIO

from pdfminer.converter import TextConverter, XMLConverter, HTMLConverter
from pdfminer.layout import LAParams
from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
from pdfminer.pdfpage import PDFPage

def _run_sample(self, sample_name, format):
        fp = file('samples/' + sample_name + '.pdf', 'rb')
        rsrcmgr = PDFResourceManager()

        result = StringIO()
        device = HTMLConverter(rsrcmgr, result, laparams=LAParams(detect_vertical = True))
        interpreter = PDFPageInterpreter(rsrcmgr, device)
        for page in PDFPage.get_pages(fp, caching=True, check_extractable=True):
            interpreter.process_page(page)
        fp.close()
        device.close()

        result.seek(0)
        return result.read().decode('utf-8')

This fails with the error:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 0: ordinal not in range(128)
in ".../2.7/lib/python2.7/StringIO.py", line 106, in seek.

I searched, and this seems to be because

The StringIO object can accept either Unicode or 8-bit strings, but mixing the two may take some care. If both are used, 8-bit strings that cannot be interpreted as 7-bit ASCII (that use the 8th bit) will cause a UnicodeError to be raised when getvalue() is called.
(from a docstring inside StringIO.py).

Should we try to uniformize this?

[euske]

The font name in PDF was Unicode and mixed with normal strings. Fixed in 14fd0fd.