Str.s is always unicode when string mode is unspecified

[trotterdylan]

In Python 2.x I would expect string literals without a mode specifier (i.e. not u'foo' or b'foo') to be type str. They always seem to be unicode:

>>> type('foo')
<type 'str'>
>>> type(pythonparser.parse('"foo"\n').body[0].value.s)
<type 'unicode'>

[trotterdylan]

I've worked around the problem locally by changing Lexer._replace_escape() so that when the result should be bytes instead of unicode (i.e. it has the 'b' prefix or it has no prefix and version < 3.0), it builds the string by joining a list of bytes objects, rather than a list of unicode objects.

This turns out to work well in most cases, but is problematic with certain file encodings. For example, if the Python file has coding: iso-8859-1 then there may be raw characters outside range(128) in string literals. I've bypassed this by making sure none of my source code has such raw characters inside string literals.

One possible solution is for the Lexer (or the Buffer maybe) to accept the encoding of the source being processed. Assuming the input buffer contains unicode data, characters in byte strings could then be encoded to bytes using that encoding.

[whitequark]

Essentially the problem you've encountered is that pythonparser currently does not attempt to determine the file encoding from the magic comment. That was a deliberate omission back when I wrote it, to simplify my job. The proper solution here is to make Buffer enquire the encoding of the file and then decode the input string from it.

It should also be fine to produce bytes() objects when the b"" prefix is present.

I'll gladly accept PRs with these two changes.