UnicodeDecodeError on python3 with pygments and unicode source file

[lewtds]

Try running python3 -m pudb test.py with this source code. Pudb will crash immediately.

Note the Unicode characters: ươ.

[inducer]

What error message are you getting? On 2.7, I get an exception about a missing encoding declaration. Once I add

# encoding: utf-8

to the top of the file, everything works. On Python 3, I get this:

Traceback (most recent call last):
  File "/usr/lib/python3.2/runpy.py", line 161, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python3.2/runpy.py", line 74, in _run_code
    exec(code, run_globals)
  File "/home/andreas/mypy3/lib/python3.2/site-packages/pudb-2012.3-py3.2.egg/pudb/__main__.py", line 3, in <module>
    main()
  File "/home/andreas/mypy3/lib/python3.2/site-packages/pudb-2012.3-py3.2.egg/pudb/run.py", line 30, in main
    steal_output=options.steal_output)
  File "/home/andreas/mypy3/lib/python3.2/site-packages/pudb-2012.3-py3.2.egg/pudb/__init__.py", line 22, in runscript
    dbg = _get_debugger(steal_output=steal_output)
  File "/home/andreas/mypy3/lib/python3.2/site-packages/pudb-2012.3-py3.2.egg/pudb/__init__.py", line 11, in _get_debugger
    dbg = Debugger(**kwargs)
  File "/home/andreas/mypy3/lib/python3.2/site-packages/pudb-2012.3-py3.2.egg/pudb/debugger.py", line 151, in __init__
    for bpoint_descr in load_breakpoints():
  File "/home/andreas/mypy3/lib/python3.2/site-packages/pudb-2012.3-py3.2.egg/pudb/settings.py", line 402, in load_breakpoints
    return parse_breakpoints(lines)
  File "/home/andreas/mypy3/lib/python3.2/site-packages/pudb-2012.3-py3.2.egg/pudb/settings.py", line 368, in parse_breakpoints
    if get_breakpoint_invalid_reason(filename, lineno) is None:
  File "/home/andreas/mypy3/lib/python3.2/site-packages/pudb-2012.3-py3.2.egg/pudb/lowlevel.py", line 48, in get_breakpoint_invalid_reason
    if lineno not in get_executable_lines_for_file(filename):
  File "/home/andreas/mypy3/lib/python3.2/site-packages/pudb-2012.3-py3.2.egg/pudb/lowlevel.py", line 23, in get_executable_lines_for_file
    codes = [compile("".join(getlines(filename)), filename, "exec")]
  File "/home/andreas/research/software/env/lib/python2.7/codecs.py", line 16
    except ImportError, why:
                      ^
SyntaxError: invalid syntax

Is this what you're seeing? (This backtrace is just plain weird to me--where on earth did Python 2.7 enter the picture? I thought this was on 3.2.)

[inducer]

Python 2.7 came from a stored breakpoint I had sitting around. (so, never mind)

So with Python 3.2 I'm getting:

'utf-8' codec can't decode byte 0xc6 in position 3: unexpected end of data

I'm not utf-8-competent enough to decide who's right here: Is what you've got valid utf-8 or no? I'm going to close this bug for now on the assumption that the complaint is legitimate. Please reopen if that's wrong or if this is somehow pudb's fault.

[rominf]

I confirm this bug. See files that contain 100% valid utf-8 symbols. One of this files crash pudb, while other doesn't.

I use latest version from pypi.

[inducer]

Reproduced.