annotate command hits unicode happy fun time

[nedbat]

Originally reported by Tom Callaway (Bitbucket: spotrh, GitHub: spotrh)

---

The annotate command chokes if unicode characters are parsed in the file. Here's a sample file (test.py):

#!python

# -*- coding: utf-8 -*-
#
# é

print "foobar"

[spot@localhost master]$ coverage run test.py
foobar
[spot@localhost master]$ coverage annotate -d covdir
Traceback (most recent call last):
  File "/usr/bin/coverage", line 9, in <module>
    load_entry_point('coverage==4.0a5', 'console_scripts', 'coverage')()
  File "/usr/lib64/python2.7/site-packages/coverage/cmdline.py", line 676, in main
    status = CoverageScript().command_line(argv)
  File "/usr/lib64/python2.7/site-packages/coverage/cmdline.py", line 451, in command_line
    directory=options.directory, **report_args)
  File "/usr/lib64/python2.7/site-packages/coverage/control.py", line 924, in annotate
    reporter.report(morfs, directory=directory)
  File "/usr/lib64/python2.7/site-packages/coverage/annotate.py", line 42, in report
    self.report_files(self.annotate_file, morfs, directory)
  File "/usr/lib64/python2.7/site-packages/coverage/report.py", line 80, in report_files
    report_fn(cu, self.coverage._analyze(cu))
  File "/usr/lib64/python2.7/site-packages/coverage/annotate.py", line 92, in annotate_file
    dest.write(line)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 2: ordinal not in range(128)

It looks like the obvious fix is to force encode the line being written to utf-8, a patch is attached against 4.0a5 which does this. This resolves the issue in my testing, but I wanted to make sure I had it right first.

Note: I do not see this issue in the 3.7 code, only in 4.0.

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/363
• This issue had attachments: python-coverage-4.0a5-unicodefix.patch. See the original issue for details.

[nedbat]

Merged in lep/coverage.py (pull request #48)

Fix #363: crash when annotating non-ascii characters in python 2.

→ <<cset 5544096a77fa (bb)>>

[nedbat]

Fix #363: crash when annotating non-ascii characters in python 2.

→ <<cset e0492ef9e320 (bb)>>

[nedbat]

Merged in lep/coverage.py (pull request #48)

Fix #363: crash when annotating non-ascii characters in python 2.

→ <<cset 5544096a77fa (bb)>>

[nedbat]

Original comment by Tom Callaway (Bitbucket: spotrh, GitHub: spotrh)

---

EDIT: Never mind. I see how you fixed this differently later.

[nedbat]

Original comment by Vjacheslav Fyodorov (Bitbucket: fvabit, GitHub: Unknown)

---

This fix crashes python 3 version in fedora 22.

[nedbat]

@fvabit Could you provide more details?

[nedbat]

Original comment by Vjacheslav Fyodorov (Bitbucket: fvabit, GitHub: Unknown)

---

In python 3 line.encode('utf-8') produces bytes, not string, so we have:

#!python

  File "/usr/lib64/python3.4/site-packages/coverage/annotate.py", line 92, in an
notate_file                                                                    
    dest.write(line.encode('utf-8'))
TypeError: must be str, not bytes

I have reverted this line back, and all works fine.

[nedbat]

@fvabit Help me out here. Please provide complete details. What Python program were you measuring? What version of Python were you using? How did you run coverage?

[nedbat]

@fvabit it looks like you are running the pull request code, not the final code? Can you try the tip on bitbucket and see if it works for you?

[nedbat]

Original comment by Vjacheslav Fyodorov (Bitbucket: fvabit, GitHub: Unknown)

---

#!console

$ uname -a
Linux localhost.localdomain 4.0.6-300.fc22.x86_64 #1 SMP Tue Jun 23 13:58:53 UTC
 2015 x86_64 x86_64 x86_64 GNU/Linux                                           

$ python3 -V
Python 3.4.2

$ dnf info python3-coverage
Last metadata expiration check performed 1:54:12 ago on Mon Jul 13 11:44:10 2015.
Installed Packages
Name        : python3-coverage
Arch        : x86_64
Epoch       : 0
Version     : 4.0
Release     : 0.6.a5.fc22
Size        : 708 k
Repo        : @System
From repo   : fedora
Summary     : Code coverage testing module for Python 3
URL         : http://nedbatchelder.com/code/modules/coverage.html
License     : BSD and (MIT or GPLv2)
Description : Coverage.py is a Python 3 module that measures code coverage
            : during Python execution. It uses the code analysis tools and
            : tracing hooks provided in the Python standard library to determine
            : which lines are executable, and which have been executed.

$ coverage3 help
Coverage.py, version 4.0a5
Measure, collect, and report on code coverage in Python programs.

usage: coverage <command> [options] [args]

Commands:
    annotate    Annotate source files with execution information.
    combine     Combine a number of data files.
    erase       Erase previously collected coverage data.
    help        Get help on using coverage.py.
    html        Create an HTML report.
    report      Report coverage stats on modules.
    run         Run a Python program and measure code execution.
    xml         Create an XML report of coverage results.

Use "coverage help <command>" for detailed help on any command.
For full documentation, see https://coverage.readthedocs.org/en/coverage-4.0a5

$ cat tmp.py
print ('Hello World!')

$ coverage3 run tmp.py
Hello World!

$ coverage3 report
Name     Stmts   Miss  Cover
----------------------------
tmp.py       1      0   100%

$ coverage3 annotate
Traceback (most recent call last):
  File "/usr/bin/coverage3", line 9, in <module>
    load_entry_point('coverage==4.0a5', 'console_scripts', 'coverage')()
  File "/usr/lib64/python3.4/site-packages/coverage/cmdline.py", line 676, in ma
in                                                                             
    status = CoverageScript().command_line(argv)
  File "/usr/lib64/python3.4/site-packages/coverage/cmdline.py", line 451, in co
mmand_line                                                                     
    directory=options.directory, **report_args)
  File "/usr/lib64/python3.4/site-packages/coverage/control.py", line 924, in an
notate                                                                         
    reporter.report(morfs, directory=directory)
  File "/usr/lib64/python3.4/site-packages/coverage/annotate.py", line 42, in re
port                                                                           
    self.report_files(self.annotate_file, morfs, directory)
  File "/usr/lib64/python3.4/site-packages/coverage/report.py", line 80, in repo
rt_files                                                                       
    report_fn(cu, self.coverage._analyze(cu))
  File "/usr/lib64/python3.4/site-packages/coverage/annotate.py", line 92, in an
notate_file                                                                    
    dest.write(line.encode('utf-8'))
TypeError: must be str, not bytes

$ 

[nedbat]

Original comment by Vjacheslav Fyodorov (Bitbucket: fvabit, GitHub: Unknown)

---

Bitbucket code differs from my (which is from Fedora 22). Sorry. It should be updated in Fedora 22 distribution.