TokenError when generating html report

[nedbat]

Originally reported by Chris AtLee (Bitbucket: chrisatlee, GitHub: Unknown)

---

I'm trying to run coverage on the buildbot source like this:

#!sh
coverage run --branch =trial buildbot.test
coverage html -d cover -i 

And get the following exception:

#!python

Traceback (most recent call last):
  File "/home/catlee/.virtualenvs/buildbot.git/bin/coverage", line 9, in <module>
    load_entry_point('coverage==3.4a1', 'console_scripts', 'coverage')()
  File "/home/catlee/src/coveragepy/coverage/cmdline.py", line 639, in main
    status = CoverageScript().command_line(argv)
  File "/home/catlee/src/coveragepy/coverage/cmdline.py", line 531, in command_line
    directory=options.directory, **report_args)
  File "/home/catlee/src/coveragepy/coverage/control.py", line 522, in html_report
    include=self.config.include
  File "/home/catlee/src/coveragepy/coverage/html.py", line 47, in report
    self.report_files(self.html_file, morfs, directory, omit, include)
  File "/home/catlee/src/coveragepy/coverage/report.py", line 63, in report_files
    report_fn(cu, self.coverage._analyze(cu))
  File "/home/catlee/src/coveragepy/coverage/control.py", line 458, in _analyze
    return Analysis(self, it)
  File "/home/catlee/src/coveragepy/coverage/results.py", line 30, in __init__
    self.statements, self.excluded = self.parser.parse_source()
  File "/home/catlee/src/coveragepy/coverage/parser.py", line 188, in parse_source
    self._raw_parse()
  File "/home/catlee/src/coveragepy/coverage/parser.py", line 91, in _raw_parse
    for toktype, ttext, (slineno, _), (elineno, _), ltext in tokgen:
  File "/usr/lib/python2.5/tokenize.py", line 292, in generate_tokens
    raise TokenError, ("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (36, 0))

I think this is because there's one of the .html files in the coverage report for some reason. I can attach the .coverage file if that helps.

This patch seems to fix the problem:

#!diff

diff --git a/coverage/parser.py b/coverage/parser.py
--- a/coverage/parser.py
+++ b/coverage/parser.py
@@ -179,17 +179,20 @@ class CodeParser(object):
 
         Return values are 1) a sorted list of executable line numbers, and
         2) a sorted list of excluded line numbers.
 
         Reported line numbers are normalized to the first line of multi-line
         statements.
 
         """
-        self._raw_parse()
+        try:
+            self._raw_parse()
+        except:
+            raise NoSource("Couldn't parse file: %s" % self.filename)
 
         excluded_lines = self.first_lines(self.excluded)
         ignore = excluded_lines + list(self.docstrings)
         lines = self.first_lines(self.statement_starts, ignore)
 
         return lines, excluded_lines
 
     def arcs(self):

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/82
• This issue had attachments: .coverage, template_pkg.tgz. See the original issue for details.

[nedbat]

This is curious. It would help if I could see the .coverage file, thanks.

[nedbat]

Original comment by Chris AtLee (Bitbucket: chrisatlee, GitHub: Unknown)

---

buildbot uses Jinja as its web template engine. I wonder if Jinja is faking out the file/line numbers in the generated python files during execution?

[nedbat]

Original comment by Chris AtLee (Bitbucket: chrisatlee, GitHub: Unknown)

---

Ooops, didn't mean to change the component

[nedbat]

Yes, "coverage debug data" shows this entry:

/home/catlee/src/buildbot/master/buildbot/status/web/templates/revmacros.html: 51 lines

Any chance you could make me a small .tar with a reproducible case?

[nedbat]

Original comment by Chris AtLee (Bitbucket: chrisatlee, GitHub: Unknown)

---

I tarred up my virtual env...and it's 31MB. I can put that up somewhere, otherwise, here are the steps to reproduce.

#!bash

virtualenv --no-site-packages coveragetest
cd coveragetest
source bin/activate
git clone http://github.com/djmitche/buildbot.git
cd buildbot/master
python setup.py develop
pip install mock
pip install coverage

coverage run --branch $VIRTUAL_ENV/bin/trial buildbot.test
coverage html -d cover -i

[nedbat]

Thanks, Chris, I'm able to reproduce it with the steps you gave.

[nedbat]

Original comment by Nathan Yergler (Bitbucket: nyergler, GitHub: nyergler)

---

We're experiencing this, as well, when attempting to generate the XML report. I've attached a small test case that you can use by:

#!bash

$ virtualenv --no-site-packages jinjatest
$ cd jinjatest
$ source bin/activate
$ ./bin/easy_install Jinja2
$ ./bin/easy_install coverage
$ tar zxvf template_pkg.tgz
$ ./bin/coverage run ./template_pkg/__init__.py
$ ./bin/coverage xml

Ka-boom!

I initially thought this was due to our use of PackageLoader, which loads templates from within packages, but rewrote my simple example to use the FileSystemLoader.

[nedbat]

In changeset <<changeset d27dbdb02d5a (bb)>>, I changed the code to not trace *.html files, fixing this problem.

[nedbat]

Original comment by Alexandre Conrad (Bitbucket: aconrad, GitHub: aconrad)

---

I have the same problem. The following change worked for me:

if filename.endswith(".html") or filename.endswith(".jinja2"):

as my templates have a .jinja2 extention name.

Would there be a way to exclude patterns from the command line? Or maybe just skip files that don't compile and report it at the end of the XML conversion? (don't know if that's acceptable though).

[nedbat]

Original comment by Anonymous

---

I'm experiencing this problem with a Jinja2 template that ends in ".js.inc" I believe this is a valid use case. The Jinja template documentation at http://jinja.pocoo.org/docs/templates/ says: "A template is simply a text file. It can generate any text-based format (HTML, XML, CSV, LaTeX, etc.). It doesn’t have a specific extension, .html or .xml are just fine."

As a workaround I can either include=[".py"] or omit=[".js.inc"]

But it seems worthwhile to investigate a fix in coverage.py. Some thoughts:

• The PyTrace._trace() function gets a frame object. Is there anything in that object that indicates that the file isn't an actual python file? Maybe something in frame.f_code?
• It seems like the report generation is only able to generate annotated source code for .py source. Would it make sense to ignore files with other extensions? Or ignore any exceptions that happen while attempting to exec the code?

[nedbat]

Original comment by Anonymous

---

(Sorry, last comment was from me. I guess I don't have a bitbucket account. My name is Mark Doliner. Email address is mark@kingant.net, if there's anything I can help with.)

[nedbat]

After soliciting opinions in http://nedbatchelder.com/blog/201203/what_files_should_coverage_measure.html , I've fixed this bug in <<changeset 316cb1ba9bea (bb)>> to not report errors if the file doesn't have a Python-ish file extension.

[nedbat]

Original comment by Mark Doliner (Bitbucket: markdoliner, GitHub: markdoliner)

---

Thanks, Ned!

[nedbat]

This seems to still be happening with Jinja:

pelican)...nt/pelican% ~/.virtualenvs/pelican/bin/coverage run `which unit2` discover                                                                                                                                             Julian@air
sh: pandoc: command not found
........No handlers could be found for logger "pelican.contents"
................s.sss.................
----------------------------------------------------------------------
Ran 46 tests in 1.580s

OK (skipped=4)
(pelican)...nt/pelican% ~/.virtualenvs/pelican/bin/coverage html                                                                                                                                                                   Julian@air
Traceback (most recent call last):
  File "/Users/Julian/.virtualenvs/pelican/bin/coverage", line 9, in <module>
    load_entry_point('coverage==3.5.2', 'console_scripts', 'coverage')()
  File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/cmdline.py", line 657, in main
    status = CoverageScript().command_line(argv)
  File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/cmdline.py", line 549, in command_line
    directory=options.directory, **report_args)
  File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/control.py", line 603, in html_report
    reporter.report(morfs)
  File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/html.py", line 87, in report
    self.report_files(self.html_file, morfs, self.config.html_dir)
  File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/report.py", line 83, in report_files
    report_fn(cu, self.coverage._analyze(cu))
  File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/control.py", line 543, in _analyze
    return Analysis(self, it)
  File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/results.py", line 30, in __init__
    self.statements, self.excluded = self.parser.parse_source()
  File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/parser.py", line 202, in parse_source
    self._raw_parse()
  File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/parser.py", line 106, in _raw_parse
    for toktype, ttext, (slineno, _), (elineno, _), ltext in tokgen:
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/tokenize.py", line 357, in generate_tokens
    raise TokenError, ("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (62, 0))

[nedbat]

Closed again with <<changeset cad566556ce0 (bb)>>

[nedbat]

Original comment by Davide Setti (Bitbucket: vad, GitHub: vad)

---

I have the same issue with coverage 3.5.3 and Jinja 2.6. My templates ends with .jinja2. Do i need to give them .html extension?

Traceback (most recent call last):
  File "manage_ve.py", line 72, in <module>
    execute_from_command_line(sys.argv)
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django_jenkins/management/commands/__init__.py", line 70, in handle
    if test_runner.run_tests(test_labels):
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django_jenkins/runner.py", line 345, in run_tests
    self.teardown_test_environment()
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django_jenkins/runner.py", line 325, in teardown_test_environment
    signals.teardown_test_environment.send(sender=self)
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 172, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django_jenkins/tasks/with_coverage.py", line 59, in teardown_test_environment
    self.coverage.xml_report(morfs=morfs, outfile=os.path.join(self.output_dir, 'coverage.xml'))
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/control.py", line 630, in xml_report
    reporter.report(morfs, outfile=outfile)
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/xmlreport.py", line 56, in report
    self.report_files(self.xml_file, morfs)
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/report.py", line 83, in report_files
    report_fn(cu, self.coverage._analyze(cu))
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/control.py", line 543, in _analyze
    return Analysis(self, it)
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/results.py", line 30, in __init__
    self.statements, self.excluded = self.parser.parse_source()
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/parser.py", line 207, in parse_source
    self._raw_parse()
  File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/parser.py", line 110, in _raw_parse
    for toktype, ttext, (slineno, _), (elineno, _), ltext in tokgen:
  File "/usr/lib/python2.7/tokenize.py", line 351, in generate_tokens
    ("<tokenize>", lnum, pos, line))
  File "<tokenize>", line 12
    <span class="famfamfamflag famfamfamflags-{{ lang_code }}"></span>
    ^
IndentationError: unindent does not match any outer indentation level

[nedbat]

Davide, no, you don't have to change what you are doing. This is a bug in coverage.py. I'll re-open this ticket and fix it soon.

[nedbat]

This is fixed in <<changeset 5b49a4eacb8c (bb)>>.

[nedbat]

Issue #217 was marked as a duplicate of this issue.