assertion rewrite confuses coverage

[ghost]

When running tests via coverage on python2.7 or python3.4, assertion rewrite confuses code coverage. It works as expected with python3.5 or when --assert=plain is specified.

The reproducer is available at https://gist.github.com/rouge8/0fd866f8122bb4c59938fab79f658c42

[nicoddemus]

Thanks for the report. I don't really understand if this is a problem on py.test's side or coverage's.

I will watch the BitBucket issue to see how it develops.

[RonnyPfannschmidt]

@benjaminp might know more

[ghost]

I narrowed down the reproducer and I'm inclined to think it could be a pytest issue. coverage.py relies on compile() to figure out which byte code maps to which line. My theory is that prior to python-3.5 multiline asserts are mapped to multiple source lines and coverage.py collects events from sys.setttraces that reference these lines and matches them to the bytecodes reported by compile() to verify coverage. When pytest rewrites asserts, I think it does so by assigning the line number of the first line of the assert to all bytecodes of the assert regardless. That works fine with python-3.5+ because that is the default behavior anyway. But that creates a problem with versions prior to python-3.5 because compile() will claim that byte code are associated with source lines for which events are never generated by sys.setttrace because the assert was re-written by pytest in a way that is no longer consistent with the source.

When rewriting asserts pytest should set the line numbers in a way that is consistent with what compile() does to not confuse coverage.py.

Does that make sense ?

[nicoddemus]

Hi @dachary, thanks.

I posted on the BitBucket issue the rewritten AST using pytest-ast-back-to-python. It might help others investigate the issue further.

[ghost]

Closing this issue while working on the assumption that it is a coverage.py bug.

[nicoddemus]

Thanks @dachary for the detailed investigation. I will continue to watch the issue on coverage's repository closely, we can reopen this if we have further findings.