Test Coverage not measured from methods implemented in Python

[karivatj]

I've written a unit test that tests a ClassMethod implemented in Python [Language = python].

The test runs okay, but the coverage is not being measured. Is Embedded Python supported?

If I execute the test, which calls a ClassMethod written in ObjectScript that, in turn, calls the Python method, the ObjectScript method's coverage is measured, but the Python method's coverage is not.

[isc-tleavitt]

Embedded python isn't supported right now. Under the hood, TestCoverage uses %Monitor.System.LineByLine (aka ^%SYS.MONLBL), which monitors coverage in .INT code which is then mapped back to the class definition. For Embedded Python, there's no .INT code, so that approach just doesn't work.

I haven't spent any time looking into EP support yet, mostly because you're the first to ask about it! A likely approach would be to use https://coverage.readthedocs.io/en/7.4.0/api.html and merge the data from that back to the unified class-level view same as we already do for .INT code.

One caveat on this will be compatibility - this package must continue to work for versions before Embedded Python existed. To enable that we'll have everything EP-dependent in the same class and produce a pre-2022.1 artifact without that class.

[isc-tleavitt]

From a bit of experimentation it seems that just using the Python coverage package might not work. Simple example:

Class EP.CoverageTest
{

ClassMethod Run() [ Language = python ]
{
from coverage import Coverage

cov = Coverage(source_pkgs=["EP"])
cov.start()
CoverageTest.DoSomething()
cov.stop()
cov.html_report(directory="C:\\Temp\\PyCoverage\\")
}

ClassMethod DoSomething() [ Language = python ]
{
return 42
}

}

I get:
*%Exception.PythonException 230 ^^0^ <class 'coverage.exceptions.NoSource'>: No source for code: 'C:\InterSystems\IRIS\mgr\user\CoverageTest'.

Because coverage is expecting to work with files, this might just not work; might instead need to go a bit deeper with https://docs.python.org/3/library/sys.html#sys.settrace

[isc-tleavitt]

So, crazy enough, this works - meaning, there's coverage of the one covered line, "return 42", shown in the report output from the following:

Class EP.CoverageTest
{

ClassMethod Run() [ Language = python ]
{
from coverage import Coverage
import iris
import traceback

try:
	iris.cls('EP.CoverageTest').OutputClass("EP.CoverageTest")
	cov = Coverage(source_pkgs=["EP"])
	cov.start()
	CoverageTest.DoSomething()
	cov.stop()
	cov.html_report(directory="C:\\Temp\\PyCoverage\\")
except:
	print(traceback.format_exc())
}

ClassMethod DoSomething() [ Language = python ]
{
return 42
}

ClassMethod OutputClass(classname As %String)
{
	set root = "C:\InterSystems\IRIS\mgr\user\" // TODO: parameterize
	set file = ##class(%Stream.FileCharacter).%OpenId(root_$Piece(classname,".",*),,.sc)
	$$$ThrowOnError(sc)
	for i=1:1:$Get(^ROUTINE(classname_".py",0,0)) {
		do file.WriteLine($Get(^ROUTINE(classname_".py",0,i)))
	}
	$$$ThrowOnError(file.%Save())
}

}

Problem is, that clearly won't work if there are multiple classes with the same short classname, and it requires writing out to a database directory which is a little odd. More testing is still needed. In case relevant, all of my testing was on:
IRIS for Windows (x86-64) 2023.1.2 (Build 450U) Mon Oct 16 2023 10:18:34 EDT

@karivatj can you indicate the priority of a solution, from your perspective? (I've spent more time on this today than I should have probably - you might know the feeling: https://xkcd.com/356/)

[karivatj]

Thanks for looking into this! In our company, Python is becoming more prevalent in our codebase, so coverage measurement of Python would be very beneficial in an otherwise excellent tool for code coverage. I would say that this would be an important feature because Python is becoming more and more common in our projects due to its flexibility and ease of use. We have a lot of unit tests that test the Python code, but unfortunately, test coverage reports do not reflect that.

If you happen to find a robust solution for this, it would be greatly appreciated.

P.S.: I know the feeling described in the xkcd comic!

[isc-tleavitt]

@karivatj we have released version 4.0.0 of TestCoverage with support for coverage measurement of Embedded Python! Feel free to check it out when you have time, we'd appreciate your feedback.