Tests that fail before the scheduler are not handled properly

[ktbarrett]

For example:

cocotb/tests/test_cases/test_cocotb/test_cocotb.py

Lines 85 to 88 in 92a032a

	@cocotb.test(expect_fail=True)
	def test_not_a_coroutine(dut):
	"""Example of a failing to use the yield keyword in a test"""
	dut._log.warning("This test will fail because we don't yield anything")

This is considered "N/A" in the regression results banner and passes whether expect_fail or expect_error are set to True or False. A traceback is printed, but it's not apparent there was any failure if you are running a number of tests, like in test_cocotb.

The regression manager nor the cocotb.test decorator make any attempt to handle potential issues.

cocotb/cocotb/regression.py

Line 157 in 92a032a

self._queue.append(_test(self._dut))

cocotb/cocotb/decorators.py

Lines 477 to 478 in 92a032a

	def __call__(self, *args, **kwargs):
	return RunningTest(self._func(*args, **kwargs), self)

I put some time into refactoring initialize and handle_result in the regression manager, but felt like I more or less had to copy-paste to victory, which I didn't like. I think a larger refactor is in order.

[ktbarrett]

Note, this isn't an issue if the test coroutine is an async function. When async functions are called, nothing executes immediately, so they cannot fail until they get to the scheduler.

[eric-wieser]

Pretty sure we could detect this with insoect.isgeneratorfunction or similar

[ktbarrett]

I just didn't find a clean way of modifying handle_result to call it from initialize in the error case, handle_result expects a RunningTest object.

[eric-wieser]

PR appreciated, this has been annoying me for a while when writing and screwing up new tests.

[ktbarrett]

I'm actually wondering if instead of modifying the regression manager I should modify the decorators so that __call__ does not immediately run the coroutine's body to the first yield. async coroutine functions have that deferred execution behavior, it doesn't make much sense for async and non-async coroutines to have different behavior.

If that's the case, this will wait until #1255 or #1259 are merged.

EDIT: I didn't quite understand what was going on, but now I do. Cocotb coroutines are not immediately started just like async coroutines. The issue is with errors that cause coroutine initialization to fail like NameError, not necessarily any exception before the first yield.

[eric-wieser]

The issue is with errors that cause coroutine initialization to fail like NameError, not necessarily any exception before the first yield.

I suspect the issue is actually errors during coroutine initialization, such as:

@cocotb.coroutine
def not_a_coroutine(dut):
    raise RuntimeError()

or

@cocotb.coroutine
def returns_a_coroutine(dut):
    return the_coroutine_with_a_typo()