Fix intermittent failuire of test_explicit_Ref_Var_port_write

[mgkwill]

Objective of issue: Fix intermittent failure of test_explicit_Ref_Var_port_write

Lava version:

• 0.2.1 (bug fixes)

I'm submitting a ...

• bug report

Current behavior:

• Intermittent failure of unit test runs:

======================================================================
FAIL: test_explicit_Ref_Var_port_write (tests.lava.magma.runtime.test_ref_var_ports.TestRefVarPorts)
Tests the connection of a RefPort to an explicitly created VarPort.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/lava/lava/tests/lava/magma/runtime/test_ref_var_ports.py", line [120](https://github.com/lava-nc/lava/runs/5238112313?check_suite_focus=true#step:5:120), in test_explicit_Ref_Var_port_write
    self.assertTrue(np.all(recv.var2.get() == np.array([7., 7., 7.])))
AssertionError: False is not true

Expected behavior:

• There is not an Intermittent failure of unit test runs.

Steps to reproduce:

• Run unit tests in CI a few times. See: https://github.com/lava-nc/lava/runs/5238112313?check_suite_focus=true#step:5:114

[mgkwill]

This may not be an issue with Ref Var code but some underlying issue with lava, maybe a race issue with port communication as I don't get this as much when using verbose python -m unittest -v. I see this locally on Linux, but only in MacOS in CI.

Saw this again in the CI today:

======================================================================
FAIL: test_implicit_Ref_Var_port_write (tests.lava.magma.runtime.test_ref_var_ports.TestRefVarPorts)
Tests the connection of a RefPort to an implicitly created VarPort.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/lava/lava/tests/lava/magma/runtime/test_ref_var_ports.py", line 206, in test_implicit_Ref_Var_port_write
    self.assertTrue(np.all(recv.var2.get() == np.array([9., 9., 9.])))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 183 tests in 83.294s

FAILED (failures=1, skipped=9)

[PhilippPlank]

Yes, I do not think this is related to MacOS. The question is what the reason for the failure is, as it is not reproduceable locally for me.

[mgkwill]

An example in windows: https://github.com/lava-nc/lava/runs/5295472855?check_suite_focus=true#step:5:124

[mgkwill]

Yes, I do not think this is related to MacOS. The question is what the reason for the failure is, as it is not reproduceable locally for me.

I've seen it locally on lab dev VM on Linux. It apparently is not OS specific but seems to happen more often on MacOS.

Can you try running unit tests on Lab VM - teams message me for more details.

[mgkwill]

@PhilippPlank might have found the issue. var get/set is not being handled correctly in

lava/src/lava/magma/core/model/py/model.py

Lines 157 to 187 in 72f31d4

	def run(self):
	"""Retrieves commands from the runtime service and calls their
	corresponding methods of the ProcessModels.
	After calling the method of the ProcessModels, the runtime service
	is informed about completion. The loop ends when the STOP command is
	received."""
	while True:
	if self._action == 'cmd':
	cmd = self.service_to_process.recv()[0]
	try:
	if cmd in self._cmd_handlers:
	self._cmd_handlers[cmd]()
	if cmd == MGMT_COMMAND.STOP[0] or self._stopped:
	return
	else:
	raise ValueError(
	f"Illegal RuntimeService command! ProcessModels of "
	f"type {self.__class__.__qualname__} "
	f"{self.model_id} cannot handle "
	f"command: {cmd} ")
	except Exception as inst:
	# Inform runtime service about termination
	self.process_to_service.send(MGMT_RESPONSE.ERROR)
	self.join()
	raise inst
	else:
	# Handle VarPort requests from RefPorts
	self._handle_var_port(self._action)
	self._channel_actions = [(self.service_to_process, lambda: 'cmd')]
	self.add_ports_for_polling()
	self._action = self._selector.select(*self._channel_actions)

FYI @ysingh7