Timed assertion

[derkork]

I'm currently writing up some tests for my game, which is mostly an integration test thing. So basically i have a driver class that provides an abstraction of the game for the tests, so I can write tests like this:

func test_i_can_start_a_new_game():
	# when
	_game_driver.main_menu.press_new_game()
	yield(_runner.simulate_until_object_signal(Global, "scene_changed"), "completed")
	
	# then
	assert_that(_game_driver.new_game_dialog.is_present()).is_true()	

These tests are very resilient to code and UI changes as they are on a high level, and they immensely help verifying that the game still works. However you can immediately see the problem with these kinds of tests. Most actions do not have an immediate effect. There are animations played which take time, there is stuff loaded which takes time, so I have to litter waits everywhere. On top of that, those waits actually need to know what is going on below the hood. E.g in the example above, when I press "New Game" in the main menu, actually a scene switch happens. So the test needs to know that and it needs to wait for signalling that this scene switch has happened.

However this is not what I want to test. On a high level I want to test:

• When: I press "New Game" in the main menu
• Then: The "New Game" dialog opens

I don't actually care if a scene needs to be loaded or if some animation plays. So I wondered if we could maybe have timed assertions. Basically the assert_that function that takes an object will get the option to switch it to a timed assert by calling the within_millis function. A timed assert allows you to repeatedly check for a condition until either the condition is satisfied or a timeout occurs. If the condition is not satisfied within the timeout, the assertion fails. Finally you can yield on such assertions. With this, I could rewrite above code like this:

func test_i_can_start_a_new_game():
	# when
	_game_driver.main_menu.press_new_game()
	
	# then
	yield( assert_that( _game_driver.new_game_dialog ) \
            .within_millis(3000).result_of_call("is_present").is_true(), "completed" )

This would make integration test code much easier to write (and read) and it would also help decouple the test code from the actual implementation. This could potentially even eliminate the need for having the simulate_until_xxx methods in the scene runner, e.g. a timed assertion could also check for signals being emitted:

func test_for_a_signal():
	# when
	_car.drive_towards(_wall)
	
	# then
        # check that the car emits a "crashed" signal with the wall as aparameter
	yield( assert_that( _car).within_millis(3000).emits_signal("crashed",  _wall), "completed" )

So basically the assertions API would be extended like this:

class_name GdUnitObjectAssert

...

# Converts an object assert into a timed assert
func within_millis(millis:int = 1000) -> GdUnitTimedAssert:
    pass

class_name GdUnitTimedAssert

# Repeatedly calls a function with the given name and arguments and asserts on the
# result value.
func result_of_call(function_name:String, arg0 = null, arg1=null, ...) -> GdUnitTimedCallAssert:
      pass

# Repeatedly gets the property with the given name and asserts on the property value.
func property(property_name:Strin) -> GdUnitTimedPropertyAssert
     pass

# yields until the signal is emitted and fails if it is not emitted within the time frame
func emits_signal(signal_name:String, arg0 = null, arg1 = null, ...)  -> GdScriptFunctionState:
      pass

class_name GdUnitTimedCallAssert 

# yields until the call returns true or a timeout occurs
func is_true() -> GdScriptFunctionState:
    pass

# yields until the call returns false or a timeout occurs
func is_false() -> GdScriptFunctionState:
    pass

# yields until the call returns a value that is equal to the given one.
func is_equal_to(value) -> GdScriptFunctionState:
    pass

class_name GdUnitTimedPropertyAssert

#basically the same as GdUnitTimedCallAssert but checks a property instead of calling a function

I would volunteer to write a proof of concept implementation of this and submit it as a pull request, but since this is a bit more work, I would first like to present the idea to you and flesh it out before spending a lot of time on coding this, only to have it rejected. What do you think?

[MikeSchulze]

Hi when i correct understand than you have to use a spy to verify a certain function is called with a expected set of arguments.
Take a look at MenuDemo2D at https://github.com/MikeSchulze/gdUnit3-examples/blob/59bc2b8785526ed83b4b03a8a6c61e364984b9e2/MenuDemo2D/test/GameSpyTest.gd.

The missing pice here is a verify_wait_until function on the given spy to wait for an function call or signal.
e.g. verify_wait_until(<spy>, <called times>, <timeout>).<function>

Test timeouts can be set right now by using the arguemt timeout as argumrnt on the test;
func test_foo(timeout=2000):
https://github.com/MikeSchulze/gdUnit3/wiki/advanced_testing#control-test-execution-time

The test could be look lke

func test_i_can_start_a_new_game(timeout=3000):
	# when
          var spy = spy(_game_driver.new_game_dialog)
         _game_driver.main_menu.press_new_game()
         	
	# then
	yield(get_tree().create_timer(<time>), "timeout")
        verify(spy).set_present(true)
        assert_that(_game_driver.new_game_dialog.is_present()).is_true()

And for signal should be duable aktually by

func test_i_can_start_a_new_game(timeout=3000):
	# when
         var spy = spy(_car)
         syp.drive_towards(_wall)
         	
	# then
        yield(get_tree().create_timer(<time>), "timeout")
        verify(spy ).emit_signal("crashed", _wall)

An example can be find here https://github.com/MikeSchulze/gdUnit3-examples/blob/59bc2b8785526ed83b4b03a8a6c61e364984b9e2/SignalsTestExamples/test/ExampleSpyWithSignalTest.gd

The examples can be installed by open the tools from the inspector and pess the "Install Examples" button.

[derkork]

Thank you for writing up a suggestion. Using a spy will not work for this. Let me quickly share the implementation of the new game dialog driver:

class_name NewGameDialogDriver
extends DriverBase

func is_present() -> bool:
	var root = _root()
	return root != null && root.name == "NewGameDialog"

So this driver basically inspects the node hierarchy and checks if a certain node is there. So there is no setter for is_present, therefore the spy has nothing it could intercept. Basically I could write up what I need like this:

func test_i_can_start_a_new_game():
	# when
	_game_driver.main_menu.press_new_game()
	
	# then
        # wait 3 seconds
        yield(get_tree().create_timer(3000), "timeout")
        # check that the dialog is visible
	assert_that( _game_driver.new_game_dialog.is_present() ).is_true()

That would work, however it is inefficient, because I wait the full 3 seconds even if the menu switch only takes 100ms. So what I want would be more like:

func test_i_can_start_a_new_game():
	# when
	_game_driver.main_menu.press_new_game()
	
	# then
        # make me a timer
        var timer = auto_free(Timer.new())
        # start it
        timer.start(3000)

        # within the next 3 seconds check if the dialog appears, every frame
        while timer.time_left > 0:
            # is the dialog visible? 
            if _game.driver.new_game_dialog.is_present():
                  break
            # else wait a frame
            yield(get_tree(), "idle_frame")
        
       # check that we didn't run in a timeout and that the dialog is now visible
       assert_that( timer.time_left > 0 and _game_driver.new_game_dialog.is_present()).is_true()

This would make the test only run for as long as the dialog actually needs to materialize but it is extremely verbose and makes the test both hard to write and hard to read.

Hence my idea of having an assert that basically does this loop:

func test_i_can_start_a_new_game():
	# when
	_game_driver.main_menu.press_new_game()
	
	# then
        # this will run the loop and either terminates after 3 seconds or when is_present returns true
	yield( assert_that( _game_driver.new_game_dialog ) \
            .within_millis(3000).result_of_call("is_present").is_true(), "completed" )

[derkork]

I got a first version of this running locally and made it even more readable by adding an assert_within_millis helper method:

func test_i_can_start_a_new_game():
	# when
	_game_driver.main_menu.press_new_game()
	
	# then
	yield( assert_within_millis(3000).return_value_of( _game_driver.new_game_dialog, "is_present" ).is_true(), "completed")

Checking a signal is equally readable and doesn't require an extra line of code to trigger the runner or create a spy:

func test_scene_changes_are_announced():
	# when
	_game_driver.main_menu.press_new_game()

	# then
	yield(assert_within_millis(3000).that_object(Global).sent_signal("scene_changed"), "completed")

[MikeSchulze]

Gotcha ;)

Ok now i fully understand the use case.
At least you want to automatically wait for a specific program state e.g. a value or function has some value, and a signal is emitted.

Yes this makes fully sense ;)

I would add a wait_until(<time to max wait>) to the all assert to be available for all types. And for signal only for object_asserts.

# to test on a function
assert_func(_instance, "get_value").wait_until(<timeout>).is_equal(10)


# to wait for a signal
assert_signal(_instance).wait_until(<timeout>).is_emitted(<signal_name>, [args]);

Give me time to verify if this works within this suggested syntax.

[MikeSchulze]

Got a first hack running

[MikeSchulze]

created feature request
#154
#155

[MikeSchulze]

@derkork feel free to play around #159
please give me a feedback

[derkork]

I have added some comment there. 😃

[MikeSchulze]

Hi @derkork i pushed the new asserts to master branch (v1.0.6) not offical released yet.
It was a hard work to achive it ;)

short test example
yield(assert_func(instance, "get_value").wait_until(3000).is_equal("foo"), "completed")

yield(assert_signal(instance).is_emitted("test_signal_counted", [20]), "completed")

for more details how to use on unit tests
https://github.com/MikeSchulze/gdUnit3/blob/master/addons/gdUnit3/test/asserts/GdUnitFuncAssertImplTest.gd
https://github.com/MikeSchulze/gdUnit3/blob/master/addons/gdUnit3/test/asserts/GdUnitSignalAssertImplTest.gd
and for scene runner
https://github.com/MikeSchulze/gdUnit3/blob/master/addons/gdUnit3/test/core/GdUnitSceneRunnerTest.gd

Have fun to test, your feedback is welcome

[MikeSchulze]

@derkork can you give me please some feedback?
I want to release v1.06. and it would be nice if you give me feedback if the new assets cover all your needs.

Best Regards

ps. Ich sehe gerade du kommst ja auch aus Leipzig ;)

[derkork]

Hi, so i gave this a spin and it seems to work quite well for signals.

Maybe some small improvement. If you have a typo in your func name, the assert_func will fail but it will actually not tell you that the method you are trying to access doesn't even exist. So it would be nice if the function would check this and fail quickly same as the assert_signal function does. Also if the assert_func fails, it would be nice if it would output the value the function had last before it failed.

Also I wasn't able to make this test green:

class_name SomeTest
extends GdUnitTestSuite

var count = 7

func get_narf():
	return count

	
func test_something_else() -> void	:
	var tween:Tween = auto_free(Tween.new())
	add_child(tween)
	
	tween.interpolate_property(self, "count", 0, 20, 2 )
	tween.start()
	
	yield(assert_func(self, "get_narf").wait_until(4500).is_equal(20), "completed")
	

[MikeSchulze]

Hi, thanks i will check it. give me some minutes ;)

[MikeSchulze]

Hm the tests fails on master branch with
line 17: Expected: is equal '20' but timed out after 4s 500m

Ok the problem is the type of count. if you change to var count := 7 than it works.
I need to investigate why this happens, seems to be a problem with the typing of values.

So it is a test setup error, you have to write is_equal(20.0) because a tween interpolates on float values and needs to compare at least against a float value

So it would be nice if the function would check this and fail quickly same as the assert_signal function does.
--good catch, i will add it--

fixed

You can reach me directly on Discord @nullpointer