Issue when running Godot 4 + GUT 9.0.1 in CI/headless mode / Invalid call. Nonexistent function / GutRunner.gd:112

[char32]

Hi,

I am seeing a strange issue with GUT 9.0.1 and Godot 4.

I have the following structure:
GUT: 9.0.1
Godot: 4.0.2
res://test/test_example.td contains my basic tests.
command line: Godot_v4.0.2-stable_linux.x86_64 -v --headless -s addons/gut/gut_cmdln.gd -gtest=res://test/test_example.gd -glog=3 -gexit

When I run that locally on my PC, everything succeeds.
But when I run the same command on my Godot CI pipeline I get:

$ godot -v --headless -s addons/gut/gut_cmdln.gd -gtest=res://test/test_example.gd -glog=3 -gexit
Godot Engine v4.0.2.stable.official.7a0977ce2 - https://godotengine.org/
TextServer: Added interface "Dummy"
TextServer: Added interface "ICU / HarfBuzz / Graphite (Built-in)"
Using "default" pen tablet driver...
JoypadLinux: udev enabled and loaded successfully.

TextServer: Primary interface set to: "ICU / HarfBuzz / Graphite (Built-in)".
CORE API HASH: 417986958
EDITOR API HASH: 95126707
Loaded builtin certs
SCRIPT ERROR: Parse Error: Could not find type "GutHookScript" in the current scope.
at: GDScript::reload (res://addons/gut/gut.gd:477)
--- Gut ---
SCRIPT ERROR: Invalid call. Nonexistent function 'new' in base 'GDScript'.
at: get_gut (res://addons/gut/gui/GutRunner.gd:112)
SCRIPT ERROR: Invalid call. Nonexistent function 'connect' in base 'Nil'.
at: _run_gut (res://addons/gut/gut_cmdln.gd:278)

The whole process freezes then.
Before calling that I have a import-asset stage that opens the editor once, so that this cannot cause an issue.

This could as-well be a bug in Godot itself. I had suspected godotengine/godot#73843 but renaming the Gut class to e.g. GutMain did not help.

Oh, and just displaying the GUT help works:
Godot_v4.0.2-stable_linux.x86_64 -v --headless -s addons/gut/gut_cmdln.gd -gh
So I expect the command is correct.

[bitwes]

From the log it looks like gut.gd isn't loading because GutHookScript isn't found. Since it doesn't load, GutRunner.gd blows up when it tries to make an instance. The only time I've seen something that like that is when Godot hasn't found all the class_name entries, which I would have though would be fixed by your import-asset stage.

gut.gd doesn't have a class_name, I've used them very sparingly. GutHookScript and GutTest might be the only two. When you do your import-asset stage, is the GUT plugin enabled? I'm not sure how you could do that with code (probably somewhere in the project settings), but if you don't use the editor to test, then the plugin doesn't have to be enabled. If it is enabled, then maybe Godot will find the GutHookScript?

This feels more like a Godot issue than GUT, but just hearing about people using GUT in their CI pipelines makes me happy...so I'll help wherever I can.

[bitwes]

I'm not sure if this would make any difference at all, but I have recently pushed up changes to godot_4 that make it easier to create a scene to run your tests. I thought maybe if you run a scene instead of using the command line it might make a difference.

You can tell Godot to run a specific scene by giving it a path to a scene instead of using the -s option. You'll have to call run_tests on the GutControl you add to the scene. You should do this after _ready, so in _post_ready_setup in the example code.

You can find the example code and more info here: https://github.com/bitwes/Gut/blob/godot_4/wiki/Running-On-Devices.md.

Once this is released the wiki will be updated.

[char32]

Hi,

sorry for answering late. First of all: Yeah I use GUT already a long time for my projects with Godot 3.x and CI Pipeline on Gitlab.
Really great work you did! Really appreciate it!

I tried it with both: GUT enabled an disabled in the Editor. But that does not do any difference so far.
What I also tried is to disable the import step. But that gives me some loader errors (I guess - I have to look up the exact error). But I think that is expected when the import is missing :-)

I did also try to update to a0513af, which changes the error to:

SCRIPT ERROR: Parse Error: Identifier "GutUtils" not declared in the current scope.
at: GDScript::reload (res://addons/gut/gut.gd:115)

Just for reference, I made a basic Gitlab CI script to reproduce the issue: gitlab-ci.yml.txt

Thank you very much for your help! I'll try that! I am just not sure if Godot will allow to run a scene in headless mode.

[bitwes]

SCRIPT ERROR: Parse Error: Identifier "GutUtils" not declared in the current scope.
at: GDScript::reload (res://addons/gut/gut.gd:115)

That makes sense. That commit introduced GutUtils as a class_name, so I think it's still the same issue, just with a different class_name that is encountered earlier than GutHookScript was.

[char32]

Ok, short update: so I did use the latest godot_4 branch together with the modifications you suggested - and it works!!

I did add the run_tests to _post_ready_setup() and did run the godot with godot --headless my_unit_test_scene.tscn and that works just fine. On the script Pipeline I still get the above script error, but it seems that this is a Godot 4 issue.
I tracked it down to the .godot folder. If that one is generated on the command line (like on a CI pipeline), it is incomplete. Only the GUI editor does correctly create the cache.
I created a issue for that in the Godot repository godotengine/godot#75684

Thanks again for your help!

[98teg]

I also encountered this issue. I tried simply uploading .godot/global_script_class_cache.cfg as a workaround for the already mentioned Godot issue. But I still got some errors such as:

ERROR: Cannot open file 'res://.godot/imported/AnonymousPro-Regular.ttf-856c843fd6f89964d2ca8d8ff1724f13.fontdata'.

So I also uploaded the .godot/imported folder. This is how my .gitignore file looks right now:

#################################################################
# TODO: Replace with .godot when the following issue is solved: #
# https://github.com/godotengine/godot/issues/75684             #
demo/.godot/editor
demo/.godot/shader_cache
#################################################################

I'm mentioning the font error since the Godot issue does not refer to it and fonts don't make sense in headless mode. Maybe GUT itself can solve this issue?

[char32]

@98teg
Hey,

I encountered that too. I found a way to prevent the upload of the .godot folder and .gitignore adjustment.

It seems like there is a bug in Godot 4 that quits the editor too early when resources are imported in --headless mode (godotengine/godot#69511).
The referenced pull-requests pointed in direction of a timing issue: godotengine/godot#68461

So essentially what I do in my pipeline is:

godot -e --headless & sleep 30 && kill -9 $(pgrep -o godot)

And then

godot --headless --quit scenes/unit_tests.tscn 2>&1 | tee $TEST_LOG_FILE

So I start in --headless and --editor mode and do not let godot exit after a first iteration (usually with --quit) but I manually kill the editor after 30 seconds.
Then the import is successful and the unit tests run fine.

[ngburke]

@char32 @98teg

As an FYI, I use the following for my CI pipeline to ensure that assets are imported without having to do the sleep/kill method (which can be fragile):

godot --headless --export-release "Linux/X11" /dev/null

Hope it helps you guys, give it a try. And oh yeah, GUT rules!

[char32]

@ngburke
Thanks! That works even better than the sleep/kill method!

[bitwes]

I'm closing this out as it doesn't appear to be a GUT issue and some workarounds have been found.