Can't build against latest gdnative headers in godot repo

[garyo]

I don't know if this is supposed to work, but I tried building godot-python using the latest 3ad9e4740 godot, using its gdnative headers and wrapper lib directly, and it doesn't build on Windows.
Here's the scons command I used:
scons -j16 platform=windows-64 backend=cpython godot_binary=../godot/bin/godot.windows.opt.tools.64.exe MSVC_USE_SCRIPT=True gdnative_include_dir=../godot/modules/gdnative/include gdnative_wrapper_lib=../godot/bin/gdnative_wrapper_code.windows.opt.tools.lib
The errors are these:

link /nologo /dll /out:pythonscript\pythonscript.dll /implib:pythonscript\pythonscript.lib /LIBPATH:platforms\windows-64\cpython\PCBuild\amd64 python36.lib ../godot/bin/gdnative_wrapper_code.windows.opt.64.lib pythonscript\pythonscript.obj pythonscript\cffi_bindings.gen.obj
cffi_bindings.gen.obj : error LNK2019: unresolved external symbol godot_nativescript_set_method_argument_information referenced in function _cffi_start_python
cffi_bindings.gen.obj : error LNK2019: unresolved external symbol godot_nativescript_set_class_documentation referenced in function _cffi_start_python
cffi_bindings.gen.obj : error LNK2019: unresolved external symbol godot_nativescript_set_method_documentation referenced in function _cffi_start_python
cffi_bindings.gen.obj : error LNK2019: unresolved external symbol godot_nativescript_set_property_documentation referenced in function _cffi_start_python
cffi_bindings.gen.obj : error LNK2019: unresolved external symbol godot_nativescript_set_signal_documentation referenced in function _cffi_start_python
cffi_bindings.gen.obj : error LNK2019: unresolved external symbol godot_nativescript_set_global_type_tag referenced in function _cffi_start_python
cffi_bindings.gen.obj : error LNK2019: unresolved external symbol godot_nativescript_get_global_type_tag referenced in function _cffi_start_python
cffi_bindings.gen.obj : error LNK2019: unresolved external symbol godot_nativescript_set_type_tag referenced in function _cffi_start_python
cffi_bindings.gen.obj : error LNK2019: unresolved external symbol godot_nativescript_get_type_tag referenced in function _cffi_start_python
cffi_bindings.gen.obj : error LNK2019: unresolved external symbol godot_nativescript_register_instance_binding_data_functions referenced in function _cffi_start_python
cffi_bindings.gen.obj : error LNK2019: unresolved external symbol godot_nativescript_unregister_instance_binding_data_functions referenced in function _cffi_start_python
cffi_bindings.gen.obj : error LNK2019: unresolved external symbol godot_nativescript_get_instance_binding_data referenced in function _cffi_start_python
pythonscript\pythonscript.dll : fatal error LNK1120: 12 unresolved externals```

It builds fine without using the `gdnative_include_dir=...` arg, so feel free to close this issue if this isn't expected to work.

[touilleMan]

I feel like I already encountered this unresolved externals error a while ago...
godotengine/godot#12849 seems to suggest this is due to a visual studio update.

Out of this, doing a diff -r godot-python/platforms/gdnative/include/ godot/modules/gdnative/include/ | grep -ve '/\*' | less shows really few changes between Godot-python's and Godot 3.0.2's gdnative includes.
I've build with problems Godot-python with Godot 3.0.2 include headers on linux so my guess is it's a toolchain issue.

[garyo]

I did that same diff (vs. godot master) and I see the exact struct that's giving the unresolved refs:

diff -w platforms/gdnative/include/gdnative_api_struct.gen.h ../godot/modules/gdnative/include/gdnative_api_struct.gen.h
> typedef struct godot_gdnative_ext_nativescript_1_1_api_struct {
>       unsigned int type;
>       godot_gdnative_api_version version;
>       const godot_gdnative_api_struct *next;
>       void (*godot_nativescript_set_method_argument_information)(void *p_gdnative_handle, const char *p name, const char *p_function_name, int p_num_args, const godot_method_arg *p_args);
>       void (*godot_nativescript_set_class_documentation)(void *p_gdnative_handle, const char *p_name, g dot_string p_documentation);
>       void (*godot_nativescript_set_method_documentation)(void *p_gdnative_handle, const char *p_name,
onst char *p_function_name, godot_string p_documentation);
>       void (*godot_nativescript_set_property_documentation)(void *p_gdnative_handle, const char *p_name  const char *p_path, godot_string p_documentation);
>       void (*godot_nativescript_set_signal_documentation)(void *p_gdnative_handle, const char *p_name,
onst char *p_signal_name, godot_string p_documentation);
>       void (*godot_nativescript_set_global_type_tag)(int p_idx, const char *p_name, const void *p_type_ ag);
>       const void *(*godot_nativescript_get_global_type_tag)(int p_idx, const char *p_name);
>       void (*godot_nativescript_set_type_tag)(void *p_gdnative_handle, const char *p_name, const void * _type_tag);
>       const void *(*godot_nativescript_get_type_tag)(const godot_object *p_object);
>       int (*godot_nativescript_register_instance_binding_data_functions)(godot_instance_binding_functio s p_binding_functions);
>       void (*godot_nativescript_unregister_instance_binding_data_functions)(int p_idx);
>       void *(*godot_nativescript_get_instance_binding_data)(int p_idx, godot_object *p_object);
> } godot_gdnative_ext_nativescript_1_1_api_struct;

So I'm thinking it has to do with this new 1.1 API struct.

[touilleMan]

Yep, it seems I've discarded the API addition too quickly when looking the diff ^^
Have you tried to replace the current includes in platforms/GDnative/include by the up to date one from Godot 3.0.2 ?
Not this is something we want to do (because it would mean we are no longer compatible with Godot 3.0), but it would narrow the trouble

[touilleMan]

Ho ! I reread your original issue and realised the trouble actually comes from what I suggest !
We'll want you can do is use the Godot 3.0 headers but link against the 3.0.2 wrapper lib.

Anyway this link error is a bit sad because the whole point of GDnative API being divided in pointer-linked structures was to allow painless backward compatibility :'-(

[garyo]

So what's the next step for this? Is there something core godot should do to prevent the new header functions from "leaking" into the API? Or should gdnative do something to account for the new 1.1 functions? I don't understand the GDnative structure well enough to know where it should be fixed (if anywhere).

[touilleMan]

I've try to compile the project and run the tests on linux with godot 3ad9e4740 headers and wrapper lib:

~/projects/godot-python-release master *23461 ♦ scons gdnative_include_dir=$HOME/projects/godot/modules/gdnative/include gdnative_wrapper_lib=$HOME/projects/godot/bin/libgdnative_wrapper_code.x11.tools.64.a  test 
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
clang -o pythonscript/pythonscript.os -c -m64 -DBACKEND_CPYTHON -I platforms/x11-64/cpython_build/include/python3.6m/ -fcolor-diagnostics -fPIC -I. -I/home/emmanuel/projects/godot/modules/gdnative/include pythonscript/pythonscript.c
. tools/venv/bin/activate && python ./tools/generate_gdnative_cffidefs.py /home/emmanuel/projects/godot/modules/gdnative/include --output=pythonscript/cdef.gen.h --bits=64 --cpp="cpp"
. tools/venv/bin/activate && python ./pythonscript/generate_cffi_bindings.py --cdef=pythonscript/cdef.gen.h --output=pythonscript/cffi_bindings.gen.c
generating pythonscript/cffi_bindings.gen.c
clang -o pythonscript/cffi_bindings.gen.os -c -m64 -DBACKEND_CPYTHON -I platforms/x11-64/cpython_build/include/python3.6m/ -fcolor-diagnostics -fPIC -I. -I/home/emmanuel/projects/godot/modules/gdnative/include pythonscript/cffi_bindings.gen.c
clang -o pythonscript/libpythonscript.so -m64 -Wl,-rpath,'$ORIGIN/lib' -shared pythonscript/pythonscript.os pythonscript/cffi_bindings.gen.os -Lplatforms/x11-64/cpython_build/lib -lpython3.6m /home/emmanuel/projects/godot/bin/libgdnative_wrapper_code.x11.tools.64.a
Generating build dir build/x11-64-cpython from platforms/x11-64/cpython_build pythonscript/libpythonscript.so pythonscript/embedded/godot pythonscript/embedded/godot/__init__.py pythonscript/embedded/godot/aabb.py pythonscript/embedded/godot/array.py pythonscript/embedded/godot/basis.py pythonscript/embedded/godot/bindings.py pythonscript/embedded/godot/color.py pythonscript/embedded/godot/dictionary.py pythonscript/embedded/godot/globals.py pythonscript/embedded/godot/node_path.py pythonscript/embedded/godot/plane.py pythonscript/embedded/godot/pool_arrays.py pythonscript/embedded/godot/quat.py pythonscript/embedded/godot/rect2.py pythonscript/embedded/godot/rid.py pythonscript/embedded/godot/transform.py pythonscript/embedded/godot/transform2d.py pythonscript/embedded/godot/vector2.py pythonscript/embedded/godot/vector3.py
Symlinking build/x11-64-cpython -> build/main
platforms/x11-64/godot.x11.opt.debug.64 --path /home/emmanuel/projects/godot-python-release/tests/bindings
OpenGL ES 3.0 Renderer: Mesa DRI Intel(R) HD Graphics 520 (Skylake GT2) 
ERROR: open_dynamic_library: Can't open dynamic library: /home/emmanuel/projects/godot-python-release/tests/bindings/pythonscript/libpythonscript.so. Error: /home/emmanuel/projects/godot-python-release/tests/bindings/pythonscript/libpythonscript.so: undefined symbol: godot_nativescript_register_instance_binding_data_functions
   At: drivers/unix/os_unix.cpp:373.
ERROR: _load: No loader found for resource: res://main.py
   At: core/io/resource_loader.cpp:186.
ERROR: poll: res://main.tscn:3 - Parse Error: [ext_resource] referenced nonexistent resource at: res://main.py
   At: scene/resources/scene_format_text.cpp:439.
ERROR: load: Condition ' err != OK ' is true. returned: RES()
   At: core/io/resource_loader.cpp:149.
ERROR: _load: Failed loading resource: res://main.tscn
   At: core/io/resource_loader.cpp:186.
ERROR: start: Failed loading scene: res://main.tscn
   At: main/main.cpp:1672.
WARNING: cleanup: ObjectDB Instances still exist!
   At: core/object.cpp:1989.
scons: done building targets.

So build is fine, but running complains godot_nativescript_register_instance_binding_data_functions is undefined. This is because scons test rule download godot binary from the internet, so we are running the project with Godot 3.0.0 (which obviously don't contain the gdnative 1.1 new api).
If I run the project with Godot build from 3ad9e4740 everything is fine.

The explanation of this is during build, gdnative_include_dir is used to parse the Godot GDnative include headers and generate cffi_bindings.gen.c. Currently there is no way to prevent cffi_bindings.gen.c from containing all the GDnative API (including the newer GDnative extensions).

[touilleMan]

So in the end I guess we should close this issue and accept we should stick with the Godot 3.0 headers that have been copied in the project, and which will be updated if we decide to support further API changes.

Obviously this means we couldn't provide optional support for the GDnative api extensions, but I would rather wait for the need to come before doing something about this ;-)

[aaronfranke]

Is this still the case with Godot 3.2.3 headers? I'm having the same problem, there's a lot of symbols it can't resolve.

[touilleMan]

Hi @aaronfranke ;-)

there's a lot of symbols it can't resolve.

Is it during the runtime of a project using Godot-Python or during compilation of the Godot-Python plugin ?
Can you provide the version of Godot, Godot-Python and godot_headers you were using ? (and also some logs of your issues)

Godot-Python architecture and Godot GDnative have changed quiet a bit since this issue:

• the gdnative_wrapper_code library has been deprecated and is no longer used (this issue was complaining about this particular)
• Godot-Python generates it bindings from the gdnative_api.json and api.json files provided by the godot_headers folder (before that, their was some code generated by hand so there could be some desynchronization between godot_headers and Godot-Python codebase)

So in theory compilation should run fine (except if you try to compile against a very old version of godot_headers !)

For the runtime part, there is no check in Godot-Python to ensure the version of Godot running is compatible with the godot_headers that have been used (this is something we may want to add to avoid this kind of issues...)

As you've noticed the project relies on Godot 3.2.3 headers (godot_headers folder is a git submodule on https://github.com/godotengine/godot-headers/tree/815f34e1e96c09122449105c55aba501654da029).

So if you use another version of Godot you might encounter hiccups. If you are compiling Godot-Python, you should try to change the godot_headers code to the correct version.

Note there is there is discussion under way to improve the plugin/godot compatibility, but it most likely won't be available before Godot 4:

• Add a way to specify compatible Godot versions for editor plugins godotengine/godot-proposals#1613
• Proposal for changes in GDNative for Godot 4.0 godotengine/godot#35467 (comment)