API functions with pointer-pointer arguments are not usable in Lua

[bfxdev]

Platform

Operating system and version: Windows 10 Pro 1909
OBS Studio version: 25.0 and 26RC2, 32 bits and 64 bits

All platforms and versions are probably affected.

Expected Behavior

It should be possible to use all arguments of functions such as gs_effect_create or gs_texture_map, which are exported to the obslua module and officially documented.

Current Behavior

The issue affects those functions with pointer arguments that will typically receive the address of an existing buffer or a newly allocated memory area (to be freed later on).

For gs_effect_t *gs_effect_create(const char *effect_string, const char *filename, char **error_string) the error_string will be allocated with the results of the compilation. Only the value nilcan be passed, passing e.g. an empty string results in expected 'char **' got 'string'.

For bool gs_texture_map(gs_texture_t *tex, uint8_t **ptr, uint32_t *linesize), same kind of result, even the result of linesize cannot be retrieved (a simple pointer).

Actually Lua does not support passing a string "by reference" (and strings are anyway not mutable). Either a mutable userdata object could be passed to receive the data, or the function needs to return an additional value.

Steps to Reproduce

Use the provided Lua script to see the currenty behavior when using gs_effect_create on some (faulty on purpose) effect code and trying to provide a variable to retrieve the error_string:

1. When a nil variable is used as argument --> remains nil
2. When expecting a second return value --> remains nil
3. When an empty string is used as argument --> logs error expected 'char **' got 'string'

test-pointer-pointer-arguments.zip

The example is only focusing on gs_effect_create but many other functions are affected.

Additional information

The issue is not new apparently.

There may be a workaround without any change in the OBS codebase by using FFI. I looked for solutions but until now I could not get something working (no experience at all with SWIG and very short experience with Lua).

Here are the alternatives I think are feasible.

Alternative 1: additional return value

The most common solution in Lua seems to be to type-map the parameter as an OUTPUT and expect it as additional return value. There is no standard typemap in typemaps.i for the conversion to a string from a char **, so one needs to be written.

The allocation aspects may play a role as well. The interface cstring.i is not implemented for Lua, too bad because %cstring_output_allocate(parm, release) seems to be the one needed in that case.

Alternative 2: mutable userdata

We could imagine a new C structure mapped to userdata, like other obs_something objects, that can be passed as argument. The structure would hold the pointer to the buffer, and frees it when garbage collected. A type-map would be needed in the interface file anyway.

That would be my preferred solution. At least it would look like a classical usage of a parameter passed by reference. No extra documentation necessary.

Other affected API functions

List of the functions that should be affected by the issue in Lua, according to a search in the build\deps\obs-scripting\obslua\CMakeFiles\obslua.dir\obsluaLUA_wrap.c file, generated during compilation (looking for calls to SWIG_isptrtype with a pointer-pointer type):

gs_create
gs_effect_create_from_file
gs_effect_create
gs_vertexshader_create_from_file
gs_pixelshader_create_from_file
gs_texture_create
gs_cubetexture_create
gs_voltexture_create
gs_vertexshader_create
gs_pixelshader_create
gs_texture_map
gs_stagesurface_map
gs_image_file::animation_frame_cache
obs_data_item_next
obs_data_item_release
obs_data_item_remove
obs_data_item_set_string
obs_data_item_set_int
obs_data_item_set_double
obs_data_item_set_bool
obs_data_item_set_obj
obs_data_item_set_array
obs_data_item_set_default_string
obs_data_item_set_default_int
obs_data_item_set_default_double
obs_data_item_set_default_bool
obs_data_item_set_default_obj
obs_data_item_set_default_array
obs_data_item_set_autoselect_string
obs_data_item_set_autoselect_int
obs_data_item_set_autoselect_double
obs_data_item_set_autoselect_bool
obs_data_item_set_autoselect_obj
obs_data_item_set_autoselect_array
obs_data_get_frames_per_second
obs_data_get_default_frames_per_second
obs_data_get_autoselect_frames_per_second
obs_data_item_set_frames_per_second
obs_data_item_set_default_frames_per_second
obs_data_item_set_autoselect_frames_per_second
obs_data_item_get_frames_per_second
obs_data_item_get_default_frames_per_second
obs_data_item_get_autoselect_frames_per_second
obs_property_next
obs_hotkey_pair_save
obs_hotkey_pair_save
obs_cmdline_args::argv
obs_open_module
obs_enum_source_types
obs_enum_input_types
obs_enum_input_types2
obs_enum_input_types2
obs_enum_filter_types
obs_enum_transition_types
obs_enum_output_types
obs_enum_encoder_types
obs_enum_service_types
obs_get_audio_monitoring_device
obs_get_audio_monitoring_device
obs_scene_insert_group
obs_scene_insert_group2
obs_encoder_get_extra_data
calldata_get_string
signal_handler_add_array
base_get_log_handler
os_fread_mbs
os_fread_utf8
os_mbs_to_wcs_ptr
os_utf8_to_wcs_ptr
os_wcs_to_mbs_ptr
os_wcs_to_utf8_ptr
os_utf8_to_mbs_ptr
os_mbs_to_utf8_ptr
os_glob

[upgradeQ]

@bfxdev I've found a forum post which uses ffi in LuaJIT to enumerate script properties using obs_property_next with double pointer argument, check out this gist (it will print to the console properties names after loading of OBS has been finished)

[bfxdev]

Good find @upgradeQ ! This is exactly the kind of solution I mentioned in the conclusion of my last post. But I was not aware that somebody already did it in OBS. I will try this. Of course it brings us very far from the original obslua bindings.

[MrLixm]

Hello, I'm bumping this old thread because I didn't find a solution to that issue yet.

I there still no way to retrieve the potential error string generated by gs_effect_create_from_file ?

It is really super tough to debug faulty hlsl shaders and I'm looking at anything that could make it easier.
However if the error generated is the the same as the one we can see in the log then it might not be that useful for debugging.