Callback types (Discussion: was: Generic interface for arrays (eg:wgpuDeviceEnumerateFeatures(WGPUDevice device, WGPUFeatureName * features))

[zokrezyl]

Hi,

I am trying to write some (partially) automated binding generators but functions like wgpuDeviceEnumerateFeatures are creating a bit of a headache as it is difficult to reason about them and to expose them.

I would like to suggest a type specific array type or generic interface for the arrays maybe inspired by

https://troydhanson.github.io/uthash/utarray.html

Example:

size_t wgpuDeviceEnumerateFeatures(WGPUDevice device, WGPUFeatureName * features))

Would become

size_t wgpuDeviceEnumerateFeatures(WGPUDevice device, WGPUFeatureNameArray *features))

// where
typedef struct WGPUFeatureNameArray {
WGPUFeatureName *array;
size_t size;
} WGPUFeatureNameArray;

One could expose some C like iterators for the xxxArray types.

This would be easier to expose as binding than a random pointer that in specific situations means C array.

Opinions?

Regards

[zokrezyl]

P.S.

Am not sure if y

Same applies to userdata for callbacks. Would be nice to have a wrapper type or typedef instead of void*. For instance

typedef void (*WGPUInstanceRequestAdapterCallback)(
     WGPURequestAdapterStatus status, 
     WGPUAdapter adapter, 
     char const * message,
     WGPU_NULLABLE void * userdata) 
           WGPU_FUNCTION_ATTRIBUTE;

would be

typedef void (*WGPUInstanceRequestAdapterCallback)(
     WGPURequestAdapterStatus status, 
     WGPUAdapter adapter, 
     char const * message, 
     WGPUUserData userdata) WGPU_FUNCTION_ATTRIBUTE;

// and

WGPU_EXPORT void wgpuInstanceRequestAdapter(
     WGPUInstance instance, 
     WGPU_NULLABLE WGPURequestAdapterOptions const * options, 
     WGPUInstanceRequestAdapterCallback callback, 
     WGPU_NULLABLE WGPUUserData userdata) WGPU_FUNCTION_ATTRIBUTE;
    

That signals the automated processor that it can automate the callback mechanism.

It would then implement itself callback like

void __binding_WGPUInstanceRequestAdapterCallback(
   WGPUAdapter adapter, 
   char const * message, 
   WGPUUserData userdata) {
       ((BindingType*)userdata)->callback("WGPUInstanceRequestAdapterCallback",   WRAP(adapter),  WRAP(message))     
   }
   

and call in the language wrapper context wgpuInstanceRequestAdapter with the callback above

[kainino0x]

We've removed *EnumerateFeatures in favor of *GetFeatures which uses the same pattern you described (returns a struct with pointer+size, and has a function to free it):

webgpu-headers/webgpu.h

Lines 5741 to 5747 in 81cc2bf

	/**
	* Get the list of @ref WGPUFeatureName values supported by the device.
	*
	* @param features
	* This parameter is @ref ReturnedWithOwnership.
	*/
	WGPU_EXPORT void wgpuDeviceGetFeatures(WGPUDevice device, WGPUSupportedFeatures * features) WGPU_FUNCTION_ATTRIBUTE;

webgpu-headers/webgpu.h

Lines 2914 to 2926 in 81cc2bf

	/**
	* Default values can be set using @ref WGPU_SUPPORTED_FEATURES_INIT as initializer.
	*/
	typedef struct WGPUSupportedFeatures {
	/**
	* Array count for `features`. The `INIT` macro sets this to 0.
	*/
	size_t featureCount;
	/**
	* The `INIT` macro sets this to `NULL`.
	*/
	WGPUFeatureName const * features;
	} WGPUSupportedFeatures WGPU_STRUCTURE_ATTRIBUTE;

webgpu-headers/webgpu.h

Lines 6006 to 6009 in 81cc2bf

	/**
	* Frees array members of WGPUSupportedFeatures which were allocated by the API.
	*/
	WGPU_EXPORT void wgpuSupportedFeaturesFreeMembers(WGPUSupportedFeatures supportedFeatures) WGPU_FUNCTION_ATTRIBUTE;

---

For the callbacks, would WGPUUserData just be a typedef for void*? I'm not sure that's really clearer because the reader has to know what type WGPUUserData is.

We can change the yaml definition to include whatever higher level information you need to generate bindings, although we already describe callbacks at a highish level (the yaml doesn't even define the userdatas, they are just in the template for the C header):

webgpu-headers/webgpu.yml

Lines 3357 to 3375 in 81cc2bf

	- name: request_adapter
	doc: |
	TODO
	style: callback_mode
	args:
	- name: status
	doc: |
	TODO
	type: enum.request_adapter_status
	- name: adapter
	doc: |
	TODO
	type: object.adapter
	passed_with_ownership: true
	- name: message
	doc: |
	TODO
	type: out_string
	passed_with_ownership: false

[kainino0x]

The main version of Dawn already implements *GetFeatures and is getting pretty close to caught up with the version of the header in this repo. But I'm not sure of the status of wgpu-native, I don't think it has been kept up to date recently.

Emscripten's implementation is frozen right now but we are working on it in a fork inside Dawn (https://source.chromium.org/chromium/chromium/src/+/main:third_party/dawn/src/emdawnwebgpu/)

[zokrezyl]

Happy to hear that I have the same thinking about the C arrays in context of webgpu-native and it is already taken care of.

Regarding callbacks: typedef would be the minimum, the more meta info the better, at least for my use-case, which I hope,
soon will be of a great interest for all.

Why this?

I have been reflecting if I should use the yaml or header for my use-case. I decided to go for the harder part for two reasons:
-> from compiler perspective a cleaner source of thruth is still the header, though from your perspective it is the yaml file (macros, types. etc). The solution I am working on involves equal or even (paradoxically) less effort to process the header than the yaml.
-> Still very early to talk about, but I am working on a generic solution to generate bindings from C header/source files. This solution is supposed to work with other C libraries as well. Hope I can publish the solution very soon. My ambition is to use the same mechanism to generate generic and minimalistic C++ bindings as well based on simple naming and function first argument types.

Long story short the solution is using compiler generated AST, but unfortunatelly for function prototypes the argument name like "userdata" is not exposed. If a specific library/header (in this case webgpu) exposes certain construct class (in this case callbacks) with a specific rule, one can easily extend "my solution" to generate specific binding extension that implement that class of construct.

An alternative and more elegant solution for the callback (in my opinion) would be to have

typedef WGPUInstanceRequestAdapterCallbackType
   /* callback function and userdata are a semantically unseparable tupple/pair */
   WGPUInstanceRequestAdapterCallback function;  // the original function prototype
   UserData userdata;
 }WGPUInstanceRequestAdapterCallbackType;
 
 // UserData can be void * or whatever, but for abstract data type modeling clarity sake should have a typedef
 
WGPU_EXPORT void wgpuInstanceRequestAdapter(
     WGPUInstance instance, 
     WGPU_NULLABLE WGPURequestAdapterOptions const * options, 
     WGPUInstanceRequestAdapterCallbackType callback) WGPU_FUNCTION_ATTRIBUTE;

In such situation then an automated binding generator would have a rule to detect "CallBackType" and generate the binding accordingly.

Thanks for your time
Z

[zokrezyl]

Changed the title, as the original problem is fixed, but I used the same issue to write another suggestion. Hope it is ok

[kainino0x]

That's OK! It's basically a discussion thread for now :)

-> Still very early to talk about, but I am working on a generic solution to generate bindings from C header/source files. This solution is supposed to work with other C libraries as well. Hope I can publish the solution very soon. My ambition is to use the same mechanism to generate generic and minimalistic C++ bindings as well based on simple naming and function first argument types.

Since every C API uses its own conventions, realistically this means you're going to end up doing a ton of custom work for each API anyway, and APIs that are more stable than ours are not going to be able to change to accommodate it. In practice in languages like Swift and Zig, you can either have a fully automatic header translation that's very C-like (and unidiomatic for the target language) or you can produce more custom bindings either manually or using higher level info (like vk.xml or webgpu.yml).

An alternative and more elegant solution for the callback (in my opinion) would be to have

I am not sure if you're looking at the latest version of our header (the latest version in this repo - which is the target for implementations, but no implementation matches it yet).

Anyway, this suggestion is also similar to what we have already done:

typedef void (*WGPURequestAdapterCallback)(WGPURequestAdapterStatus status,
    WGPUAdapter adapter, WGPUStringView message,
    WGPU_NULLABLE void* userdata1, WGPU_NULLABLE void* userdata2) WGPU_FUNCTION_ATTRIBUTE;

typedef struct WGPURequestAdapterCallbackInfo {
    WGPUChainedStruct const * nextInChain;
    WGPUCallbackMode mode;
    WGPURequestAdapterCallback callback;
    WGPU_NULLABLE void* userdata1;
    WGPU_NULLABLE void* userdata2;
} WGPURequestAdapterCallbackInfo WGPU_STRUCTURE_ATTRIBUTE;

This structure is used consistently and may be enough for you to identify the userdatas without looking at the field names.

[zokrezyl]

Yep, was a trap. Was looking further at the wgpu webgpu.h headers. Lessons learned. Thanks for your time

Closing this then.

[kainino0x]

No problem!