Source of truth

[cerisano]

Just a question about the "source of truth" for webidl-napi, the generator seems to depend on an idl definition, a impl.h and a impl.cc definition. Are all required, or can the impl.h and impl.cc be generated from the idl?

My understanding was that webidl-napi generates the impl base classes which are then subclassed to integrate with native libraries (eg. generates webgpu-impl.cc and webgpu-impl.h containing all N-API wrappers which are then subclassed to integrate with Dawn or other webgpu native implmentations.)

Is it the case that webidl-napi is going even further and referencing the native headers as well (eg. webgpu-native?) That would be pretty staggeringly awesome ...

webidl-napi/test/class/CMakeLists.txt

Line 17 in 085e885

COMMAND node ${REPO_ROOT}/index.js -i class-impl.h -n node_api.h -o ${CMAKE_CURRENT_BINARY_DIR}/class.cc ${CMAKE_CURRENT_SOURCE_DIR}/class.idl

[gabrielschulhof]

The generator assumes that for a WebIDL definition of

interface Something {
  unsigned long someMethod(DOMString someParam);
};

the C++ compiler and linker will be able to resolve the calls

new Something();

and

something->someMethod(someParam);

where something is of type Something* and someParam is of type DOMString.

Currently the compiler is able to resolve them because a -impl.h file, which is hand-written, is given to the generator to #include at the top of the generated code, and the linker is able to resolve them because a -impl.cc, also hand-written, is compiled and linked into the final binary.

The generator could conceivably generate the -impl.h file, but the -impl.cc file would have to actually contain the ties to the native library, hand-written.

[cerisano]

I see. I don't think webidl-napi should need to have ties to the native libraries. It could just generate base classes encapsulating all the N-API bindings.

The native libraries would be integrated as subclasses in separate projects that generate the base classes using webidl-napi, not? For eg. I intend to do exactly that for WebGPU-NAPI (that is where dawn and webidl-napi are included as submodules).

Other projects could use exactly the same webidl-napi superclasses to integrate with other native implementations (eg. wgpu).

[kainino0x]

FYI the way it works in Blink is similar - the headers are handwritten, referencing generated classes, and called into by generated code. This is necessary because we can add private data to the header class definition. I don't think generating superclasses helps that much because the subclass still has to declare every function it's overloading so it would still be verbose.

[cerisano]

If webidl-napi generates the base class headers (impl.h) and NAPI binding stubs (impl.cc), then any private data would be included in the subclasses. It would make webidl-napi the official "source of truth", and allow subclasses (implementations) to overload only the methods they require. Would not have to manually maintain the complete headers or stubs separately in webidl-napi (downstream task).

[kainino0x]

Wouldn't all of the superclass methods be virtual so have to be implemented by the subclass anyway?

[cerisano]

I was thinking many would be stubs that would be called by the subclasses for boilerplate code that is common to N-API bindings. The idea being that webidl-napi could actually generate complete bindings (eg. to Dawn, which in many cases is a straight pass-through - identical signatures in both webidl for WebGPU-API and webgpu-native)

Where it is not a straight pass-through (surprisingly few cases in Dawn, for eg.), a subclass would manually perform the translation from webidl to native sigs.

Note for readership: we are trying to inherit as much N-API magic as possible. A single webidl-napi based implementation of a web api (eg. WebGPU-API addon implemented with Dawn or wgpu) could be loaded into any native platform (nodejs, chromium, moz, electron, BN, etc) and attach to any supported version (ABI stable) of any supported JS engine (V8, JScore, JerryScript, Chakra, etc) on any supported OS (Lin, Win, OSX, mobile, etc).

This would be true for any web api (WebGPU is just the pilot), and completely defined by webidl (source of truth) and enabled by webidl-napi.

[gabrielschulhof]

@cerisano @kainino0x correct me if I'm wrong:

Every implementation of WebGPU (be it Dawn or WGPU) ultimately provides an implementation for the APIs declared in https://github.com/webgpu-native/webgpu-headers/blob/main/webgpu.h. Thus, the job of webgpu-napi would be to provide a header file called, say, webgpu-impl.h and a source filed called, say webgpu-impl.cc which implements the classes in webgpu-impl.h using calls to the APIs defined in https://github.com/webgpu-native/webgpu-headers/blob/main/webgpu.h. APIs declared in webgpu-impl.h and implemented in webgpu-impl.cc look exactly like the WebIDL, but implemented in C++. webidl-napi would generate code that exposes the contents of webgpu-impl.h to JavaScript and calls APIs defined in webgpu-impl.h whenever a call comes in from JavaScript.

So, the triggers are as follows:

• if The WebIDL changes, the code needs to be regenerated, and webgpu-impl.h and webgpu-impl.cc need to both be changed by hand.
• if https://github.com/webgpu-native/webgpu-headers/blob/main/webgpu.h changes
  • if the WebIDL needs to be changed by hand, it triggers the change mentioned in the first point.
  • if the WebIDL does not need to be changed, only webgpu-impl.cc needs to be changed by hand.

[cerisano]

That is also my understanding, with the observation that the regenerated impl.h and impl.cc should serve as base bindings and be as fully formed as possible to reduce the amount of manual coding in any project implementing a web api.

As mentioned in the readme, webidl-napi is intended to be part of a build chain for a web api implementation. It fills in the gap between webidl and a native api header, by generating the napi bindings. The question is how fully formed is the generated code (impl.cc), and can the bindings be complete (no manual coding required) in cases where the signatures allow generated pass-through boilerplate from webidl to the native layer.

Note that in the case of webgpu, the webidl and native header signatures are very similar (because webgpu was always intended to be a web api). In blink there are even utilities for converting from webidl to native (dawn) types. This would likely not be the case for most web apis that bind to native libraries not specifically designed for that.

Perhaps completely fully formed impl.cc bindings are currently out of scope for webidl-napi, but should keep it in mind and avoid anything that might prevent realizing that goal in the future.

[kainino0x]

correct me if I'm wrong:

Yes, all looks correct to me. Only thing I would point out is that - for the purposes of this project - the contents of the impl.cc, and the existence of webgpu.h, don't really matter. We could be exposing global.extraSpecialSquare(x) which just implements x*x in C++.

Perhaps completely fully formed impl.cc bindings are currently out of scope for webidl-napi, but should keep it in mind and avoid anything that might prevent realizing that goal in the future.

I don't see what this project could possibly do that would make it impossible to do that. I think that -impl.h and -impl.cc should be out-of-scope of the webidl-napi generator, and downstream projects like webgpu-napi can create those files however they want, including by auto-generation. (From my experience implementing WebGL and WebGPU in Chromium, it is my opinion in the case of WebGPU, and probably every API, that these files should be handwritten. I don't think trying to generate them buys anything - hand-writing some small, repetitive "pass-through" functions is more maintainable than trying to auto-generating them IMO. And I think you'll find that fewer can be auto-generated than you think. What could be auto-generated is some helper functions to be called from the handwritten -impl.cc, like enum conversions, if that's found to be useful.)

It's important that this project not try to specialize for anything in WebGPU. It definitely should not try to generate any code that would call into a C implementation like webgpu.h. Anything like that must be downstream.

[cerisano]

Please see WebGPU-NAPI repo for an example of what @kainino0x described above. It includes the WebIDL-NAPI compiler as a submodule (which will replace existing very manual binding compiler).

The current inclusion of WebGPU in WebIDL-NAPI is for development purposes only. WebIDL and N-API are perceived as very good integration candidates for each other, and WebGPU is seen as an excellent test bed to complete the integration and ensure the WebIDL-NAPI compiler is comprehesive enough for use with any other Web API.

[kainino0x]

The current inclusion of WebGPU in WebIDL-NAPI is for development purposes only.

Of course, and I agree with using it as a test case, that's not what I was talking about. I'm just saying it does not need to go any farther with auto-generation than it does now.

[cerisano]

Agreed, as I mentioned above that would require generating C-style native headers and certainly out of scope for the current WebGPU test bed, which already has them (and are an excellent example of what a generated web api native header could look like). Mostly curious about this for custom ex-browser WebAPI development in the future.

[gabrielschulhof]

@kainino0x I am purposely avoiding familiarizing myself with how WebGPU works. I'm sticking to the WebGPU IDL and implementing what I think are its semantics, i.e., things like "Well, this says it returns a promise, so let's make it return a promise, and worry about the two cases – the promise is resolved synchronously, and the promise is resolved asynchronously". I'm basically incorporating a lot of the principles I discovered when writing Node.js bindings for pretty much any library using N-API into the generator.

This should keep things native-agnostic.

[gabrielschulhof]

I also agree that the -impl.cc and -impl.h should be written by hand.