Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

$250 - Common HL/HXCPP native C API binding format #2

Open
larsiusprime opened this Issue Oct 3, 2018 · 20 comments

Comments

Projects
None yet
6 participants
@larsiusprime
Copy link
Owner

larsiusprime commented Oct 3, 2018

Introductory Information:

Hashlink is basically the successor Neko in terms of Haxe VM targets, with the added advantage of having both a compiled-C and a bytecode target.

HXCPP is the traditional workhorse of Haxe native C++ targets.

Hashlink promises some advantages to workflow -- particularly faster compilation speed during development, but it's still relatively new and most of the C/C++ bindings for existing libraries for Haxe native targets are still written for HXCPP, and HXCPP is a solid well-maintained target that will surely see continued use alongside HashLink. This puts native extension library maintainers in a dilemma, do we support HXCPP or Hashlink, or both? Right now supporting both means maintaining two parallel sets of bindings, which is not only double the work, but also fertile ground for subtle duplication/oops-I-forgot-to-update-the-other-one errors.

This bounty is for developing a common format for writing native extensions to C API's for Haxe, that is compatible with both HXCPP and HashLink.

This is a separate issue from automatically generating native extensions with minimal effort from, say, raw C files. It's okay for the sake of this bounty if the accepted solution requires a bit of manual labor in writing the bindings themselves, so long as each binding doesn't have to be written twice, once for HXCPP and once for HashLink.

The Project

  • Produce a method that bridges a C API to Haxe
  • The solution should produce both HL bindings and HXCPP bindings
  • The programmer should only have to write an "exposure" for a given C API function once, and get both bindings out of it
  • Provide a working proof of concept example of limited scope
  • Open source the whole shebang under MIT license or equivalent
  • The exposed Haxe interface should match the C interface as closely as possible
  • Absolute minimum dependencies
  • Account for and note any "leaky abstractions" / incompatibilities between HXCPP & HL
  • Minimize use of operations that generate expensive side-effects (such as hidden Dynamics on HXCPP target) -- libraries that rely on this solution should be suitable for high-performance things like physics and rendering.
  • Exact approach (e.g. using CFFI, CFFI Prime, C externs, untyped C blocks, etc) is entirely up to you, just provide a reasoning for why your solution is the best one you could think of compared to the alternatives.

The sample should:

  • Demonstrate passing two numbers from Haxe, adding them together on the C side, and returning them to Haxe
  • Demonstrate passing a file name from Haxe, loading the file and resolving it as raw bytes on the C side, and returning the bytes to Haxe
  • Demonstrate passing two (ASCII) strings from Haxe, concatenate them together on the C side, and return them to Haxe
  • Demonstrate all the above, with the functions implemented in loose .c and/or .h files
  • Demonstrate all the above, with the functions implemented in a .dll file
  • Be able to be compiled and run against hxcpp target and hl target with a simple .hxml script

Out of scope

None of these things are part of the project:

  • C++ API support (we only care about supporting plain C API's for now, save this for later)
  • Exhaustive documentation (just a simple example and a quick description is fine)
  • Unicode string support in the sample (just ASCII is fine for now)
  • Pass a structured object from Haxe, mutate some of its values on the C side, and return it to Haxe
  • Properly handle a C api that asks for a C data structure as input (important to solve eventually but let's not worry about it just yet)
  • Automatic generation of bindings from a minimal description (such as c header files alone or whatever)

Prerequisites

  • Knowledge of HXCPP
  • Knowledge of Hashlink
  • Knowledge of Macros
  • Knowledge of C

What's been done so far

Nothing specifically on this project, but here's some prior work to learn form:

How Much can Lars help?

  • I'm available to talk to and ask/answer questions but you'll be doing the majority if not all the work here. I recognize I'm out of my depth.

What remains to be done

  • The whole thing!

Budget/Timeline:

  • No specific timeline. 1 month would be great? I'd like to see it done before November.

Budget is $100 USD. $200 USD $250 USD

@nadako

This comment has been minimized.

Copy link

nadako commented Oct 3, 2018

For the people looking into this:

I experimented with auto-generating glue for Java's JNI API for hashlink before and had some success. The idea is pretty simple: we define an extern class in Haxe like this and then process it with a macro and generate glue C code somewhat like this. That particular code is purely experimental and JNI-specific of course, but I think the idea is really nice and the same approach can be used for generating both HL and HXCPP glue code.

@ncannasse

This comment has been minimized.

Copy link

ncannasse commented Oct 3, 2018

I have also some experiment to generate both Emscriptem and HL bindings from an IDL structure, see https://github.com/ncannasse/webidl

@larsiusprime

This comment has been minimized.

Copy link
Owner Author

larsiusprime commented Oct 3, 2018

@ncannasse -- very interesting, Nicolas! Do you actually use it in production anywhere or is it just tinkering for now? Presumably this format could be used to generate hxcpp bindings in addition to your emscriptem/HL bindings?

@ncannasse

This comment has been minimized.

Copy link

ncannasse commented Oct 3, 2018

@larsiusprime not yet in production, but I did use it for some Bullet3D experiments here https://github.com/HaxeFoundation/hashlink/tree/master/libs/bullet

The nice thing it that it generates both Haxe externs (using macros) and C stub code from IDL.
Cons is that it only supports C++ for now, but I guess the IDL could be enriched with additional annotations to support C.

One big cons versus "manual" wrapper is that it does not allow much cooperation with GC so it requires for the Haxe wrapper objects to keep references on the C++ objects. And when the the Haxe object gets GC'ed to delete the C++ object automatically. This also requires to make sure that Haxe references are keeping each other the way C++ objects are, which is extra post wrapping work.

@larsiusprime

This comment has been minimized.

Copy link
Owner Author

larsiusprime commented Oct 4, 2018

@ncannasse speaking of "manual" wrapper -- is it possible in principle to write the "manual" part, with GC fiddling and all, and have a script/macro/etc generate both a HXCPP and a Hashlink binding from whatever intermediate format is used? Or is the way GC behaves in Hashlink and HXCPP sufficiently different to make this not possible?

@kevinresol

This comment has been minimized.

Copy link

kevinresol commented Oct 4, 2018

Any ways to pool a budget? Maybe something like OpenBounty?

@ncannasse

This comment has been minimized.

Copy link

ncannasse commented Oct 4, 2018

I think the problem of "automatic" wrapper generation VS an IDL and the problem of a "common" way to express the wrappers are two separate problems. Once the "common" problem is solved, it will help a lot the "automatic" problem.

Regarding a good way to express the "common" wrapper generation, there are two ways to deal with it:

a) write C code with an #include <common_wrapper.h> that defines an API that behind the scene will call HL API or HxCPP one (or other platforms). But this is quite low level and I'm not sure how much additional platform-specific things will be required there.

b) write Haxe code that also implements a cross platform "extern" API, that does not get compiled but that gets read by macros to output the corresponding platform-specific C code. That's slightly higher level and allows to add platform specific constructors automatically by detecting the places they are needed from the Haxe AST.

@larsiusprime

This comment has been minimized.

Copy link
Owner Author

larsiusprime commented Oct 4, 2018

@kevinresol -- possibly! So far I've just done everything informally. Rest assured, if several people contribute something significant I'll do my best to split the bounty fairly, and maybe even bump it up a little if that feels necessary.

@kevinresol

This comment has been minimized.

Copy link

kevinresol commented Oct 4, 2018

I mean the other way round, I probably would like to contribute some money to the budget

@larsiusprime

This comment has been minimized.

Copy link
Owner Author

larsiusprime commented Oct 4, 2018

@kevinresol -- oh, interesting! That's a swell idea, can you link to any services you like? What's this OpenBounty thing for instance?

@kevinresol

This comment has been minimized.

Copy link

kevinresol commented Oct 4, 2018

Not really, I just randomly searched it without deeply looking at it yet. Maybe we should discuss in a new thread.

@larsiusprime

This comment has been minimized.

Copy link
Owner Author

larsiusprime commented Oct 4, 2018

@kevinresol:
Great, let's discuss that here.

@ncannasse:
Thanks for the input! Re this:

I think the problem of "automatic" wrapper generation VS an IDL and the problem of a "common" way to express the wrappers are two separate problems. Once the "common" problem is solved, it will help a lot the "automatic" problem.

I'll update the issue to indicate the scope of this bounty is solely the "common" problem. If we can solve that, we can tackle the "automatic" problem as a separate issue.

@kevinresol

This comment has been minimized.

Copy link

kevinresol commented Oct 13, 2018

How about extracting (one of) the libs into a separate repo to demonstrate how to write a native lib for hl?

Take mysql as an example, move all the mysql source, glue code, haxe extern files, build instructions (cmake) into a repo which can be consumed by simply adding sth like -lib hl-mysql to a hl project. This is similar to what linc does. I actually like linc because I found its self-contained approach makes it much easier to develop and consume native extensions.

@kevinresol

This comment has been minimized.

Copy link

kevinresol commented Nov 6, 2018

How about extracting (one of) the libs into a separate repo to demonstrate how to write a native lib for hl?

Take mysql as an example, move all the mysql source, glue code, haxe extern files, build instructions (cmake) into a repo which can be consumed by simply adding sth like -lib hl-mysql to a hl project. This is similar to what linc does. I actually like linc because I found its self-contained approach makes it much easier to develop and consume native extensions.

@ncannasse what's your opinion on this?

@ncannasse

This comment has been minimized.

Copy link

ncannasse commented Nov 6, 2018

@kevinresol on one hand I like the idea , otoh this would require users of these libraries to install Visual Studio C++ in order to compile them. I kind of like that you can ship precompiled binaries so users don't have to deal with native compilation at all -- that works especially for windows users.

@kevinresol

This comment has been minimized.

Copy link

kevinresol commented Nov 6, 2018

@ncannasse Ideally haxeilb should be able to selectively download files per user's current OS. npm does that smoothly and is even able to compile the binaries during a library install.

Anyway, compared to the binary delivery problem I think I am far more concerned about the fact that the current approach with hl extensions is not scalable at all. To be honest, I am unaware of any HL extensions outside the HL repo.

In the early days hxcpp suffered from the problem as well. At that time native extensions are mostly tied to OpenFL (because OpenFL provided a native binding mechanism? Not very sure). But since the introduction of linc, there are so many framework-agnostic libraries for hxcpp emerged which is really helped users a lot.

@larsiusprime

This comment has been minimized.

Copy link
Owner Author

larsiusprime commented Nov 6, 2018

Yeah, I'm a big fan of the linc standard!

FYI, I'm bumping the value of this bounty to $200.

@larsiusprime larsiusprime changed the title $100 - Common HL/HXCPP native C API binding format $200 - Common HL/HXCPP native C API binding format Nov 6, 2018

@larsiusprime larsiusprime changed the title $200 - Common HL/HXCPP native C API binding format $250 - Common HL/HXCPP native C API binding format Nov 13, 2018

@larsiusprime

This comment has been minimized.

Copy link
Owner Author

larsiusprime commented Nov 13, 2018

Bumping the value of this bounty to $250

@Sunjammer

This comment has been minimized.

Copy link

Sunjammer commented Nov 14, 2018

linc + inline cpp has been a revelation, but it's less a standard than a guideline. From lib to lib Linc authors still pick divergent approaches, especially when it comes to managing call backs. I don't necessarily mind this, it reflects the different designs in the c libraries themselves. Hard to normalize this.

@chfoo

This comment has been minimized.

Copy link

chfoo commented Dec 27, 2018

I had some time looking at this and I created a proof-of-concept. It basically is a command line tool that reads in XML files and generates a Haxe extern class file and a C function wrapper file. The result is a one-to-one mapping of parameters but only of very limited types for now.

I used externs because it was the approach used in the standard library which provided lots of examples. The process is a bit fragile as it simply spits out Haxe and C code without any framework or error checking. The example might show warnings or not compile, particularly in HashLink, because I am using uint64_t* to bind to size_t* for the sake of simplicity. But overall, it can pass numbers, raw bytes, and strings (with caveats noted in the readme).

@skial skial referenced this issue Jan 9, 2019

Closed

Haxe Roundup 462 #576

1 of 1 task complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.