Remaining tasks for dart:ffi for Dart 2.6

[mkustermann]

Let's accumulate here everything we want to have in Dart 2.6 that is still open:

• [done, Samir] Dart -> C and C -> Dart callbacks re-landed, tested and work in all modes.
• [in-progress, Daco] Fast performance of all memory operations with nice API (to avoid suggesting to users usage of ExternalTypedData hack)
• [open] In addition to Utf8 strings, add more convenience utilities to package:ffi, e.g. memory arenas
• [open] Crawl test results database and flakiness data ensure that all ffi tests work in all configurations and there are no flaky failures
• [In-progress, Daco] FFI uses extension methods

Please just edit this post if you want to add more tasks or mark some as completed.

/cc @mraleph @mit-mit

[mkustermann]

@mit-mit Would it make sense to have a centralized place under https://dart.dev/guides for introduction to dart:ffi for Dart 2.6?

[sjindel-google]

I updated the list to include extension methods.

[dcharkes]

I've been experimenting with extension methods and providing feedback. Note that we cannot land extension methods until the extension methods flag is on by default. Though that should happen before 2.6.

[mit-mit]

@mit-mit Would it make sense to have a centralized place under https://dart.dev/guides for introduction to dart:ffi for Dart 2.6?

Currently the docs are here: https://dart.dev/guides/libraries/c-interop. But we clearly need to expand them for 2.6.

[devkabiir]

Hi @mit-mit, In dart 2.5 announcement, you mentioned about code generation based on C header files. I didn't find a place to discuss that topic, so posting here. FFI is foreign to me as it is, But generators and parsers are not. I'd love to give this a try, would you mind guiding me?.

Looking at the examples at https://github.com/dart-lang/samples/tree/master/ffi. The difficult part is the intermediate representation of header files which the generator would understand and generate code for accordingly. This would probably require a parser for c headers 😅.

A dart-only approach in which the bindings are hand-written, to me looks like:

// interop.dart

@C.DynamicLibrary('my_lib.dylib') /// Sets the default dyLib to load for everything in this dart library
library interop;

import 'package:ffi_annotation/ffi_annotation.dart' as C;

part 'interop.g.dart';

/// Says hello
@C.Function('hello_world') // Annotation provides the function name in c header 
void helloWorld() => _$helloWorld(); // all boilerplate code generated

/// Says hi to [name]
@C.Function('say_hi')
void sayHi(String name) => _$sayHi(name); // The generator will know how to map primitive types

@C.Function('distance_from', dyLib: 'utils.dylib') // The generator should be able to infer and map user-defined types
double distance(Coordinate from, Coordinate to) => _$distanceFrom(from, to);

class Coordinate extends C.Struct<Coordinate> {
    double latitude;
    double longitude;
	factory Coordinate({double latitude, double longitude}) = _$Coordinate; // Forwards to the generated factory

	@C.Function('coordinate_difference')
	static Coordinate difference(Coordinate  a, Coordinate b) => _$coordinateDifference(a, b);
}

Thoughts?

[dcharkes]

I think the approach should be to read an .h-file to generate a .dart-file. Code generators with package:source_gen only work from .dart-file to .dart-file (often by matching on annotations) and uses the .part language feature to combine the generated .dart-file with the original .dart-file. Unfortunately, we do not have a package which aids parsing .h-files. For code generation of .dart-files you can either use string concatenation or package:code_builder to build an AST. (You could use package:build_runner to automate re-generating the .dart-file if/when the .h-file is changed, but I think that would be of later concern. First, a manually run generator would do the job.)

[devkabiir]

The dart-only approach I suggested tries to solve the boilerplate problem for dart bindings that are hand-written. It will generate the code that will do things like automatic conversion of dart String to Pointer<Utf8> or similar depending upon annotation details

This is the kind of boilerplate code that would be generated

class _$Coordinate extends Coordinate{

	factory _$Coordinate(double latitude, double longitude) {
	  final createCoordinatePointer = dylib
	      .lookup<ffi.NativeFunction<create_coordinate_func>>('create_coordinate');
	  final createCoordinate =
	      createCoordinatePointer.asFunction<CreateCoordinate>();
	  final coordinatePointer = createCoordinate(latitude, longitude);
	  return coordinate = coordinatePointer.load();
	}

}

A user would only have to do Coordinate(1.0, 2.0) to get an instance.
It should be possible to infer all the necessary information from dart-code and annotations to generate such code from the snippet I wrote in the earlier comment.
It will also inline with the already existing builders and the build_runner package.
(Builders (package:build) can actually take any arbitrary file and produce output for it not just dart-files,
The only restriction is that a builder's outputs should be known beforehand.
So a builder can register it's inputs as *.h and outputs as *.dart.)

In this approach where inputs are hand-written bindings .dart files and outputs are .g.dart files that contain code similar to above code, generator/builder would need to be able to:

1. Require hand-written dart code with annotations
2. parse, validate, understand dart code (which is already possible)
3. have a mapping of how certain things in dart like class Coordinate {} can be bound to c code like struct Coordinate {}. This is sparsely documented but it is already possible based on the samples

In the approach where .h files are converted to .dart files, a generator/builder would need to be able to:

1. parse .h files
2. validate the parsed input
3. construct a meaningful intermediary representation (e.g. AST)
4. Have a mapping of how certain things from .h are to be represented/converted to .dart syntax and also write the bindings in dart. So struct Coordinate is to be written in dart like class Coordinate{} along with the necessary code to bind those two together

A user of these bindings would prefer a fluent api like new Coordinate(1, .556) instead of new Coordinate.allocate(1, .556) or createCoordinate(1, .556) or new Coordinate.createCoordinate(1, .556). This cannot be done without some human intervention or if there is a spec that the header file follows (which hardly ever is)

A generator that parses .h and produces .dart files might not be able to produce such fluent api. It would just be a dumb generator (even thought it's smart enough to convert .h to sound dart code)

Thoughts?

P.S. I read my own comment and it looks like I favor one approach over the other. This is not the case, I'm still very much in the discussion phase.

[dcharkes]

@devkabiir Let's discuss this further in the issue dedicated to this: #35843.