Use native-assets to vendor MediaPipe SDK

[craiglabenz]

Uses Dart and Flutter's new "native assets" feature (dart-lang/sdk#50565 and flutter/flutter#129757) to allow Dart and Flutter's tooling to do the heavy work of embedding MediaPipe's SDK in the final app.

The vision for this is to precompile the MediaPipe SDKs (one for each target platform / architecture permutation) separately and then download / correctly place those libraries during the build.dart build hook that is invoked by the new native assets feature. build.dart is currently empty pending answers to the following questions:

• Where will we compile the full matrix of MediaPipe libraries?
• Where will we host the assets for build-time download?
• How will the build.dart file determine when a new download is needed without doing so every build? (Should we store a checksum?)

cc @schmidt-sebastian

[craiglabenz]

Note that, as of this writing, Flutter's tooling buries build.dart's stdout and stderr (flutter/flutter#136386), which may complicate the authorship of build.dart once we have the asset details in place to begin writing.

[craiglabenz]

From @dcharkes, development-cycle trick when we're ready:

$ dart build.dart --config=.dart_tool/native_assets_builder/<long-hash>/config.yaml

[dcharkes]

• How will the build.dart file determine when a new download is needed without doing so every build? (Should we store a checksum?)

build.dart does not get reinvoked if

1. the BuildConfig didn't change, and
2. none of the files in dependencies list in BuildOutput was modified after the last build.

[craiglabenz]

Thanks for that info, @dcharkes.

How does one control when the BuildConfig changes outside of build.dart itself? The example shows that object being instantiated within build.dart.

[dcharkes]

Thanks for that info, @dcharkes.

How does one control when the BuildConfig changes outside of build.dart itself? The example shows that object being instantiated within build.dart.

Let me rephrase, the config.yaml contents that is being used to instantiate the BuildConfig is hashed.

When downloading, your dependencies list in BuildOutput should be:

Any file in your package that if it changes when developing locally build.dart should be rerun (this also covers when users depend on you with a path dependency for whatever reason). For example build.dart itself if it contains the URL where to download the binary from and that URL changes every time you update the binary.

When other devs depend on your package, any new version of the package ends up in a different location in the pub cache. For example: /Users/<user-name>/.pub-cache/hosted/pub.dev/file-6.1.4/. Because the package root is part of the config. Users upgrading your package via pub will lead to a new package root, which is included in the hash.

[craiglabenz]

@dcharkes If I'm following the plot, all change detection must be contained within the repository; as build.dart is not invoked until a change is detected. I had envisioned detecting the change within build.dart itself, but it seems that's not the right approach.

Is it appropriate to store a hash of latest versions of the assets in question in config.yaml (and then ideally automate its updates)?

[dcharkes]

@dcharkes If I'm following the plot, all change detection must be contained within the repository; as build.dart is not invoked until a change is detected. I had envisioned detecting the change within build.dart itself, but it seems that's not the right approach.

Typically build systems output a list of dependencies so that the invoker of the build can decide whether to redo the build. That saves a process start (and if the process is a interpreter/JIT as in python/dart all the internal startup).

Is it appropriate to store a hash of latest versions of the assets in question in config.yaml (and then ideally automate its updates)?

Yes, that's a fine approach.
However, why not depend on the URL you embed in build.dart? Once you create new binaries, you're likely going to need to update the URL, because you want your old binaries to still be available for older published versions of your package.

[craiglabenz]

However, why not depend on the URL you embed in build.dart? Once you create new binaries, you're likely going to need to update the URL, because you want your old binaries to still be available for older published versions of your package.

Yes, we could definitely do that. I suppose I have Docker on the mind and was thinking about a permanent :latest alias for the most up to date assets, but there are multiple ways of doing this.

[dcharkes]

However, why not depend on the URL you embed in build.dart? Once you create new binaries, you're likely going to need to update the URL, because you want your old binaries to still be available for older published versions of your package.

Yes, we could definitely do that. I suppose I have Docker on the mind and was thinking about a permanent :latest alias for the most up to date assets, but there are multiple ways of doing this.

I think using a latest alias might not work. Imagine you do a breaking change to your C API, and someone has an older version of the Dart code with bindings generated to an older version of your C API, that would crash at runtime.

[craiglabenz]

After finally poking at this today, I have a question about how to hold it:

You've described build.dart as getting re-invoked when there are changes to any of the dependencies indicated in config.yaml, and that as such, we could store the URL to the latest version of the MediaPipe URL in that file. Unless I'm missing something, it seems that config.yaml is a generated file that lives under .dart_tool/native_assets_builder/<hash>/. That path is both typically gitignored and is programmatically difficult to access with the hash, and so would be a difficult place to store the pointer to the latest MediaPipe binary.

My question here is - am I missing anything regarding the location or nature of config.yaml?

(If I am in fact missing something and my above understanding is meaningfully incomplete, then ignore everything to come.)

My early sense is that the native assets feature design may currently have better support for the Dart-compiles-your-C-code scenario than the build.dart-downloads-your-artifacts scenario; due to the apparent Catch-22's in detecting when those external dependencies have been updated.

[dcharkes]

After finally poking at this today, I have a question about how to hold it:

You've described build.dart as getting re-invoked when there are changes to any of the dependencies indicated in config.yaml, and that as such, we could store the URL to the latest version of the MediaPipe URL in that file.

No, URL should be a string inside build.dart itself. And build.dart should output itself as one of the dependencies.

The script should be something along the lines of this. (But more sophisticated taking OS/arch into account.)

void main(List<String> args) async {
  final buildConfig = await BuildConfig.fromArgs(args);
  final buildOutput = BuildOutput();
  final downloadUri = Uri.parse('https://bladibla.com/my_package.v1.so');
  final downloadedFile = buildConfig.outDir.resolve('my_package.v1.so');
  // await download.
  buildOutput.dependencies.dependencies
      .add(buildConfig.packageRoot.resolve('build.dart'));
  buildOutput.assets.add(
    Asset(
      id: 'package:$packageName/${packageName}_bindings_generated.dart',
      linkMode: LinkMode.dynamic,
      target: Target.linuxX64,
      path: AssetAbsolutePath(downloadedFile),
    ),
  );
  await buildOutput.writeToFile(outDir: buildConfig.outDir);
}

[Piinks]

LGTM-d over GVC 😉