☂️ Implement pluggable transformation (via packages) of assets at build time

[andrewkolos]

Update. This feature is fully-implemented and available in the master branch.

Pending official documentation: flutter/website#10471
Pending sample project, which you can use to play with this feature: flutter/samples#2267

---

This tracks the implementation of the feature requested in #101077. The rest of this issue assumes the reader has a basic understanding of application assets in Flutter; see https://docs.flutter.dev/ui/assets/assets-and-images for more.

This feature allows users to configure their assets to be transformed by Dart CLI programs.

When declaring an asset in their pubspec, Flutter app developers can specify a transformer along with it. When building the app, the flutter CLI tool will run the transformer against the original asset file, with the output routed to the appropriate directory.

Example use case. Consider a scenario where a user wants to use the vector_graphics package to render SVG images in their app. To make an SVG file renderable by vector_graphics the user would normally have to manually run the vector_graphics_compiler package against it. The user would then need to have this transformed file bundled into their app (not the original). However, they can now instead do this:

# pubspec.yaml
...
flutter:
...
  assets:
    - path: assets/icon.svg
      transformers:
        - package: vector_graphics_compiler

Then, in their app code, the developer can the vector_graphics package to render the asset.

import 'package:vector_graphics/vector_graphics.dart';
// ...
final Widget icon = VectorGraphic(loader: AssetBytesLoader('assets/icon.svg'))

This makes adding SVGs to a Flutter app simpler. While the flutter_svg package could already render SVGs without requiring precompilation, it is significantly slower since it needs to decode and parse XML (amongst other reasons). With this feature, developers can enjoy convenience similar to that of using flutter_svg while also enjoying the superior rendering performance of vector_graphics.

vector_graphics_compiler can be used as an asset transformer because it is a CLI Dart application that accepts an --input argument (the path to the original SVG file) and an --output argument (where the output the file that can be rendered by the vector_graphics package. Any Dart CLI application that accepts these arguments and writes an output file to the provided --output path can be used as an asset transformer.

Tasks:

• Decide on the scheme that users use to declare transformations in the pubspec, and augment tool code to validate and parse this
  • add parsing of assets transformer declarations in pubspec.yaml #143557
• Enable asset transformation for workflows that use copyAssets to do bundle and write assets. If I'm not mistaken, this includesflutter build for Android, iOS/macOS, Windows, Linux, and macOS.
  • Enable asset transformation for flutter build for iOS, Android, Windows, MacOS, Linux, and web (also flutter run without hot reload support) #143815
• Enable asset transformation for non-web flutter run scenarios.
  • Enable asset transformation feature in hot reload workflow (excluding Web) #144161
• Enable asset transformation for flutter run for web.
  • Enable asset transformation for flutter run -d <browser> and flutter test #144734
• Enable asset transformation for flutter_tester.
  • Enable asset transformation for flutter run -d <browser> and flutter test #144734
• Make hot reload reflect changes to asset transformers or configurations
  • dart run <package> can fail if a dart pub get operation is running concurrently dart-lang/pub#4161
  • make hot reload reflect changes to asset transformer configurations #144660

Cleanup tasks:

• For flutter build, see if we can make caching work in the presence of transformers that are path dependencies (see Enable asset transformation for flutter build for iOS, Android, Windows, MacOS, Linux, and web (also flutter run without hot reload support) #143815 (comment))
• Consider getting this under integration test
  • Add integration test for asset transformation feature #145715
• Consider having the flutter tool, when invoking a transformer, expose relevant information via environment variables. This would include information like what mode we are building in (debug, profile, release).
  • Expose build mode in environment of asset transformer processes #144752 (review)

[christopherfujino]

One way we could integration test this is by adding https://github.com/andrewkolos/asset_transformers_test to the github.com/flutter/tests registry (this solves needing to check in test assets to flutter/flutter). If it turns out to be more trouble than its worth, we can easily delete it.

[dcharkes]

Drive by comment.

We might be trying to solve similar use cases in the work on "native assets" (#129757), data assets (dart-lang/native#154), and hook/link.dart (dart-lang/native#153). Maybe we're trying to solve different use cases, so let's elaborate a bit.

Some of our uses cases:

• For package messages: we'd have a translation files that will be pruned by a hook/link.dart to only include keys that are reachable in Dart code after AOT compilation with the "const finder".
• For package intl: We have a Rust lib compiled to a static library, and link.dart will use the reachable symbols (again AOT & const finder) to use the linker to slim down the native code to a dylib that only contains the native code reachable from Dart.

The transformers in our use cases (hook/link.dart) are mostly about shrinking assets. And the goal is that any package can define a hook/link.dart.

Note that both of these use cases need to work in Dart standalone and Flutter. So unfortunately a Flutter-specific solution for resource transformation doesn't work.

We should maybe sync up on what work overlaps so we can leverage each others' work. cc @mosuem

Detailed comments:

Users author Dart packages that transform a file. These packages follow two rules: 1) they accept (the location of) an input file via an --input option, and 2) they write an output file the path provided by an --output option.

@andrewkolos Where do the entry points for these transformation scripts live? bin/?
Having a predefined CLI sounds like a hook/ (dart-lang/sdk#54334). In our work with with native assets we considered using the bin/ directory for these. However, since these hooks are not necessarily meant to be invoked by users of the package, but rather by flutter_tools and Dart tooling, we have reserved a new directory for such use cases.

When declaring an asset in their pubspec, Flutter app developers can specify a transformer along with it.

@mosuem This sound rather similar to in our proposal having hook/build.dart outputting BuildOutput.addAssetForLinking(String targetPackage, Asset asset). And then getting that assets as input LinkInput.assets in hook/link.dart. The package defining the asset specifies the linker/shrinker/compiler to use for that asset.

When building the app, the flutter CLI tool will run the transformer against the original asset file, with the output routed to the appropriate directory.

This is precisely what flutter_tools does with the dylibs output by hook/build.dart. And we'd envision flutter_tools (and the Dart SDK) something similar for "data assets" after shrinking/linking.
@andrewkolos Are you envisioning the shrunken/compiled assets to be available as the normal asset bundle in Flutter?

(Side note: hook/build.dart is currently build.dart and still have to be renamed.)

[andrewkolos]

Users author Dart packages that transform a file. These packages follow two rules: 1) they accept (the location of) an input file via an --input option, and 2) they write an output file the path provided by an --output option.

@andrewkolos Where do the entry points for these transformation scripts live? bin/? Having a predefined CLI sounds like a hook/ (dart-lang/sdk#54334). In our work with with native assets we considered using the bin/ directory for these. However, since these hooks are not necessarily meant to be invoked by users of the package, but rather by flutter_tools and Dart tooling, we have reserved a new directory for such use cases.

Yes, they live in bin. When we invoke an asset transformer, we simply run dart run <package name>, which will invoke <package dir>/bin/<package_name>.dart (docs for this behavior). I chose this for simplicity. I am also not aware of any way to invoke a program from a pub package.

When building the app, the flutter CLI tool will run the transformer against the original asset file, with the output routed to the appropriate directory.

This is precisely what flutter_tools does with the dylibs output by hook/build.dart. And we'd envision flutter_tools (and the Dart SDK) something similar for "data assets" after shrinking/linking. @andrewkolos Are you envisioning the shrunken/compiled assets to be available as the normal asset bundle in Flutter?

Yes, the file output by the dart run invocation is available for loading through the AssetBundle framework API like any other asset. For an example, see the example app (setup, dart code). These assets are only special in that they are transformed as they are copied to the device/build directory.

[dcharkes]

Users author Dart packages that transform a file. These packages follow two rules: 1) they accept (the location of) an input file via an --input option, and 2) they write an output file the path provided by an --output option.

@andrewkolos Where do the entry points for these transformation scripts live? bin/? Having a predefined CLI sounds like a hook/ (dart-lang/sdk#54334). In our work with with native assets we considered using the bin/ directory for these. However, since these hooks are not necessarily meant to be invoked by users of the package, but rather by flutter_tools and Dart tooling, we have reserved a new directory for such use cases.

Yes, they live in bin. When we invoke an asset transformer, we simply run dart run <package name>, which will invoke <package dir>/bin/<package_name>.dart (docs for this behavior). I chose this for simplicity. I am also not aware of any way to invoke a program from a pub package.

For the native assets feature, dartdev and flutter_tools do the invocation with a dart /absolute/path/to/hook/build.dart, instead of relying on dart pub run.

When building the app, the flutter CLI tool will run the transformer against the original asset file, with the output routed to the appropriate directory.

This is precisely what flutter_tools does with the dylibs output by hook/build.dart. And we'd envision flutter_tools (and the Dart SDK) something similar for "data assets" after shrinking/linking. @andrewkolos Are you envisioning the shrunken/compiled assets to be available as the normal asset bundle in Flutter?

Yes, the file output by the dart run invocation is available for loading through the AssetBundle framework API like any other asset. For an example, see the example app (setup, dart code). These assets are only special in that they are transformed as they are copied to the device/build directory.

Cool! The "data assets" for Dart standalone packages such as package:intl will be available with the same API in Flutter as well. (And we need to port the API to dart: as well.)

[andrewkolos]

For the native assets feature, dartdev and flutter_tools do the invocation with a dart /absolute/path/to/hook/build.dart, instead of relying on dart pub run.?

How are the dependencies of the package resolved then? For example, if I

1. create a new flutter project,
2. add vector_graphics_compiler as a dev dependency, and
3. directly invoke the package via dart run Users/andrewkolos/.pub-cache/hosted/pub.dev/vector_graphics_compiler-1.1.9+2/bin/vector_graphics_compiler.dart, then

I will get compiler errors:

Error: Couldn't resolve the package 'args' in 'package:args/args.dart'.
Error: Couldn't resolve the package 'vector_graphics_compiler' in 'package:vector_graphics_compiler/src/svg/colors.dart'.
Error: Couldn't resolve the package 'vector_graphics_compiler' in 'package:vector_graphics_compiler/vector_graphics_compiler.dart'.
Error: Couldn't resolve the package 'path' in 'package:path/path.dart'.
../../.pub-cache/hosted/pub.dev/vector_graphics_compiler-1.1.9+2/bin/vector_graphics_compiler.dart:7:8: Error: Not found: 'package:args/args.dart'
import 'package:args/args.dart';
       ^
../../.pub-cache/hosted/pub.dev/vector_graphics_compiler-1.1.9+2/bin/vector_graphics_compiler.dart:8:8: Error: Not found: 'package:vector_graphics_compiler/src/svg/colors.dart'
import 'package:vector_graphics_compiler/src/svg/colors.dart';
       ^
../../.pub-cache/hosted/pub.dev/vector_graphics_compiler-1.1.9+2/bin/vector_graphics_compiler.dart:9:8: Error: Not found: 'package:vector_graphics_compiler/vector_graphics_compiler.dart'
import 'package:vector_graphics_compiler/vector_graphics_compiler.dart';
       ^
../../.pub-cache/hosted/pub.dev/vector_graphics_compiler-1.1.9+2/bin/util/isolate_processor.dart:10:8: Error: Not found: 'package:vector_graphics_compiler/vector_graphics_compiler.dart'
import 'package:vector_graphics_compiler/vector_graphics_compiler.dart';
       ^
Error: Couldn't resolve the package 'vector_graphics_compiler' in 'package:vector_graphics_compiler/src/debug_format.dart'.
../../.pub-cache/hosted/pub.dev/vector_graphics_compiler-1.1.9+2/bin/vector_graphics_compiler.dart:12:8: Error: Not found: 'package:path/path.dart'
import 'package:path/path.dart' as p;

[dcharkes]

How are the dependencies of the package resolved then?

The packages_config.json for the root package is used.

(We've had a bunch of conversations whether we should allow hooks of different packages to resolve their dependencies to different versions, but decided that that can have negative consequences. Especially if the code in src/ in the root package imports package:a and uses its Dart API, and passes an asset that is transformed by package:a in a hook, the author of package:a probably wants to rely on that being the same version of package:a. See dart-lang/pub#3794 and dart-lang/native#160 for some more considerations.)

flutter_tools and dartdev take care of passing the package config to the dart invocation. For example:

flutter/packages/flutter_tools/lib/src/build_system/targets/native_assets.dart

Lines 71 to 81 in d98d842

	final PackageConfig packageConfig = await loadPackageConfigWithLogging(
	packagesFile,
	logger: environment.logger,
	);
	final NativeAssetsBuildRunner buildRunner = _buildRunner ??
	NativeAssetsBuildRunnerImpl(
	projectUri,
	packageConfig,
	fileSystem,
	environment.logger,
	);

[christopherfujino]

A few thoughts:

1. before we ship, we should add a printTrace to the tool about each asset transformer we're invoking and the path/arguments we called it with (edit by @andrewkolos: done in print traces when transforming an asset #146374)
2. not a blocker, but we should consider in non-verbose mode adding the time asset transformers took to the durations we print out about how long the hot reload took (e.g. Reloaded 0 libraries in 17,285ms (compile: 28 ms, reload: 0 ms, reassemble: 54 ms).). Consider just adding a stopwatch for how long processing the asset bundle took.
3. not a blocker, but we should add analytics around this feature.