[native_assets_cli] Dependencies should be per asset

[dcharkes]

Currently, the dependencies is a list of dependencies for all assets together. (This was modeled after the Rust build.rs.) But in the context of hot-reload/hot-restart we might want to only rebuild some assets (#1207). So then we'd need to know the dependencies per asset.

Some open questions:

1. What if multiple assets are produced by the same list of dependencies? Should we try to deduplicate such list of dependencies?
2. We've been considering making assetId optional. E.g. an assetId would only be needed for things referenced from Dart code. And not for JARs for example. But we'll likely need some way to uniquely identify an asset.

Version skew:

• Older build.dart newer SDK: Deps added to the general config could be interpreted as deps of all assets.
• Newer build.dart older SDK: Output all deps combined.

[dcharkes]

We should probably merge the addition of dependencies and assets into a single add function.

Merging in addAsset, optional named arg

  /// Adds [Asset] produced by this build or dry run.
  ///
  /// If provided, [dependencies] adds files used to build this [asset]. If any
  /// of the files are modified after [timestamp], the build will be re-run. If
  /// omitted, and [Asset.file] is outside the [BuildConfig.outputDirectory], a
  /// dependency on the file is added implicitly.
  ///
  /// If the [linkInPackage] argument is specified, the asset will not be
  /// bundled during the build step, but sent as input to the link hook of the
  /// specified package, where it can be further processed and possibly bundled.
  void addAsset(
    Asset asset, {
    Iterable<Uri>? dependencies,
    String? linkInPackage,
  });

  /// Adds [Asset]s produced by this build or dry run.
  ///
  /// If provided, [dependencies] adds files used to build these [assets]. If
  /// any of the files are modified after [timestamp], the build will be re-run.
  /// If omitted, and [Asset.file] is outside the [BuildConfig.outputDirectory],
  /// a dependency on the file is added implicitly.
  ///
  /// If the [linkInPackage] argument is specified, the assets will not be
  /// bundled during the build step, but sent as input to the link hook of the
  /// specified package, where they can be further processed and possibly
  /// bundled.
  void addAssets(
    Iterable<Asset> assets, {
    Iterable<Uri>? dependencies,
    String? linkInPackage,
  });

Usage:

    output.addAsset(
      NativeCodeAsset(
        package: 'add_asset_link',
        name: 'dylib_add_link',
        linkMode: builtDylib.linkMode,
        os: builtDylib.os,
        architecture: builtDylib.architecture,
        file: builtDylib.file,
      ),
      dependencies: [
        config.packageRoot.resolve('hook/link.dart'),
      ],
    );

Should the dependencies param be optional? In dryRun no dependencies have to be passed.
And data assets which are verbatim in an assets/ dir which could be skipped copying to the output dir, could have an implicit dependency on themselves by default.

If it should be optional, it needs to be a named param.

Alternatively, we can make it a positional param.

wdyt @mosuem ?

(bad option) Keeping addDependencies separate from addAsset

  /// Adds file used by this link hook for creating [asset].
  ///
  /// If any of the files are modified after [timestamp], the link will be
  /// re-run.
  void addDependency(Asset asset, Uri dependency);

  /// Adds files used by this link hook for creating [asset].
  ///
  /// If any of the files are modified after [timestamp], the link will be
  /// re-run.
  void addDependencies(Asset asset, Iterable<Uri> dependencies);

Keeping addAsset and addDependency separate leads to two issues.

1. An asset might be used in addDependency but never added with addAsset.
2. addAsset and addDependency leads to awkward code where an asset always needs to be in a variable to be used in both:

    final asset = NativeCodeAsset(
      package: 'add_asset_link',
      name: 'dylib_add_link',
      linkMode: builtDylib.linkMode,
      os: builtDylib.os,
      architecture: builtDylib.architecture,
      file: builtDylib.file,
    );
    output
      ..addAsset(asset)
      ..addDependency(asset, config.packageRoot.resolve('hook/link.dart'));
  });

[dcharkes]

Nesting dependencies under assets will not fully work.

A build hook can also compile an executable that is then stored in the metadata, which is then used other build hooks.

For example (along the lines of flutter/flutter#144259).

• package:my_3d_engine compiles a C library that produces a my_shader_compiler.exe and puts the path to the executable in the metadata.
  • It also provides a helper function CompiledShader compileShader(ShaderSource source) to be called in build hooks that looks up the executable in the metadata.
  • It provides an engine that takes CompiledShaders (which are just a wrapper around Uint8List).
• package:my_game has a dependency on package:my_3d_engine
  • It has a build hook that invokes compileShader for a bunch of shaders and bundles the output as DataAssets
  • Its runtime code loads those data assets as Uint8Listss and passes them to the game engine to display.

If we want to support hot-reload for those compiled shaders, the question is whether the my_shader_compiler.exe should be rebuilt or not. In this case likely not.

However, if we're cold starting, most likely we should check if any of the dependencies for the my_shader_compiler.exe was modified. (E.g. pub upgrade pulled in a new version of package:my_3d_engine.)

So we need to also store dependencies that are not tied to an asset, but tied to the hook itself. We could specify that these dependencies are ignored when hot-reloading/hot-restarting. But maybe there are use cases where something from the meta-data should actually be rebuilt in a hot-reload?

(If we want to go really wild, we could rethink whether storing an executable in meta data in an unstructured way is the right way to go. Then we could possibly track the dependencies per file that we possible store in metadata. But now we're building a full fledged build system. 😆 )

Thoughts @mkustermann @mosuem @HosseinYousefi?

[dcharkes]

Nesting dependencies under assets will not fully work.

Another situation in which this will not fully work is when writing hooks that report all files in an assets/ directory as assets:

void main(List<String> args) async {
  await build(
    args,
    (config, output) async {
      final assetDirectory =
          Directory.fromUri(config.packageRoot.resolve('assets/'));

      // If assets are added, rerun hook.          <------------------------
      output.addDependency(assetDirectory.uri);

      await for (final dataAsset in assetDirectory.list()) {
        if (dataAsset is! File) {
          continue;
        }
        final name = dataAsset.uri.pathSegments.last;
        output.addAsset(
          DataAsset(
            package: config.packageName,
            name: name,
            file: dataAsset.uri,
          ),
          dependencies: [
            dataAsset.uri,
          ],
        );
      }
    },
  );
}

So, we'll likely need to keep the "hook dependencies" in addition to asset dependencies. Maybe the "hook dependencies" need to be reported per asset type, because the hot restart hook would only want to build some asset types (#1207).

[dcharkes]

Circling back here after discussing #1368 with @mosuem.

One aspect of the design we don't really like is having both output.addAssetTypeDependency() and output.addAsset(<asset>, dependencies: [ <...> ]).

We need some kind of global dependencies for knowing that a hook needs to be re-run if a file gets added in a directory:

The current PR looks like:

void main(List<String> args) async {
  await build(args, (config, output) async {
    final packageName = config.packageName;
    final assetDirectory =
        Directory.fromUri(config.packageRoot.resolve('assets/'));
    // If assets are added, rerun hook.
    output.addAssetTypeDependency(DataAsset.type, assetDirectory.uri);

    await for (final dataAsset in assetDirectory.list()) {
      if (dataAsset is! File) {
        continue;
      }

      // The file path relative to the package root, with forward slashes.
      final name = dataAsset.uri
          .toFilePath(windows: false)
          .substring(config.packageRoot.toFilePath(windows: false).length);

      output.addAsset(
        DataAsset(
          package: packageName,
          name: name,
          file: dataAsset.uri,
        ),
        linkInPackage: config.linkingEnabled ? packageName : null,
        dependencies: [dataAsset.uri],
      );
    }
  });
}

Some alternatives:

1. Keep "global" dependencies next to asset dependencies, but remove the assetType-part. addGlobalDepencies. Reasoning: likely only data assets will be in the style where file-existence creates new assets. For NativeCodeAssets, JarAssets, ProguardRuleAssets we'll likely have a fixed set of assets.
2. Remove the concept of global dependencies and reuse asset dependencies to track global dependencies. Reasoning: we only need one concept.
   a. Report the directory dependency on a dummy DataAsset. Downside: this data asset needs to have a file, and both the path to the file and the file itself will be bundled in the final app.
   b. Add a Routing.onlyForDependencies in the style of [native_assets_cli] Unify metadata and assets #1251 (comment). This enables using an asset just for dependency tracking and it will not be sent to any build hook, link hook or be bundled.
3. Have dependencies reported on a GlobalDependencies() and AssetsDependencies(List<Asset> assets). This would also enable not having to repeat dependencies if they are just for more than one asset. Downsides: API becomes more complicated and indirect because it doesn't have a addAsset(dependencies: ) call, but a separate call for adding dependencies.

I think we're leaning towards 2.b.

Friendly ping for @HosseinYousefi and @mkustermann.

[dcharkes]

From discussion with @HosseinYousefi:

• If 1: addAssetTypeDependency -> addHookDependency or addDepedency
• If 1: We do need an asset type for the "global" dependencies ("hook" dependencies), some mavenDownloaderHelper can make a list of JarAssets.
• If we have this list of dependencies per asset being reported, we should also give that list back on rerun, so that users inside the hook can also skip rerunning things inside hooks.
  • Merging [native_assets_cli] BuildOutput dependencies per asset and asset type #1368 without providing any befits for users moves us in the right direction for supporting hot restart at some point, but it creates a more complicated API in the mean-time.
• If we want to support caching between the main build and the hot restart build, it needs to happen in the same directory. But then we also need to keep track of the fact that DataAssets have been built at timestamp x (hot restart), but the NativeCodeAssets have last been build at x-1 (the cold start).

[dcharkes]

From more discussion with @HosseinYousefi:

If we report dependencies per asset, we expect the build system to be smart and do something per asset as well (#1375 option 2). But it's not clear that we would benefit from doing that unless we make a fine-grained build graph that tracks every asset and its dependencies individually with a closure to rerun that specific part of the build. And that would lead us to basically making a full build system, which we're trying to avoid.

@HosseinYousefi suggested we take a completely different direction: #1207 (comment).

[dcharkes]

As discussed in #1207 (comment), we don't want assets per dependency.