[native_assets_cli] Redesign API step 1

[dcharkes]

This PR completely redesigns the API of native_assets_cli:

• BuildConfig now has getters/methods instead of fields.
• BuildOutput has add methods.
• Dependency and Metadata classes are hidden, the contents are available in methods.
• The implementations and interfaces live in part files to enable final classes to ensure the casts succeed.
  • We'd like to keep the API/implementation separation because build.dart writers should not be distracted in the API with methods only useful for package:native_assets_builder. Neither do we want to duplicate the logic between package:native_assets_cli and package:native_assets_builder.
• Asset is renamed to NativeCodeAsset.
• NativeCodeAssets have OS & Architecture instead of Target. Architecture can be omitted in dry runs.
• Added DataAssets. [native_assets_cli] data assets #154
• BuildConfig supplies a list supported asset types. [native_assets_cli] Open ended asset types list #968
• The protocol now does not crash on unknown asset types (but they are ignored). [native_assets_cli] Open ended asset types list #968
• Removed Target from the public API. All APIs use Architecture and OS now.
• Asset PathType is replaced with a LinkMode hierarchy. Only system dylibs have a path now.
• Assets have an optional file which designates the file on the file system after build.dart invocation.
• BuildConfigs main constructor is now the one that takes the command-line arguments.
• Asset.id is now automatically created from package and name.

This PR also revamps the protocol. However it does so in a non-breaking way by:

• Not breaking build_config.yaml (only extending). So old build.dart can parse it from a newer SDK.
• Being able to parse older build_config.yamls. So new build.dart can parse it form an older SDK.
• Outputting old build_output.yamls if the build_config was an old version. So old SDKs can parse from newer build.dart.
• Being able to parse older build_output.yamls. So new SDKs can parse from older build.dart.

Suggested review order:

1. The public API for build.dart writers: pkgs/native_assets_cli/lib/src/api/* + pkgs/native_assets_cli/example/*.
2. The implementation of the public API in pkgs/native_assets_cli/lib/src/model/*.
3. Backwards compatibility in the protocol in pkgs/native_assets_cli/lib/src/model/build_config_CHANGELOG.md and pkgs/native_assets_cli/lib/src/model/build_output_CHANGELOG.md.
4. The changes in the other packages.

This PR will need manual rolling:

• The native_assets_cli_internal classes are now renamed to Impl, this will need to be taken into account when rolling.
• Some of the constructor parameters and fields of classes are changed.

Related issues:

• ☂️ [native_assets_cli] Overhaul dart API for build.dart #882
• ☂️ [native_assets_cli] Protocol overhaul #992
• [native_assets_cli] data assets #154

Closes: #1001
Closes: #188
Closes: #25

[github-actions]

PR Health

Breaking changes ✔️

Details

Package	Change	Current Version	New Version	Needed Version	Looking good?
native_assets_cli	Breaking	0.4.3-wip	0.5.0-wip	0.5.0-wip	✔️

Changelog Entry ✔️

Details

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

Coverage ⚠️

Details

File	Coverage
pkgs/native_assets_cli/example/local_asset/build.dart	💔 Not covered
pkgs/native_assets_cli/example/native_add_library/build.dart	💔 Not covered
pkgs/native_assets_cli/example/use_dart_api/build.dart	💔 Not covered
pkgs/native_assets_cli/lib/native_assets_cli.dart	💔 Not covered
pkgs/native_assets_cli/lib/native_assets_cli_internal.dart	💔 Not covered
pkgs/native_assets_cli/lib/src/api/architecture.dart	💚 100 %
pkgs/native_assets_cli/lib/src/api/asset.dart	💔 Not covered
pkgs/native_assets_cli/lib/src/api/build.dart	💚 100 %
pkgs/native_assets_cli/lib/src/api/build_config.dart	💚 100 %
pkgs/native_assets_cli/lib/src/api/build_mode.dart	💔 Not covered
pkgs/native_assets_cli/lib/src/api/build_output.dart	💚 100 %
pkgs/native_assets_cli/lib/src/api/c_compiler_config.dart	💔 Not covered
pkgs/native_assets_cli/lib/src/api/data_asset.dart	💚 100 %
pkgs/native_assets_cli/lib/src/api/ios_sdk.dart	💔 Not covered
pkgs/native_assets_cli/lib/src/api/link_mode_preference.dart	💔 Not covered
pkgs/native_assets_cli/lib/src/api/native_code_asset.dart	💚 100 %
pkgs/native_assets_cli/lib/src/api/os.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/architecture.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/asset.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/build_config.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/build_mode.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/build_output.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/c_compiler_config.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/data_asset.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/dependencies.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/ios_sdk.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/link_mode_preference.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/metadata.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/native_code_asset.dart	💚 95 %
pkgs/native_assets_cli/lib/src/model/os.dart	💚 100 %
pkgs/native_assets_cli/lib/src/model/target.dart	💚 100 %

This check for test coverage is informational (issues shown here will not fail the PR).

This check can be disabled by tagging the PR with skip-coverage-check

License Headers ✔️

Details

// Copyright (c) 2024, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
no missing headers

All source files should start with a license header.

Unrelated files missing license headers

Files
pkgs/ffi/example/main.dart
pkgs/ffigen/example/libclang-example/generated_bindings.dart
pkgs/ffigen/example/objective_c/avf_audio_bindings.dart
pkgs/ffigen/example/shared_bindings/generate.dart
pkgs/ffigen/example/shared_bindings/lib/generated/a_gen.dart
pkgs/ffigen/example/shared_bindings/lib/generated/a_shared_b_gen.dart
pkgs/ffigen/example/shared_bindings/lib/generated/base_gen.dart
pkgs/ffigen/example/simple/generated_bindings.dart
pkgs/ffigen/example/swift/swift_api_bindings.dart
pkgs/ffigen/lib/src/config_provider/config_spec.dart
pkgs/ffigen/lib/src/header_parser/clang_bindings/clang_bindings.dart
pkgs/ffigen/test/collision_tests/expected_bindings/_expected_decl_decl_collision_bindings.dart
pkgs/ffigen/test/collision_tests/expected_bindings/_expected_decl_symbol_address_collision_bindings.dart
pkgs/ffigen/test/collision_tests/expected_bindings/_expected_decl_type_name_collision_bindings.dart
pkgs/ffigen/test/collision_tests/expected_bindings/_expected_reserved_keyword_collision_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_comment_markup_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_dart_handle_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_forward_decl_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_functions_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_imported_types_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_native_func_typedef_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_opaque_dependencies_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_packed_structs_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_regress_384_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_struct_fptr_fields_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_typedef_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_unions_bindings.dart
pkgs/ffigen/test/header_parser_tests/expected_bindings/_expected_varargs_bindings.dart
pkgs/ffigen/test/large_integration_tests/_expected_cjson_bindings.dart
pkgs/ffigen/test/large_integration_tests/_expected_libclang_bindings.dart
pkgs/ffigen/test/large_integration_tests/_expected_sqlite_bindings.dart
pkgs/ffigen/test/native_test/_expected_native_test_bindings.dart
pkgs/jni/lib/src/lang/jcharacter.dart
pkgs/jni/lib/src/third_party/generated_bindings.dart
pkgs/jni/lib/src/third_party/global_env_extensions.dart
pkgs/jni/lib/src/third_party/jni_bindings_generated.dart
pkgs/jnigen/android_test_runner/lib/main.dart
pkgs/jnigen/example/in_app_java/lib/android_utils.dart
pkgs/jnigen/example/kotlin_plugin/example/lib/main.dart
pkgs/jnigen/example/kotlin_plugin/lib/kotlin_bindings.dart
pkgs/jnigen/example/kotlin_plugin/lib/kotlin_plugin.dart
pkgs/jnigen/example/pdfbox_plugin/lib/pdfbox_plugin.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/_init.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/pdmodel/PDDocument.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/pdmodel/PDDocumentInformation.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/pdmodel/_package.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/text/PDFTextStripper.dart
pkgs/jnigen/example/pdfbox_plugin/lib/src/third_party/org/apache/pdfbox/text/_package.dart
pkgs/jnigen/lib/src/bindings/descriptor.dart
pkgs/jnigen/lib/src/elements/elements.g.dart
pkgs/jnigen/test/jackson_core_test/third_party/c_based/dart_bindings/com/fasterxml/jackson/core/_package.dart
pkgs/jnigen/test/jackson_core_test/third_party/dart_only/dart_bindings/com/fasterxml/jackson/core/_package.dart
pkgs/jnigen/tool/command_runner.dart

Package publish validation ✔️

Details

Package	Version	Status
package:ffi	2.1.2	already published at pub.dev
package:ffigen	12.0.0-wip	WIP (no publish necessary)
package:jni	0.8.0-wip	WIP (no publish necessary)
package:jnigen	0.8.0-wip	WIP (no publish necessary)
package:native_assets_cli	0.5.0-wip	WIP (no publish necessary)

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

[dcharkes]

@lrhn You suggested adding a BuildStuff, I've named it BuildState now. It could also be AssetBuild with one static method AssetBuild.run. Also I'm not entirely sure that adding methods to BuildState that just forward to output is the best thing to do. Maybe .output.addAssets(...) is just fine. It reduces clutter in the API.

@mkustermann This PR (finally) addresses #3 (comment). PTAL.

@mosuem PTAL.

I'll probably shoot in a meeting to discuss the design as well.

[mkustermann]

My original suggestion was to make config & output extensible tree structures, e.g.:

///// Core API (possibly in one package)

// An object operating on a JSON-like data structure.
class Config {
  final Map map;
}
// An object operating on a JSON-like data structure.
class Output {
  final Map map;
}

// Basic things that every config should have, e.g. what assets the
// `dart build` / `flutter build` wants.
// Avoid using an enum, as we may add more later.
extension on Config {
  GeneralConfig get general => GeneralConfig(map['general'] ?? {});
}

extension type GeneralConfig(Map map) {
  String outputPath => map['output'];
  Set<String> get assetTypes => Set(map['assetTypes']!);
}


///// Possibly in a native assets package:

// Extend the configuration with native library specific settings
extension on Config {
  bool get buildNative => general.assetTypes.contains('native');
  NativeConfig get native => NativeConfig(map.putIfAbsent('native', () => []));
}
// Provide access to abi / cflags / paths to compiler/linker/...
extension type NativeConfig(Map map) {
  Abi get abi => Abi(map['abi']);
}
// Extend the output to support adding native assets.
extension on Output {
  NativeAssets get native => NativeAssets(map.putIfAbsent('nativeAssets', () => {}));
}
// Have simple methods that build up the native assets.
extension type NativeAssets(Map map) {
  void addSystemInstalledLibrary(String assetId, Abi abi, ...) { ... }
}


///// Possibly in a android assets package:

// Extend the configuration with android specifics.
extension on Config {
  bool get buildAndroid => assetTypes.contains('android');
  AndroidConfig get android => AndroidConfig(map.putIfAbsent('android', () => []));
}
extension type AndroidConfig(Map map) {
  int get apiLevel => Abi(map['apiLevel']);
}
// Extend the output to support adding native assets.
extension on Output {
  AndroidAssets get androidAssets => AndroidAssets(map.putIfAbsent('androidAssets', () => {}));
}
// Have simple methods that build up the android-specific assets.
extension type AndroidAssets(Map map) {
  void addJarFile(...) { ... }
  void addProguardRules(...) { ... }
}

That would allow adding more asset types later on (e.g. jar files, images, ...) by

• adding a new asset type name (we document what asset kinds are supported)
• adding more packages that add extensions on the Config & Output (we document the json structure for the config & output)

So the minimal one needs is a 3 liner to get started with native assets:

main(args) {
  final (config, output) = loadConfig(args);
  output.native.addSystemInstalledLibrary('<asset-id>', 'libexif.so');
  writeOutput(config.outputPath, output.map);
}

or

main(args) async {
  await build(args, (config, output) {
    output.native.addSystemInstalledLibrary('<asset-id>', 'libexif.so');
  });
}

By not having a monolithic BuildState - which is just a wrapper around config and output, one can structure helper packages differently

main(args) async {
  await build(args, (config, output) {
    if (config.buildNative) {
      CBuilder(config.general, config.native, output.native).buildLibrary('<asset-id>', 'foo.c');
    }
    if (config.buildAndroid) {
      AndroidBuilder(config.general, config.android, output.android).addJar(...);
    }
  });
}

i.e. one can pass the sub-parts of Config and Output that are relevant for that asset type down to the CBuilder (which doesn't need access to config.android / output.android) and AndroidBuilder (which doesn't need access to config.native, output.native).

[dcharkes]

That would allow adding more asset types later on (e.g. jar files, images, ...) by

• adding a new asset type name (we document what asset kinds are supported)
• adding more packages that add extensions on the Config & Output (we document the json structure for the config & output)

Adding the assets in extensions in different packages would mean that we would not share the serializing/deserializing code between the flutter_tools/dartdev and build.dart scripts. And that native_assets_cli does not need to have a minor version increase. This is in the same spirit "Stop letting native_assets_builder depend on native_assets_cli." in #949 but splits BuildConfig deserializing and BuildOutput serializing over multiple packages instead of only native_assets_cli.

However, I think the API becomes harder to discover. One would have to know which helper packages there, add them to the pubspec, pub get and import them in their build.dart file. This is opposed to having addAsset or addAssets and being able to click on the API docs of abstract class Asset and see all the concrete implementations: CodeAsset, DataAsset, ProguardRulesAsset.

main(args) async {
  await build(args, (config, output) {

👍 Applied, I've removed BuildState again.

Note that that means we're passing in config and output as two objects to most helper methods (CBuilder, AssetDownloader).

output.native.addSystemInstalledLibrary('<asset-id>', 'libexif.so');

Note from @lrhn: He's worried when he sees addX( instead of add(X. So we should probably stick to having an addAsset method and constructors for the different asset types.

Thanks for your input so far! I've scheduled a follow up discussion!

[coveralls]

coverage: 92.613% (-0.07%) from 92.684%
when pulling 8dc1ef7 on native-assets-cli-build-output
into afa1cce on main.

[dcharkes]

Thanks @mosuem! I like the new shorter names!

Happy to merge this if you are happy with this!