Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native assets support for MacOS and iOS #130494

Merged
merged 150 commits into from Sep 10, 2023
Merged

Conversation

dcharkes
Copy link
Contributor

@dcharkes dcharkes commented Jul 13, 2023

Support for FFI calls with @Native external functions through Native assets on MacOS and iOS. This enables bundling native code without any build-system boilerplate code.

For more info see:

Implementation details for MacOS and iOS.

Dylibs are bundled by (1) making them fat binaries if multiple architectures are targeted, (2) code signing these, and (3) copying them to the frameworks folder. These steps are done manual rather than via CocoaPods. CocoaPods would have done the same steps, but (a) needs the dylibs to be there before the xcodebuild invocation (we could trick it, by having a minimal dylib in the place and replace it during the build process, that works), and (b) can't deal with having no dylibs to be bundled (we'd have to bundle a dummy dylib or include some dummy C code in the build file).

The dylibs are build as a new target inside flutter assemble, as that is the moment we know what build-mode and architecture to target.

The mapping from asset id to dylib-path is passed in to every kernel compilation path. The interesting case is hot-restart where the initial kernel file is compiled by the "inner" flutter assemble, while after hot restart the "outer" flutter run compiled kernel file is pushed to the device. Both kernel files need to contain the mapping. The "inner" flutter assemble gets its mapping from the NativeAssets target which builds the native assets. The "outer" flutter run get its mapping from a dry-run invocation. Since this hot restart can be used for multiple target devices (flutter run -d all) it contains the mapping for all known targets.

Example vs template

The PR includes a new template that uses the new native assets in a package and has an app importing that. Separate discussion in: #131209.

Tests

This PR adds new tests to cover the various use cases.

  • dev/devicelab/bin/tasks/native_assets_ios.dart
    • Runs an example app with native assets in all build modes, doing hot reload and hot restart in debug mode.
  • dev/devicelab/bin/tasks/native_assets_ios_simulator.dart
    • Runs an example app with native assets, doing hot reload and hot restart.
  • packages/flutter_tools/test/integration.shard/native_assets_test.dart
    • Runs (incl hot reload/hot restart), builds, builds frameworks for iOS, MacOS and flutter-tester.
  • packages/flutter_tools/test/general.shard/build_system/targets/native_assets_test.dart
    • Unit tests the new Target in the backend.
  • packages/flutter_tools/test/general.shard/ios/native_assets_test.dart
  • packages/flutter_tools/test/general.shard/macos/native_assets_test.dart
    • Unit tests the native assets being packaged on a iOS/MacOS build.

It also extends various existing tests:

  • dev/devicelab/bin/tasks/module_test_ios.dart
    • Exercises the add2app scenario.
  • packages/flutter_tools/test/general.shard/features_test.dart
    • Unit test the new feature flag.

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

@github-actions github-actions bot added platform-ios iOS applications specifically tool Affects the "flutter" command-line tool. See also t: labels. labels Jul 13, 2023
@dcharkes dcharkes added the platform-mac Building on or for macOS specifically label Jul 14, 2023
@dcharkes dcharkes marked this pull request as ready for review July 15, 2023 07:04
@github-actions github-actions bot added the framework flutter/packages/flutter repository. See also f: labels. label Jul 20, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Sep 10, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Sep 10, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Sep 11, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Sep 11, 2023
auto-submit bot pushed a commit to flutter/packages that referenced this pull request Sep 11, 2023
flutter/flutter@7c28e8e...219efce

2023-09-11 polinach@google.com Cover more tests with leak tracking. (flutter/flutter#134363)
2023-09-11 engine-flutter-autoroll@skia.org Roll Flutter Engine from e9f1696e5060 to 2b5961cbc40d (1 revision) (flutter/flutter#134424)
2023-09-11 engine-flutter-autoroll@skia.org Roll Flutter Engine from ac6be4dbf821 to e9f1696e5060 (1 revision) (flutter/flutter#134410)
2023-09-11 engine-flutter-autoroll@skia.org Roll Flutter Engine from 1aa11b644855 to ac6be4dbf821 (1 revision) (flutter/flutter#134407)
2023-09-11 engine-flutter-autoroll@skia.org Roll Flutter Engine from 7e41b064eb23 to 1aa11b644855 (2 revisions) (flutter/flutter#134404)
2023-09-11 engine-flutter-autoroll@skia.org Roll Flutter Engine from 04d836f69571 to 7e41b064eb23 (1 revision) (flutter/flutter#134396)
2023-09-10 engine-flutter-autoroll@skia.org Roll Flutter Engine from 2acd60d6baed to 04d836f69571 (1 revision) (flutter/flutter#134392)
2023-09-10 engine-flutter-autoroll@skia.org Roll Flutter Engine from 59b08263f7cc to 2acd60d6baed (1 revision) (flutter/flutter#134391)
2023-09-10 engine-flutter-autoroll@skia.org Roll Flutter Engine from a2af0c6965be to 59b08263f7cc (1 revision) (flutter/flutter#134384)
2023-09-10 engine-flutter-autoroll@skia.org Roll Flutter Engine from 47d1a045d140 to a2af0c6965be (1 revision) (flutter/flutter#134375)
2023-09-10 dacoharkes@google.com Native assets support for MacOS and iOS (flutter/flutter#130494)
2023-09-10 engine-flutter-autoroll@skia.org Roll Flutter Engine from feea98d6d0ae to 47d1a045d140 (1 revision) (flutter/flutter#134368)
2023-09-10 engine-flutter-autoroll@skia.org Roll Flutter Engine from 4edbefa5e8f7 to feea98d6d0ae (1 revision) (flutter/flutter#134365)
2023-09-09 rmolivares@renzo-olivares.dev SelectableRegion onSelectionChange should be called when the selection changes (flutter/flutter#134215)
2023-09-09 engine-flutter-autoroll@skia.org Roll Flutter Engine from b8afb6deacf3 to 4edbefa5e8f7 (1 revision) (flutter/flutter#134360)
2023-09-09 engine-flutter-autoroll@skia.org Roll Flutter Engine from d1913cb6a276 to b8afb6deacf3 (1 revision) (flutter/flutter#134356)

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-packages
Please CC camillesimon@google.com,rmistry@google.com,stuartmorgan@google.com on the revert to ensure that a human
is aware of the problem.

To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
auto-submit bot pushed a commit that referenced this pull request Sep 14, 2023
Template plugin_ffi uses FFIgen and generates both the FFIgen inputs and the generated file.

We rolled FFIgen to 9.0.0 in #130494, which means a slight change to the generated file.

* https://github.com/dart-lang/ffigen/issues/619

Note, because of #105695, we run the test on the FFIgen repo rather than on the flutter CI.
auto-submit bot pushed a commit that referenced this pull request Sep 18, 2023
Support for FFI calls with `@Native external` functions through Native assets on Linux. This enables bundling native code without any build-system boilerplate code.

For more info see:

* #129757

### Implementation details for Linux.

Mainly follows the design of #130494.

Some differences are:

* Linux does not support cross compiling or compiling for multiple architectures, so this has not been implemented.
* Linux has no add2app.

The assets copying is done in the install-phase of the CMake build of a flutter app.
CMake requires the native assets folder to exist, so we create it also when the feature is disabled or there are no assets.

### Tests

This PR adds new tests to cover the various use cases.

* packages/flutter_tools/test/general.shard/linux/native_assets_test.dart
  * Unit tests the Linux-specific part of building native assets.

It also extends various existing tests:

* packages/flutter_tools/test/integration.shard/native_assets_test.dart
  * Runs (incl hot reload/hot restart), builds, builds frameworks for Linux and flutter-tester.
auto-submit bot pushed a commit that referenced this pull request Sep 19, 2023
Move test added in #130494 out of staging.

* https://ci.chromium.org/ui/p/flutter/builders/luci.flutter.staging/Mac_ios%20native_assets_ios
* https://ci.chromium.org/ui/p/flutter/builders/luci.flutter.staging/Mac_ios%20native_assets_ios_simulator

> Monitor the CI results of the new shard on the [Flutter build dashboard](https://flutter-dashboard.appspot.com/#/build). After 50 consecutive passing builds without any flakes, the flake bot will create a PR to remove the bringup: true parameter from .ci.yaml in the Framework tree. This will allow the test to block the tree, preventing breakages. With this change, the new shard will start running in presubmit automatically, unless specify presubmit: false.

https://github.com/flutter/flutter/wiki/Adding-a-new-Test-Shard#steps-to-add-a-new-framework-test-shard

Umbrella issue:

* #129757
auto-submit bot pushed a commit that referenced this pull request Sep 22, 2023
Reland of #134031. (Reverted in #135069.) Contains the fix for b/301051367 together with cl/567233346.

Support for FFI calls with `@Native external` functions through Native assets on Linux. This enables bundling native code without any build-system boilerplate code.

For more info see:

* #129757

### Implementation details for Linux.

Mainly follows the design of #130494.

Some differences are:

* Linux does not support cross compiling or compiling for multiple architectures, so this has not been implemented.
* Linux has no add2app.

The assets copying is done in the install-phase of the CMake build of a flutter app.
CMake requires the native assets folder to exist, so we create it also when the feature is disabled or there are no assets.

### Tests

This PR adds new tests to cover the various use cases.

* packages/flutter_tools/test/general.shard/linux/native_assets_test.dart
  * Unit tests the Linux-specific part of building native assets.

It also extends various existing tests:

* packages/flutter_tools/test/integration.shard/native_assets_test.dart
  * Runs (incl hot reload/hot restart), builds, builds frameworks for Linux and flutter-tester.
Mairramer pushed a commit to Mairramer/flutter that referenced this pull request Oct 10, 2023
Template plugin_ffi uses FFIgen and generates both the FFIgen inputs and the generated file.

We rolled FFIgen to 9.0.0 in flutter#130494, which means a slight change to the generated file.

* https://github.com/dart-lang/ffigen/issues/619

Note, because of flutter#105695, we run the test on the FFIgen repo rather than on the flutter CI.
Mairramer pushed a commit to Mairramer/flutter that referenced this pull request Oct 10, 2023
Support for FFI calls with `@Native external` functions through Native assets on Linux. This enables bundling native code without any build-system boilerplate code.

For more info see:

* flutter#129757

### Implementation details for Linux.

Mainly follows the design of flutter#130494.

Some differences are:

* Linux does not support cross compiling or compiling for multiple architectures, so this has not been implemented.
* Linux has no add2app.

The assets copying is done in the install-phase of the CMake build of a flutter app.
CMake requires the native assets folder to exist, so we create it also when the feature is disabled or there are no assets.

### Tests

This PR adds new tests to cover the various use cases.

* packages/flutter_tools/test/general.shard/linux/native_assets_test.dart
  * Unit tests the Linux-specific part of building native assets.

It also extends various existing tests:

* packages/flutter_tools/test/integration.shard/native_assets_test.dart
  * Runs (incl hot reload/hot restart), builds, builds frameworks for Linux and flutter-tester.
Mairramer pushed a commit to Mairramer/flutter that referenced this pull request Oct 10, 2023
Move test added in flutter#130494 out of staging.

* https://ci.chromium.org/ui/p/flutter/builders/luci.flutter.staging/Mac_ios%20native_assets_ios
* https://ci.chromium.org/ui/p/flutter/builders/luci.flutter.staging/Mac_ios%20native_assets_ios_simulator

> Monitor the CI results of the new shard on the [Flutter build dashboard](https://flutter-dashboard.appspot.com/#/build). After 50 consecutive passing builds without any flakes, the flake bot will create a PR to remove the bringup: true parameter from .ci.yaml in the Framework tree. This will allow the test to block the tree, preventing breakages. With this change, the new shard will start running in presubmit automatically, unless specify presubmit: false.

https://github.com/flutter/flutter/wiki/Adding-a-new-Test-Shard#steps-to-add-a-new-framework-test-shard

Umbrella issue:

* flutter#129757
Mairramer pushed a commit to Mairramer/flutter that referenced this pull request Oct 10, 2023
Reland of flutter#134031. (Reverted in flutter#135069.) Contains the fix for b/301051367 together with cl/567233346.

Support for FFI calls with `@Native external` functions through Native assets on Linux. This enables bundling native code without any build-system boilerplate code.

For more info see:

* flutter#129757

### Implementation details for Linux.

Mainly follows the design of flutter#130494.

Some differences are:

* Linux does not support cross compiling or compiling for multiple architectures, so this has not been implemented.
* Linux has no add2app.

The assets copying is done in the install-phase of the CMake build of a flutter app.
CMake requires the native assets folder to exist, so we create it also when the feature is disabled or there are no assets.

### Tests

This PR adds new tests to cover the various use cases.

* packages/flutter_tools/test/general.shard/linux/native_assets_test.dart
  * Unit tests the Linux-specific part of building native assets.

It also extends various existing tests:

* packages/flutter_tools/test/integration.shard/native_assets_test.dart
  * Runs (incl hot reload/hot restart), builds, builds frameworks for Linux and flutter-tester.
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Nov 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a: desktop Running on desktop a: tests "flutter test", flutter_test, or one of our tests autosubmit Merge PR when tree becomes green via auto submit App d: api docs Issues with https://api.flutter.dev/ d: examples Sample code and demos f: integration_test The flutter/packages/integration_test plugin platform-ios iOS applications specifically platform-mac Building on or for macOS specifically tool Affects the "flutter" command-line tool. See also t: labels.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants