Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to strip build-id and embeded path from the artifact? #52506

Closed
linsui opened this issue May 24, 2023 · 15 comments
Closed

How to strip build-id and embeded path from the artifact? #52506

linsui opened this issue May 24, 2023 · 15 comments
Labels
area-vm Use area-vm for VM related issues, including code coverage, FFI, and the AOT and JIT backends.

Comments

@linsui
Copy link

linsui commented May 24, 2023

Thank you for taking the time to file an issue!

This tracker is for issues related to:

  • dev_compiler

The .so files produced by flutter (libapp.so) embed some paths of the build environment and the build-id. Some other languages provide options to remove them so that we can make the artifacts reproducible. If there any methed to strip the path and the build-id? How can I pass an argument to the linker so that it doesn't add the build-id (e.g. -Wl,--build-id=none)?

Thanks!
@lrhn lrhn added the area-vm Use area-vm for VM related issues, including code coverage, FFI, and the AOT and JIT backends. label May 24, 2023
@mkustermann
Copy link
Member

We intentionally want to embed a build-id in order to allow stripping symbolic information from the libapp.so. The build-id is then used to identify which separate symbol file to use for symbolization of stack traces. Though the calculation of build-id is based on hashing of certain other parts of the ELF file - so if other parts haven't changed, the build-id shouldn't change.

Some other languages provide options to remove them so that we can make the artifacts reproducible.

@linsui could you elaborate what makes your builds non-reproducible?

/cc @sstrickl @mraleph

@linsui
Copy link
Author

linsui commented May 25, 2023

(Background: F-Droid builds apks on a buildserver and compares them with the apks published by upstream devs.) We need to make the apk built on different machines/environments reproducible. When the Android NDK is installed in different path, the build-id is different. The path of the source code is also embeded leading to different apks.

@mkustermann
Copy link
Member

AFAIK by default the paths that are used to identify dart code should use import uris (e.g. package:flutter/.../foo.dart). Only if one is opting into resolving those paths to absolute paths via --extra-gen-snapshot-options=resolve-dwarf-paths would one get the paths on disc.

@linsui Could you provide us with the paths that you see in the libapp.so file?

@linsui
Copy link
Author

linsui commented May 25, 2023

This is the diffoscope output of one of the apps. You can see something like

file:///builds/bleonard252/fdroiddata/build/xyz.u1024256.bodacious.fdroid/.dart_tool/flutter_build/dart_plugin_registrant.dart and file:///src/bleonard252/bodacious/.dart_tool/flutter_build/dart_plugin_registrant.dart in the diff.

@mkustermann
Copy link
Member

Thanks, that's very helpful!

That's a very special file. It's neither included in the Dart app nor any dependent packages. Rather it's an artificially created file by the flutter tools. It contains logic to run plugin registration logic.

A flutter build will eventually compile the Dart application where it will add <dir>/.dart_tool/flutter_build/dart_plugin_registrant.dart as an extra source file (see here). Additionally it will also inject that uri as a constant into Dart source code via a -Dflutter.dart_plugin_registrant=<uri> - see here.

=> So we'll have that string twice in the app - once as the name of the Dart VM Library object representing the Dart file and once the Dart constant.

Once the app runs it will access that package:flutter/src/dart_plugin_registrant.dart:dartPluginRegistrantLibrary constant and use it to look up the library object and then invoke the plugin registration logic - see engine code.

There's several ways one could fix this:

  • to use a relative path (e.g. .dart_tool/flutter_build/dart_plugin_registrant.dart) instead of an absolute one
  • use custom scheme to refer to generated dart code

/cc @zanderso is that something the flutter tools team could look into?

@linsui
Copy link
Author

linsui commented May 25, 2023

Thanks!

This is an example of the build id problem. We finally solved it by using the same NDK path.

@zanderso
Copy link
Member

cc @christopherfujino and @chingjun may have the most recent context on the dart_plugin_registrant.dart import path.

@christopherfujino
Copy link
Member

christopherfujino commented May 25, 2023 •
edited

cc @christopherfujino and @chingjun may have the most recent context on the dart_plugin_registrant.dart import path.

I don't have recent context, making hte import path relative SGTM. Although, would that even solve your problem @linsui, if I'm reading your diff correctly, it seems like there are still a lot of other differences.

@linsui
Copy link
Author

linsui commented May 26, 2023

Although, would that even solve your problem @linsui, if I'm reading your diff correctly, it seems like there are still a lot of other differences.

I'm not sure. When we used the same path to build the apk, other differences are also fixed.

@linsui
Copy link
Author

linsui commented Oct 24, 2023

We got another problem in a flutter app. The path of the shaders/ink_sparkle.frag is embeded. Not sure should I report it here. Should I open another issue in the flutter repo?

@mraleph
Copy link
Member

mraleph commented Oct 25, 2023

@linsui shaders are not a thing Dart knows about, it's Flutter tooling related stuff, so yeah please do file another issue

@a-siva
Copy link
Contributor

a-siva commented Dec 7, 2023

This seems like a Flutter issue and so closing this here,

@a-siva a-siva closed this as completed Dec 7, 2023
@linsui
Copy link
Author

linsui commented Dec 23, 2023 •
edited

I opened an issue in flutter issue tracker for the embeded path. flutter/flutter#140558 I thought the build-id problem is still a dart issue?

@a-siva a-siva mentioned this issue Jan 5, 2024
Open
arnout pushed a commit to buildroot/buildroot that referenced this issue Jan 20, 2024
@yann-morin-1998
package/flutter-gallery: fix build options
a821aee 
As the flutter-gallery package is a reference package for users wishing to use
Flutter for their UX with Buildroot, this package must have the correct build
options. Indeed, this package currently starts and runs, but only because of
the 0001-remove-GetStorage.patch. Through testing, flutter-gallery fails to
run during the following scenario:
  - The xdg-user-dirs package is ported and present.
  - flutter-gallery depends on xdg-user-dirs.
  - The 0001-remove-GetStorage.patch file is removed.

After extensive testing and comparing the current build arguments against what
the meta-flutter repository for Yocto passes to all of the applications that
inherit flutter-app, it is clear that handling the dart_plugin_registrant.dart
file is missing from the dart arguments in the flutter-gallery build step.

As the documentation for the dart_plugin_registrant.dart file is nonexistent
in any official documentation. However, there is a comment from an issue on
the official dart-lang/sdk page on Github that explains what this file is
(and refers to the Dark SDK source code instead of official documentation.)

From dart-lang/sdk#52506 (comment):
```
The dart_plugin_registrant.dart is a very special file. It's neither included
in the Dart app nor any dependent packages. Rather it's an artificially
created file by the flutter tools. It contains logic to run plugin
registration logic.

A flutter build will eventually compile the Dart application where it will add
<dir>/.dart_tool/flutter_build/dart_plugin_registrant.dart as an extra source
file (see here). Additionally it will also inject that uri as a constant into
Dart source code via a -Dflutter.dart_plugin_registrant=<uri>.

Once the app runs it will access the
package:flutter/src/dart_plugin_registrant.dart:dartPluginRegistrantLibrary
constant and use it to look up the library object and then invoke the plugin
registration logic.
```

Now that what the dart_plugin_registrant.dart does is understood, we need to
pass the following to the dart binary during the flutter-gallery build step:

  -Dflutter.dart_plugin_registrant=file://[...]/dart_plugin_registrant.dart:
   Injects a file containing the logic to run the plugin registration logic as
   a constant into the flutter-application source code.

  --source file://$(@d)/.dart_tool/flutter_build/dart_plugin_registrant.dart:
    Adds the dart_plugin_registrant.dart file as a source file to compile.

  --source package:flutter/src/dart_plugin_registrant.dart:
    Binds the plugin implementation to the platform interface based on the
    configuration of the app's pubpec.yaml, and the plugin's pubspec.yaml.

The native_assets.yaml file provides the native-assets mapping for
@Native external functions. The flutter-gallery package has no functions
marked as @Native; however, calling "flutter build bundle" creates a blank
template "native_assets.yaml" file, which is safe to include in the build.
This line, while not necessary for flutter-gallery, may be helpful for other
users who use @Native external functions in their applications, and this
example makes porting other applications quicker and easier.

Finally, there is a known issue when using the dart_plugin_registrant.dart
file outlined here: flutter/flutter#137972.

To summarize: If a user fails to pass the --obfuscate flag to gen_snapshsot
when using the dart_plugin_registrant.dart file, their application may fail
to start. One such application is Gallery, which I have independently verified.

As such, pass the --obfuscate flag to gen_snapshot to ensure that
flutter-gallery properly starts when building with the additional
dart_plugin_registrant.dart arguments above.

However, I acknowledge that the obfuscate flag hides function and class names
in compiled Dart code, and there are some cases when a user should avoid using
the flag. For example, when using the runtimeType API:
https://api.flutter.dev/flutter/dart-core/Object/runtimeType.html. However,
this is not the case with flutter-gallery, and the --obfuscate flag is needed.

Signed-off-by: Adam Duskett <adam.duskett@amarulasolutions.com>
[yann.morin.1998@free.fr: restore FLUTTER_RUNTIME_MODES]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
arnout pushed a commit to buildroot/buildroot that referenced this issue Feb 3, 2024
@jacmet
package/flutter-gallery: fix build options
1d13497 
As the flutter-gallery package is a reference package for users wishing to use
Flutter for their UX with Buildroot, this package must have the correct build
options. Indeed, this package currently starts and runs, but only because of
the 0001-remove-GetStorage.patch. Through testing, flutter-gallery fails to
run during the following scenario:
  - The xdg-user-dirs package is ported and present.
  - flutter-gallery depends on xdg-user-dirs.
  - The 0001-remove-GetStorage.patch file is removed.

After extensive testing and comparing the current build arguments against what
the meta-flutter repository for Yocto passes to all of the applications that
inherit flutter-app, it is clear that handling the dart_plugin_registrant.dart
file is missing from the dart arguments in the flutter-gallery build step.

As the documentation for the dart_plugin_registrant.dart file is nonexistent
in any official documentation. However, there is a comment from an issue on
the official dart-lang/sdk page on Github that explains what this file is
(and refers to the Dark SDK source code instead of official documentation.)

From dart-lang/sdk#52506 (comment):
```
The dart_plugin_registrant.dart is a very special file. It's neither included
in the Dart app nor any dependent packages. Rather it's an artificially
created file by the flutter tools. It contains logic to run plugin
registration logic.

A flutter build will eventually compile the Dart application where it will add
<dir>/.dart_tool/flutter_build/dart_plugin_registrant.dart as an extra source
file (see here). Additionally it will also inject that uri as a constant into
Dart source code via a -Dflutter.dart_plugin_registrant=<uri>.

Once the app runs it will access the
package:flutter/src/dart_plugin_registrant.dart:dartPluginRegistrantLibrary
constant and use it to look up the library object and then invoke the plugin
registration logic.
```

Now that what the dart_plugin_registrant.dart does is understood, we need to
pass the following to the dart binary during the flutter-gallery build step:

  -Dflutter.dart_plugin_registrant=file://[...]/dart_plugin_registrant.dart:
   Injects a file containing the logic to run the plugin registration logic as
   a constant into the flutter-application source code.

  --source file://$(@d)/.dart_tool/flutter_build/dart_plugin_registrant.dart:
    Adds the dart_plugin_registrant.dart file as a source file to compile.

  --source package:flutter/src/dart_plugin_registrant.dart:
    Binds the plugin implementation to the platform interface based on the
    configuration of the app's pubpec.yaml, and the plugin's pubspec.yaml.

The native_assets.yaml file provides the native-assets mapping for
@Native external functions. The flutter-gallery package has no functions
marked as @Native; however, calling "flutter build bundle" creates a blank
template "native_assets.yaml" file, which is safe to include in the build.
This line, while not necessary for flutter-gallery, may be helpful for other
users who use @Native external functions in their applications, and this
example makes porting other applications quicker and easier.

Finally, there is a known issue when using the dart_plugin_registrant.dart
file outlined here: flutter/flutter#137972.

To summarize: If a user fails to pass the --obfuscate flag to gen_snapshsot
when using the dart_plugin_registrant.dart file, their application may fail
to start. One such application is Gallery, which I have independently verified.

As such, pass the --obfuscate flag to gen_snapshot to ensure that
flutter-gallery properly starts when building with the additional
dart_plugin_registrant.dart arguments above.

However, I acknowledge that the obfuscate flag hides function and class names
in compiled Dart code, and there are some cases when a user should avoid using
the flag. For example, when using the runtimeType API:
https://api.flutter.dev/flutter/dart-core/Object/runtimeType.html. However,
this is not the case with flutter-gallery, and the --obfuscate flag is needed.

Signed-off-by: Adam Duskett <adam.duskett@amarulasolutions.com>
[yann.morin.1998@free.fr: restore FLUTTER_RUNTIME_MODES]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
(cherry picked from commit a821aee)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
@a-siva
Copy link
Contributor

a-siva commented Apr 17, 2024

I opened an issue in flutter issue tracker for the embeded path. flutter/flutter#140558 I thought the build-id problem is still a dart issue?

build ids are different because of the differing paths which the flutter issue is tracking.

@linsui
Copy link
Author

linsui commented Apr 18, 2024

build ids are different because of the differing paths which the flutter issue is tracking.

In https://gitlab.com/fdroid/fdroiddata/-/merge_requests/14837 the libflutter.so files have different ndk build-id. This shouldn't be related to the embeded path which is in libapp.so. Any idea why the ndk build-id is different?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-vm Use area-vm for VM related issues, including code coverage, FFI, and the AOT and JIT backends.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@mraleph @lrhn @mkustermann @zanderso @christopherfujino @a-siva @linsui