Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native executables created with dart2native do not support signing #39106

Open
mit-mit opened this issue Oct 25, 2019 · 37 comments
Open

Native executables created with dart2native do not support signing #39106

mit-mit opened this issue Oct 25, 2019 · 37 comments

Comments

@mit-mit
Copy link
Member

@mit-mit mit-mit commented Oct 25, 2019

NOTE: This issue specifically refers to dart2native tool, it does not refer to Flutter on Desktop, as Flutter does not use dart2native

The executables created with bin/dart2native use a format that as discussed in the original issue is not compatible with signing tools such as codesign and signtool.

@anaisbetts
Copy link

@anaisbetts anaisbetts commented Nov 6, 2019

Confirmed this on Windows 10 - signing a binary with signtool on Windows will cause it (strangely enough) to act like dart.exe

C:\Users\ana\code> cat .\test.dart
main() {
        print("Hello world!");
}

C:\Users\ana\code> dart2native -k exe -o test.exe .\test.dart
Generated: c:\users\ana\code\test.exe
C:\Users\ana\code> .\test.exe
Hello world!

C:\Users\ana\code>signtool.exe sign /f mycert.pfx /p sekritPassword .\test.exe
Done Adding Additional Store
Successfully signed: .\test.exe

C:\Users\ana\code> .\test.exe
Usage: dart [<vm-flags>] <dart-script-file> [<script-arguments>]

Executes the Dart script <dart-script-file> with the given list of <script-arguments>.

Common VM flags:
--enable-asserts
  Enable assert statements.
--help or -h
  Display this message (add -v or --verbose for information about
  all VM options).
--package-root=<path> or -p<path>
  Where to find packages, that is, "package:..." imports.

@knopp
Copy link

@knopp knopp commented Nov 12, 2019

Are there any plans of addressing this? Distributing binaries on Mac and Windows that are not codesigned is not feasible (on Windows you will not get through smart screen, on Mac since Catalina you need hardened runtime and notarization, before catalina there's gatekeeper).

@dnfield
Copy link
Contributor

@dnfield dnfield commented Jan 3, 2020

I was hoping to use this to be able to ship a tool from the Flutter engine repo to the Flutter tool. I won't be able to do so without this feature.

@tonosama-atlacatl
Copy link

@tonosama-atlacatl tonosama-atlacatl commented Jan 30, 2020

I WANT IT!!!! I Need IT!! I wrote a tool to organize photos/videos/etc... take it from someone that has spent hours helping others install the SDK over the phone... just so they can test my script.

@mcaubrey
Copy link

@mcaubrey mcaubrey commented Feb 3, 2020

Is this a planned feature? It seems like there's been relatively silence on this for the past few months.

@alexgoussev
Copy link

@alexgoussev alexgoussev commented Mar 3, 2020

I very hope this feature will be implemented in the future! I need it!

@stephantual
Copy link

@stephantual stephantual commented Mar 30, 2020

Adding a vote for this feature, massively important if desktop dart wants to get into the big leagues.

@ghost
Copy link

@ghost ghost commented Jul 31, 2020

Windows desktop apps without signning, most of people or security software will not trust them.
Hope fix it quickly.

@jodinathan
Copy link

@jodinathan jodinathan commented Aug 19, 2020

is there a workaround to this?

@Timmmm
Copy link

@Timmmm Timmmm commented Aug 19, 2020

I WANT IT!!!! I Need IT!! I wrote a tool to organize photos/videos/etc... take it from someone that has spent hours helping others install the SDK over the phone... just so they can test my script.

It is still possible to run unsigned binaries on Mac and Windows. On Mac you have to right-click the App and select Open (rather than double clicking it), and on Windows you have to click "More info" and then "Run anyway". Completely not obvious, but you shouldn't need to install the SDK just to get a friend to run something!

Plenty of free software is still distributed unsigned on both platforms, though it would obviously be great if code signing was supported.

@knopp
Copy link

@knopp knopp commented Aug 19, 2020

I WANT IT!!!! I Need IT!! I wrote a tool to organize photos/videos/etc... take it from someone that has spent hours helping others install the SDK over the phone... just so they can test my script.

It is still possible to run unsigned binaries on Mac and Windows. On Mac you have to right-click the App and select Open (rather than double clicking it), and on Windows you have to click "More info" and then "Run anyway". Completely not obvious, but you shouldn't need to install the SDK just to get a friend to run something!

Plenty of free software is still distributed unsigned on both platforms, though it would obviously be great if code signing was supported.

None of these is feasible. Especially if you want to distribute commercial software to general public.

@Timmmm
Copy link

@Timmmm Timmmm commented Aug 21, 2020

Not for commercial software, no. I just wanted to point out that you don't need to go as far as installing the Dart SDK to work around this!

@etx
Copy link

@etx etx commented Oct 16, 2020

We build a command line app with dart2native the acts as a bridge in Chrome using it's NativeMessaging api. But with the binary in a mac app bundle we can't codesign or notarize 😭

@gkjpettet
Copy link

@gkjpettet gkjpettet commented Oct 30, 2020

This is a compete dealbreaker for our business. How on Earth can one make a commercial product using dart if you can’t code sign?

Has there been any progress here since the issue has been open for a year?

@thunderstorm010
Copy link

@thunderstorm010 thunderstorm010 commented Jul 9, 2021

Is this being worked on? There is nearly 2 years past when this issue was created, and there is no PR still.

@knopp
Copy link

@knopp knopp commented Jul 9, 2021

@thunderstorm010, the comment here would sadly suggest otherwise.

@timotheux
Copy link

@timotheux timotheux commented Jul 10, 2021

So you can't sign the exe, add icon to it, It is not showing any author Metadata like Author, Version, File version, product name, product version etc., which makes it impossible to identify the origin of the exe.

Infact mraleph stop short of calling applications produced by dart2native a virus by saying "executables produced by dart2native are not exactly adhering to OS standard executable format (PE on Windows, Mach-O on Mac OS X or ELF on Linux)".

How did this advance past the prototyping stage?

@maks
Copy link

@maks maks commented Jul 10, 2021

@timotheux just because it doesn't work for your requirements it doesn't follow that the feature isn't useful to others. Dart exe's are already useful for a number of production use cases, especially serverside, internal tooling etc.

Sure it would make it more useful to be able to be able to sign executables produced by dart2native but please don't imply that it's not fit for purpose already for production use in some environments.

@timotheux
Copy link

@timotheux timotheux commented Jul 11, 2021

@maks I am not referring to a use case scenario, a requirement or implying that dart2native is not useful. Lets stay on point here.
I have used dart2native for some project where it performed extremely well in the task but couldn't make it to production because it looks like a suspicious exe.

I started wondering why the Dart team will put in a huge amount of work and leave out the basics. I am simply saying dart2native should not have passed the prototyping stage without an identity. Its like having a car on the road without VIN, Make, Model and paint job.

No country will allow such vehicle on their roads because of the inherent dangers.

@mraleph
Copy link
Member

@mraleph mraleph commented Jul 11, 2021

I suggest to take this discussion somewhere else, because it not relevant for this issue. If you have some concrete use case - it is enough to up vote the issue and leave a comment.

We recognise the need to support the signing on platforms like Mac OS X and Windows, but so far we did not have resources to make necessary changes. I am hopeful that eventually we will find an opening and address the issue, but we have not made any concrete decisions on this yet.

@timotheux

I started wondering why the Dart team will put in a huge amount of work and leave out the basics.

That's where you misunderstand how things went - we did an absolute bare minimum of work to support dart2native, because we did not have resources to spare on something more robust and that's why its implementation is so hacky. We considerably underestimated the actual demand for this feature. We expected the usage mostly on Linux in server-side environments, and not deployments of CLI programs on Windows/Mac.

@timotheux
Copy link

@timotheux timotheux commented Jul 11, 2021

@mraleph I will let this rest for now, so that my intensions are not misunderstood. I speak frankly because I love dartlang and dart2native, and I don't want it to end up like the Qt framework (which I once loved too).

When it comes to the cross platform frameworks out there, dart2native has almost all the right ideas. Hence, my confusion at the lack of basic functions.

I wish you and your team the best and godspeed as you improve the language and framework we all love to use.

Cheers

@illia-romanenko
Copy link

@illia-romanenko illia-romanenko commented Jul 12, 2021

@mraleph, thanks for the insight into your decision-making process and your assumptions about the usage. Also, it's great that you see rising demand in codesigning and the new ways of using dart2native. I'm sure that many people are waiting for this feature, so I hope that managers can prioritize.

In the meantime, could somebody provide a bit more context of that hacky implementation? What is missing, what needs to be done, any relevant documentation (if exists)?

@mraleph
Copy link
Member

@mraleph mraleph commented Jul 13, 2021

@illia-romanenko dart2native is currently implemented in the following way: it takes AOT runtime binary (which is a valid platform executable produced by a native C++ toolchain) and an AOT snapshot (ELF file produced by Dart AOT compiler) and simply concatenates them together. From OS perspective the result is still a valid executable file which simply has a bunch of garbage at the end - but OS can still run it. When AOT runtime binary is run it looks at its own file to see if there is a snapshot appended to it - and if there is then it loads and runs it.

Problems begin when you try to edit the binary using tools which work with native executable formats (e.g. sign it, change an icon, etc). For these tools the snapshot at the end of the file are simply garbage bytes, so they usually just remove them.

To fix the problem we need to start properly linking AOT runtime and AOT snapshot together so that the result is a native binary. The simplest way to achieve might be to take a dependency on an existing linker which supports the platforms we support (lld probably fits the bill), figuring out few missing pieces (e.g. we need to figure out what format of object files to feed to lld and implement necessary support in AOT backend, etc), the figuring out packaging (e.g. I don't think lld binary should be shipped with Dart SDK - instead some sort of cloud storage should be setup for on demand SDK components).

@Levi-Lesches
Copy link

@Levi-Lesches Levi-Lesches commented Jul 13, 2021

Thanks for the detailed explanation, it helps put the issue in perspective

@Timmmm
Copy link

@Timmmm Timmmm commented Jul 14, 2021

Rather than appending it could you add the AOT snapshot as a data segment to the AOT runtime? Then you don't need a linker, just an ELF/Mach-O/PE editor, which is a fair bit simpler (you could probably even write one in Dart fairly easily).

@mraleph
Copy link
Member

@mraleph mraleph commented Jul 14, 2021

@Timmmm yeah, we considered that as a middle ground option as well (between simple concat and full linking), but it was still way more investment than we could afford to make.

@lexaknyazev
Copy link
Contributor

@lexaknyazev lexaknyazev commented Jul 14, 2021

... and simply concatenates them together. From OS perspective the result is still a valid executable file which simply has a bunch of garbage at the end - but OS can still run it. When AOT runtime binary is run it looks at its own file to see if there is a snapshot appended to it - and if there is then it loads and runs it.

A similar approach is also used to create self-extracting archives (e.g. see 7-Zip).

Problems begin when you try to edit the binary using tools which work with native executable formats (e.g. sign it, change an icon, etc). For these tools the snapshot at the end of the file are simply garbage bytes, so they usually just remove them.

Windows signtool simply appends another payload at the end thus breaking the following part of AOT snapshot loading:

// Check for payload appended at the end of the container file.
// If header is found, jump to payload offset.
int64_t appended_header[2];
if (!file->SetPosition(file->Length() - sizeof(appended_header))) {
return nullptr;
}
if (!file->ReadFully(&appended_header, sizeof(appended_header))) {
return nullptr;
}
// Length is always encoded as Little Endian.
const uint64_t appended_offset =
Utils::LittleEndianToHost64(appended_header[0]);
if (memcmp(&appended_header[1], appjit_magic_number.bytes,
appjit_magic_number.length) != 0 ||
appended_offset <= 0) {
return nullptr;
}

Probably, a low-hanging fruit here is to make this code resilient to extra data located after the snapshot.

@lexaknyazev
Copy link
Contributor

@lexaknyazev lexaknyazev commented Jul 14, 2021

AOT runtime binary (which is a valid platform executable produced by a native C++ toolchain)

Strangely enough, Apple's objdump does not like it:

% objdump -x dartaotruntime

dartaotruntime:	file format mach-o 64-bit x86-64

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/objdump: error: 'dartaotruntime': Invalid/Unsupported object file format

@mraleph
Copy link
Member

@mraleph mraleph commented Jul 14, 2021

@lexaknyazev Hmm, that's a bit peculiar. otool -al does work on it though.

@yu-pri
Copy link

@yu-pri yu-pri commented Nov 12, 2021

Rather than appending it could you add the AOT snapshot as a data segment to the AOT runtime? Then you don't need a linker, just an ELF/Mach-O/PE editor, which is a fair bit simpler (you could probably even write one in Dart fairly easily).

Hey, @mraleph!

I'm looking into making a workaround for creating signed apps using this option, but I'm not quite sure of the scope of changes for this.

  • Does this approach require modification of the Dart runtime? My guess is that it probably requires changes in how the snapshot is found and loaded, when it is a data segment instead of a lump of bytes at the end of the file.
  • Should it be sufficient to just add the whole snapshot to runtime as a data segment?

@Timmmm
Copy link

@Timmmm Timmmm commented Nov 12, 2021

I think you just need to:

  1. Find/make a library that is capable of adding segments to ELF/PE/Mach-O files. Something like LIEF.
  2. Change the code that appends the snapshot to instead insert it as a new ELF section.
  3. Change the code that loads the snapshot to instead find it in the ELF file.

I think the code is around TryReadAppendedAppSnapshotElf() and TryReadAppSnapshotElf() in runtime/bin/snapshot_utils.cc.

@yu-pri
Copy link

@yu-pri yu-pri commented Nov 12, 2021

Thanks a lot, @Timmmm!

That's almost verbatim what I've had in mind, wanted to make sure I'm not missing something here.

@yu-pri
Copy link

@yu-pri yu-pri commented Dec 2, 2021

@Timmmm, @mraleph
I've tried doing a small Proof of Concept using @Timmmm's proposed solution.

The process is the following:

  1. Modify the dart-sdk files to look for the snapshot in a data segment (file offset specified manually for now).
  2. Compile the SDK, dartaotruntime and the application snapshot.
  3. Sign the snapshot.
  4. Add the snapshot as a data segment to the runtime file, using macholib library.
  5. Sign the combined binary.

The above process works well, unless I'm using hardened runtime (codesign -o runtime ...).

The codesigning process concludes successfully, and if I'm checking the signature using codesign -dv -- it says everything's in order. But at runtime the binary gets killed with a Code Signature Invalid exception.

Not really sure what might be causing it -- can the runtime be loading some non-signed libraries during runtime?

UPD: Adding a com.apple.security.cs.allow-unsigned-executable-memory entitlement fixed the problem. Now trying to get the binary notarized for distribution...

UPD2: Managed to successfully notarize the binary, and run it via terminal on several Mac machines without getting security popups. Will see if we'll be able to make a PR sometime later.

@mraleph
Copy link
Member

@mraleph mraleph commented Dec 6, 2021

Not really sure what might be causing it -- can the runtime be loading some non-signed libraries during runtime?

It loads the snapshot which (as it is currently implemented) requires to map parts of the binary as executable pages even though originally it was not executable - hence the need for com.apple.security.cs.allow-unsigned-executable-memory as you have discovered.

Will see if we'll be able to make a PR sometime later.

FWIW I am not sure we would take a dependency on Python and macholib - so we are unlikely to take it, but it's great that you made it work for your usecase.

@BobEvans
Copy link

@BobEvans BobEvans commented Dec 15, 2021

UPD2: Managed to successfully notarize the binary, and run it via terminal on several Mac machines without getting security popups. Will see if we'll be able to make a PR sometime later.

@yu-pri
Would you be willing to share your code? I have exactly the same need and would like to not reinvent the wheel.

@illia-romanenko
Copy link

@illia-romanenko illia-romanenko commented Dec 22, 2021

@BobEvans I worked with @yu-pri on this - and it's rather in the POC stage, so nothing really to share. A lot of steps are just manual to make it work. Currently, that's where we will leave it until it is required for the project we are working on. If you need any tips, you can ping me directly via some social media.

@BobEvans
Copy link

@BobEvans BobEvans commented Jan 13, 2022

@illia-romanenko Thanks Illia. I got it working for our case as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet