FR: Bundle a dart library/snapshot + VM as an executable #27596

matanlurey · 2016-10-16T22:14:30Z

Strawman

Builds a.dar, or Dart archive using the program dar, bundled as part of the SDK.

Assume the following program structure:

hello_world/
  bin/
    hello_world.dart
  lib/
    hello_world_lib.dart <-- Imports package:path/path.dart (?)

import 'package:path/path.dart' as p;

// hello_world.dart
void main(List<String> args) {
  print('Hello World');
  print('Your arguments are: $args');
  print('Your CWD is ${p.current}');
  print('Goodbye!');
}

We can run dar build to bundle a package + an entrypoint:

dar build \
  --entrypoint hello_world/bin/hello_world.dart
  --output hello_world.dar

> Archiving package:hello_world as a standalone executable...
> SDK: Dart version 1.20.0-alpha
> Resolving dependencies...
> Building Dart VM snapshot of hello_world/bin/hello_world.dart...
> Wrote hello_world.dar (18619 bytes)

Example use (on BASH compatible OS):

$ ./hello_world.dar --test

> Hello World.
> Your arguemnts are: ['--test']
> Your CWD is: /usr/foo/bar/examples
> Goodbye!

Cross platform support

For (non-bash) windows support, it would be nice to support --output-exe:

dar.exe build \
  --entrypoint hello_world/bin/hello_world.dart
  --output-exe hello_world.exe

hello_world.exe --test

> Hello World.
> Your arguemnts are: ['--test']
> Your CWD is: C:\some_dir\examples
> Goodbye!

Future optimizations and ideas

"Native" compilation

Like used in Flutter, perhaps we could pre-JIT a snapshot for a platform:

dar build \
  --entrypoint hello_world/bin/hello_world.dart
  --output hello_world.dar
  --experimental-pre-compile

In the future future if we support pure dart2native AOT, this could start emitting native code.

Without bundling the VM

For platforms where the Dart VM already exists (certain OSs, but also on a managed VM, etc), we could omit bundling the VM/SDK, and just bundle the Dart files. That would make the "executable" significantly smaller for these systems if we can trust the version.

Tree-shaking and optimizations

Once the Kernel is complete and we have extra cycles - revive dart2dart?

NodeJS support

Similar to work done by @srawlins and @kevmoo - being able to bundle as a NodeJS executable means we could run natively on google cloud functions and other platforms that take NodeJS. This would require use of the dart dev compiler today, maybe dart2js in the future.

Potential stakeholders

@jakemac53 - Can we support a .sar like output for Bazel using this scheme?
@nex3 for a new high-profile tool (see internal discussion)
@srawlins for deployment of dart binaries to cloud VM
@yjbanov for flutter tools (?)

/cc @anders-sandholm @a-siva @floitschG @kevmoo

The text was updated successfully, but these errors were encountered:

zoechi · 2016-10-17T06:27:39Z

Would that be a .dar file per platform or would it contain the executable for every supported platform?

kasperl · 2016-10-17T07:24:35Z

We're already doing tree-shaking and a few optimizations as IR-to-IR transformations. I don't see the need for dart2dart in this setup, but maybe I'm missing something.

nex3 · 2016-10-17T19:39:00Z

I'm very excited about this general idea. It would make deploying Dart applications to non-Dart users extremely attractive.

We can run dar build to bundle a package + an entrypoint:

dar build \
  --entrypoint hello_world/bin/hello_world.dart
  --package hello_world
  --output hello_world.dar

> Archiving package:hello_world as a standalone executable...
> SDK: Dart version 1.20.0-alpha
> Resolving dependencies...
> Building Dart VM snapshot of hello_world/bin/hello_world.dart...
> Wrote hello_world.dar (18619 bytes)

What does the --package flag do here?

Cross platform support

For (non-bash) windows support, it would be nice to support --output-exe:

dar.exe build \
  --entrypoint hello_world/bin/hello_world.dart
  --package hello_world
  --output-exe hello_world.exe

hello_world.exe --test

> Hello World.
> Your arguemnts are: ['--test']
> Your CWD is: C:\some_dir\examples
> Goodbye!

Ideally, I'd like to control both the target OS and the target architecture, even if that means downloading a new Dart binary. I'd expect to be able to generate executables for the same targets that Dart itself is distributed for: Linux, Windows, and OS X both ia32 and x64.

Without bundling the VM

For platforms where the Dart VM already exists (certain OSs, but also on a managed VM, etc), we could omit bundling the VM/SDK, and just bundle the Dart files. That would make the "executable" significantly smaller for these systems if we can trust the version.

Isn't this just a snapshot like we have today?

NodeJS support

Similar to work done by @srawlins and @kevmoo - being able to bundle as a NodeJS executable means we could run natively on google cloud functions and other platforms that take NodeJS. This would require use of the dart dev compiler today, maybe dart2js in the future.

If you just mean a JavaScript entrypoint that can be run using node out.js, this is pretty easy to do today with a small amount of scaffolding even on dart2js.

matanlurey · 2016-10-17T19:48:52Z

@zoechi:

Would that be a .dar file per platform or would it contain the executable for every supported platform?

I would prefer a .dar file per platform. On windows you might just want to call it an exe.

@kasperl:

We're already doing tree-shaking and a few optimizations as IR-to-IR transformations. I don't see the need for dart2dart in this setup, but maybe I'm missing something.

I realize that the IR/Kernel means dart2dart might not make any sense. My question is that if I have in my bin/hello_world.dart script:

void main() {
  print('Hello World');
}

void intentionallyUnusedFunction() {
  print('Will somebody use me?');
}

And I create a VM snapshot, will intentionallyUnusedFunction remain?

@nex3:

What does the --package flag do here?

I realize you could just assume the package based on bin/path.dart, so I've updated my proposal above to remove this flag.

... Isn't this just a snapshot like we have today?

It is, but the difference here is I'd like it to a simple single file you run, i.e.

$ wget https://some.site/http_server.dar
$ chmod +x http_server.dar
$ ./http_server.dart --port 8080

> Running server at 0.0.0.0:8080...

(There is no abstraction leak that this is even a Dart binary, in this case)

If you just mean a JavaScript entrypoint that can be run using node out.js, this is pretty easy to do today with a small amount of scaffolding even on dart2js.

I imagine it's not a lot of work, but it's the bundling and making it easy/first-class that I'm interested in here. i.e., deploy your package to Google cloud function with a simple script with little or no manual work. (Plus it's important to use DDC because it maps dart:io to NodeJs correctly)

nex3 · 2016-10-17T20:01:33Z

It is, but the difference here is I'd like it to a simple single file you run, i.e.
$ wget https://some.site/http_server.dar
$ chmod +x http_server.dar
$ ./http_server.dart --port 8080

> Running server at 0.0.0.0:8080...
(There is no abstraction leak that this is even a Dart binary, in this case)

So the difference between this and a snapshot is that this looks up the Dart VM on the path? How do you handle version skew between the snapshot and the VM? Historically, each VM version only supports its own snapshot version.

I imagine it's not a lot of work, but it's the bundling and making it easy/first-class that I'm interested in here. i.e., deploy your package to Google cloud function with a simple script with little or no manual work. (Plus it's important to use DDC because it maps dart:io to NodeJs correctly)

I didn't realize DDC had a dart:io wrapper built in. @jmesserly, how/where is that defined? Could we generalize it enough to make it work with dart2js as well? Since DDC is generally dev-mode-focused rather than performance-focused, it seems like dart2js is a better place to focus our efforts to make deployable Node apps.

matanlurey · 2016-10-17T20:04:31Z

@nex3:

How do you handle version skew between the snapshot and the VM? Historically, each VM version only supports its own snapshot version.

Good point. Depends how clever we want it to be then, I guess you could deploy as a .dar that does require the Dart SDK, and builds a snapshot on demand (a sort of JIT). Ideally most users would deploy standalone with an SDK included, though.

Could we generalize it enough to make it work with dart2js as well?

I think the Node support was an experimental 20% by @ochafik - not sure though.

jmesserly · 2016-10-17T20:28:39Z

It looks like it is defined here: https://github.com/dart-lang/sdk/blob/master/pkg/dev_compiler/tool/input_sdk/patch/io_patch.dart -- looks like just stubs though.

nex3 · 2016-10-17T20:55:43Z

Oh, I see—that's about the same level of support as dart2js.

matanlurey · 2016-10-17T22:03:16Z

Cool, then yes dart2js might be a fine target.

kasperl · 2016-10-18T06:39:06Z

We're already doing tree-shaking and a few optimizations as IR-to-IR transformations. I don't see
the need for dart2dart in this setup, but maybe I'm missing something.

I realize that the IR/Kernel means dart2dart might not make any sense. My question is that if I have
in my bin/hello_world.dart script:
...
And I create a VM snapshot, will intentionallyUnusedFunction remain?

@matanlurey: There's a separate IR-to-IR transformation that you can apply before you generate a snapshot: https://github.com/dart-lang/kernel/blob/master/lib/transformations/treeshaker.dart. After running the separate tree shaker on the IR input, the output IR will not contain intentionallyUnusedFunction, so we will never see it when we generate the snapshot.

matanlurey · 2016-10-18T06:43:27Z

@kasperl Nice! OK, then sounds at least part of this FR already exists 😅

Does the rest of this sound reasonable?

orklann · 2017-08-23T00:53:47Z

What's the status of this FR? Is it included into dart?

matanlurey · 2017-08-23T04:06:17Z

No I don't believe so, hence why it isn't closed.

natalie-o-perret · 2019-03-07T08:59:07Z

+1 and kudos for a self-contained approach, that would make dart deployment much easier.

Adds: - dart2aot, a script similar to dart2js which compiles my.dart to my.dart.aot. - dartaotruntime, a minimal Dart runtime that only runs AOT blobs. - some extra tooling like gen_kernel and gen_snapshot used by the above. - build rules for all of the above, including adding it to the full SDK builds. Bug:#27596 Change-Id: Ic35f832b2b86be959212b8d21cfc5a082da5ced4 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/97627 Reviewed-by: Alexander Thomas <athom@google.com> Reviewed-by: Vyacheslav Egorov <vegorov@google.com> Commit-Queue: Clement Skau <cskau@google.com>

renatoathaydes · 2019-07-04T21:23:19Z

Doesn't dartaotruntime and dart2aot solve this problem almost completely now? Or is it still difficult to kind of "merge" dartaotruntime plus the main.dart.aot generated by dart2aot into just one main.exe? Generating single binaries like Go would be a killer!

mit-mit · 2019-08-09T11:17:32Z

Yes, I'm going to close the current issue. Initial support for native execution landed some months back and is documented here: https://dart.dev/tutorials/server/get-started#8-compile-for-production

Support for single executable is tracked in #36915

a-siva added the area-sdk Use area-sdk for general purpose SDK issues (packaging, distribution, …). label Oct 18, 2016

nex3 added customer-dart-sass enhancement labels Feb 2, 2017

kevmoo added type-enhancement A request for a change that isn't a bug and removed enhancement labels Feb 9, 2017

matanlurey mentioned this issue Jul 14, 2018

Possible to run dart scripts standalone? #33853

Closed

ghost changed the title ~~FR: Bundle a dart library/snapshot + VM as an exexcutable~~ FR: Bundle a dart library/snapshot + VM as an executable Dec 20, 2018

mit-mit assigned ghost Feb 28, 2019

nex3 mentioned this issue May 16, 2019

chocolatey - Can't update dart from 2.2.0 to 2.3.0 because of sass sass/dart-sass#678

Closed

mit-mit closed this as completed Aug 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FR: Bundle a dart library/snapshot + VM as an executable #27596

FR: Bundle a dart library/snapshot + VM as an executable #27596

matanlurey commented Oct 16, 2016 •

edited

Loading

zoechi commented Oct 17, 2016

kasperl commented Oct 17, 2016

nex3 commented Oct 17, 2016

Cross platform support

Without bundling the VM

NodeJS support

matanlurey commented Oct 17, 2016

nex3 commented Oct 17, 2016

matanlurey commented Oct 17, 2016

jmesserly commented Oct 17, 2016

nex3 commented Oct 17, 2016

matanlurey commented Oct 17, 2016

kasperl commented Oct 18, 2016

matanlurey commented Oct 18, 2016

orklann commented Aug 23, 2017

matanlurey commented Aug 23, 2017

natalie-o-perret commented Mar 7, 2019

renatoathaydes commented Jul 4, 2019

mit-mit commented Aug 9, 2019

FR: Bundle a dart library/snapshot + VM as an executable #27596

FR: Bundle a dart library/snapshot + VM as an executable #27596

Comments

matanlurey commented Oct 16, 2016 • edited Loading

Strawman

Cross platform support

Future optimizations and ideas

"Native" compilation

Without bundling the VM

Tree-shaking and optimizations

NodeJS support

Potential stakeholders

zoechi commented Oct 17, 2016

kasperl commented Oct 17, 2016

nex3 commented Oct 17, 2016

Cross platform support

Without bundling the VM

NodeJS support

matanlurey commented Oct 17, 2016

nex3 commented Oct 17, 2016

matanlurey commented Oct 17, 2016

jmesserly commented Oct 17, 2016

nex3 commented Oct 17, 2016

matanlurey commented Oct 17, 2016

kasperl commented Oct 18, 2016

matanlurey commented Oct 18, 2016

orklann commented Aug 23, 2017

matanlurey commented Aug 23, 2017

natalie-o-perret commented Mar 7, 2019

renatoathaydes commented Jul 4, 2019

mit-mit commented Aug 9, 2019

matanlurey commented Oct 16, 2016 •

edited

Loading