FR: Bundle a dart library/snapshot + VM as an exexcutable #27596

Open
matanlurey opened this Issue Oct 16, 2016 · 11 comments

Projects

None yet

7 participants

@matanlurey
Contributor
matanlurey commented Oct 16, 2016 edited

Strawman

Builds a.dar, or Dart archive using the program dar, bundled as part of the SDK.

Assume the following program structure:

hello_world/
  bin/
    hello_world.dart
  lib/
    hello_world_lib.dart <-- Imports package:path/path.dart (?)
import 'package:path/path.dart' as p;

// hello_world.dart
void main(List<String> args) {
  print('Hello World');
  print('Your arguments are: $args');
  print('Your CWD is ${p.current}');
  print('Goodbye!');
}

We can run dar build to bundle a package + an entrypoint:

dar build \
  --entrypoint hello_world/bin/hello_world.dart
  --output hello_world.dar

> Archiving package:hello_world as a standalone executable...
> SDK: Dart version 1.20.0-alpha
> Resolving dependencies...
> Building Dart VM snapshot of hello_world/bin/hello_world.dart...
> Wrote hello_world.dar (18619 bytes)

Example use (on BASH compatible OS):

$ ./hello_world.dar --test

> Hello World.
> Your arguemnts are: ['--test']
> Your CWD is: /usr/foo/bar/examples
> Goodbye!

Cross platform support

For (non-bash) windows support, it would be nice to support --output-exe:

dar.exe build \
  --entrypoint hello_world/bin/hello_world.dart
  --output-exe hello_world.exe
hello_world.exe --test

> Hello World.
> Your arguemnts are: ['--test']
> Your CWD is: C:\some_dir\examples
> Goodbye!

Future optimizations and ideas

"Native" compilation

Like used in Flutter, perhaps we could pre-JIT a snapshot for a platform:

dar build \
  --entrypoint hello_world/bin/hello_world.dart
  --output hello_world.dar
  --experimental-pre-compile

In the future future if we support pure dart2native AOT, this could start emitting native code.

Without bundling the VM

For platforms where the Dart VM already exists (certain OSs, but also on a managed VM, etc), we could omit bundling the VM/SDK, and just bundle the Dart files. That would make the "executable" significantly smaller for these systems if we can trust the version.

Tree-shaking and optimizations

Once the Kernel is complete and we have extra cycles - revive dart2dart?

NodeJS support

Similar to work done by @srawlins and @kevmoo - being able to bundle as a NodeJS executable means we could run natively on google cloud functions and other platforms that take NodeJS. This would require use of the dart dev compiler today, maybe dart2js in the future.

Potential stakeholders

  • @jakemac53 - Can we support a .sar like output for Bazel using this scheme?
  • @nex3 for a new high-profile tool (see internal discussion)
  • @srawlins for deployment of dart binaries to cloud VM
  • @yjbanov for flutter tools (?)

/cc @anders-sandholm @a-siva @floitschG @kevmoo

@zoechi
Contributor
zoechi commented Oct 17, 2016

Would that be a .dar file per platform or would it contain the executable for every supported platform?

@kasperl
Contributor
kasperl commented Oct 17, 2016

We're already doing tree-shaking and a few optimizations as IR-to-IR transformations. I don't see the need for dart2dart in this setup, but maybe I'm missing something.

@nex3
Member
nex3 commented Oct 17, 2016

I'm very excited about this general idea. It would make deploying Dart applications to non-Dart users extremely attractive.

We can run dar build to bundle a package + an entrypoint:

dar build \
  --entrypoint hello_world/bin/hello_world.dart
  --package hello_world
  --output hello_world.dar

> Archiving package:hello_world as a standalone executable...
> SDK: Dart version 1.20.0-alpha
> Resolving dependencies...
> Building Dart VM snapshot of hello_world/bin/hello_world.dart...
> Wrote hello_world.dar (18619 bytes)

What does the --package flag do here?

Cross platform support

For (non-bash) windows support, it would be nice to support --output-exe:

dar.exe build \
  --entrypoint hello_world/bin/hello_world.dart
  --package hello_world
  --output-exe hello_world.exe
hello_world.exe --test

> Hello World.
> Your arguemnts are: ['--test']
> Your CWD is: C:\some_dir\examples
> Goodbye!

Ideally, I'd like to control both the target OS and the target architecture, even if that means downloading a new Dart binary. I'd expect to be able to generate executables for the same targets that Dart itself is distributed for: Linux, Windows, and OS X both ia32 and x64.

Without bundling the VM

For platforms where the Dart VM already exists (certain OSs, but also on a managed VM, etc), we could omit bundling the VM/SDK, and just bundle the Dart files. That would make the "executable" significantly smaller for these systems if we can trust the version.

Isn't this just a snapshot like we have today?

NodeJS support

Similar to work done by @srawlins and @kevmoo - being able to bundle as a NodeJS executable means we could run natively on google cloud functions and other platforms that take NodeJS. This would require use of the dart dev compiler today, maybe dart2js in the future.

If you just mean a JavaScript entrypoint that can be run using node out.js, this is pretty easy to do today with a small amount of scaffolding even on dart2js.

@matanlurey
Contributor

@zoechi:

Would that be a .dar file per platform or would it contain the executable for every supported platform?

I would prefer a .dar file per platform. On windows you might just want to call it an exe.

@kasperl:

We're already doing tree-shaking and a few optimizations as IR-to-IR transformations. I don't see the need for dart2dart in this setup, but maybe I'm missing something.

I realize that the IR/Kernel means dart2dart might not make any sense. My question is that if I have in my bin/hello_world.dart script:

void main() {
  print('Hello World');
}

void intentionallyUnusedFunction() {
  print('Will somebody use me?');
}

And I create a VM snapshot, will intentionallyUnusedFunction remain?

@nex3:

What does the --package flag do here?

I realize you could just assume the package based on bin/path.dart, so I've updated my proposal above to remove this flag.

... Isn't this just a snapshot like we have today?

It is, but the difference here is I'd like it to a simple single file you run, i.e.

$ wget https://some.site/http_server.dar
$ chmod +x http_server.dar
$ ./http_server.dart --port 8080

> Running server at 0.0.0.0:8080...

(There is no abstraction leak that this is even a Dart binary, in this case)

If you just mean a JavaScript entrypoint that can be run using node out.js, this is pretty easy to do today with a small amount of scaffolding even on dart2js.

I imagine it's not a lot of work, but it's the bundling and making it easy/first-class that I'm interested in here. i.e., deploy your package to Google cloud function with a simple script with little or no manual work. (Plus it's important to use DDC because it maps dart:io to NodeJs correctly)

@nex3
Member
nex3 commented Oct 17, 2016

It is, but the difference here is I'd like it to a simple single file you run, i.e.

$ wget https://some.site/http_server.dar
$ chmod +x http_server.dar
$ ./http_server.dart --port 8080

> Running server at 0.0.0.0:8080...

(There is no abstraction leak that this is even a Dart binary, in this case)

So the difference between this and a snapshot is that this looks up the Dart VM on the path? How do you handle version skew between the snapshot and the VM? Historically, each VM version only supports its own snapshot version.

I imagine it's not a lot of work, but it's the bundling and making it easy/first-class that I'm interested in here. i.e., deploy your package to Google cloud function with a simple script with little or no manual work. (Plus it's important to use DDC because it maps dart:io to NodeJs correctly)

I didn't realize DDC had a dart:io wrapper built in. @jmesserly, how/where is that defined? Could we generalize it enough to make it work with dart2js as well? Since DDC is generally dev-mode-focused rather than performance-focused, it seems like dart2js is a better place to focus our efforts to make deployable Node apps.

@matanlurey
Contributor

@nex3:

How do you handle version skew between the snapshot and the VM? Historically, each VM version only supports its own snapshot version.

Good point. Depends how clever we want it to be then, I guess you could deploy as a .dar that does require the Dart SDK, and builds a snapshot on demand (a sort of JIT). Ideally most users would deploy standalone with an SDK included, though.

Could we generalize it enough to make it work with dart2js as well?

I think the Node support was an experimental 20% by @ochafik - not sure though.

@jmesserly
Member

It looks like it is defined here: https://github.com/dart-lang/sdk/blob/master/pkg/dev_compiler/tool/input_sdk/patch/io_patch.dart -- looks like just stubs though.

@nex3
Member
nex3 commented Oct 17, 2016

Oh, I see—that's about the same level of support as dart2js.

@matanlurey
Contributor

Cool, then yes dart2js might be a fine target.

@kasperl
Contributor
kasperl commented Oct 18, 2016

We're already doing tree-shaking and a few optimizations as IR-to-IR transformations. I don't see
the need for dart2dart in this setup, but maybe I'm missing something.

I realize that the IR/Kernel means dart2dart might not make any sense. My question is that if I have
in my bin/hello_world.dart script:
...
And I create a VM snapshot, will intentionallyUnusedFunction remain?

@matanlurey: There's a separate IR-to-IR transformation that you can apply before you generate a snapshot: https://github.com/dart-lang/kernel/blob/master/lib/transformations/treeshaker.dart. After running the separate tree shaker on the IR input, the output IR will not contain intentionallyUnusedFunction, so we will never see it when we generate the snapshot.

@matanlurey
Contributor

@kasperl Nice! OK, then sounds at least part of this FR already exists 😅

Does the rest of this sound reasonable?

@a-siva a-siva added the area-sdk label Oct 18, 2016
@kevmoo kevmoo added Type: enhancement and removed enhancement labels Feb 9, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment