Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add script to generate zinc native-images, with example usage #8036

Conversation

@cosmicexplorer
Copy link
Contributor

commented Jul 10, 2019

Problem

A script to automatically create zinc native-images from pants for JVM code is currently being created in #7506. The script will infer macro usages from the compiler in an initial run with the native-image java agent, described in https://github.com/oracle/graal/blob/master/substratevm/CONFIGURE.md. This requires providing a library built from the graal repo on the runtime library search path.

Solution

  • Pipe in LD_LIBRARY_PATH or DYLD_LIBRARY_PATH into the hermetic zinc invocation.
  • Pull down graal and other dependencies by git clone for now.
  • Use ./pants classmap to get the targets providing the classes corresponding to macros, and generate a BUILD file with a fake jvm_app() bundling all those targets.

Result

A script works to generate zinc images for arbitrary scala code!

@cosmicexplorer cosmicexplorer requested review from stuhood and illicitonion Jul 10, 2019

@illicitonion
Copy link
Contributor

left a comment

Thanks! Just one comment around hermeticism...

@cosmicexplorer cosmicexplorer requested a review from illicitonion Jul 14, 2019

cosmicexplorer added a commit that referenced this pull request Jul 15, 2019

embed a graal native-image configuration inside the zinc wrapper (#7506)
### Problem

[`native-image`](https://www.graalvm.org/docs/reference-manual/aot-compilation/) is a tool from the [Graal](https://github.com/oracle/graal) project which can compile jvm bytecode to a native executable with very short startup time.

As of [1.0.0rc13](https://github.com/oracle/graal/releases), `native-image` became able to use special files in `META-INF/` to configure the command-line arguments and json configuration it needed to run, described in https://medium.com/graalvm/simplifying-native-image-generation-with-maven-plugin-and-embeddable-configuration-d5b283b92f57. This allows us to publish a zinc artifact which works the exact same when invoked by `java`, but can also be made into a `native-image` without any further configuration.

### Solution

- Add the reflective/resource access json files into `META-INF/` in the zinc jar, as well as substitutions.
- Add descriptive readmes linking to external documentation and blog posts about Graal which were used to construct the configuration being added.

### Result

#8036 (once the artifacts from this PR are published to maven central) will be able to remove an extra git clone from its script which does all of the configuration automatically!

@cosmicexplorer cosmicexplorer force-pushed the cosmicexplorer:zinc-native-image-agent-auto-macro branch 2 times, most recently from 7e893da to b6398a9 Jul 16, 2019

@cosmicexplorer cosmicexplorer changed the title pipe in --native-image-agent-library-dir to hermetic zinc compiles add script to generate zinc native-images, with example usage Jul 17, 2019

@stuhood
Copy link
Member

left a comment

Thanks!

Show resolved Hide resolved build-support/native-image/README.md
Show resolved Hide resolved build-support/native-image/generate-native-image-for-pants-targets.bash Outdated
Show resolved Hide resolved build-support/native-image/generate-native-image-for-pants-targets.bash Outdated
Show resolved Hide resolved build-support/native-image/build-zinc-native-image.bash
Show resolved Hide resolved build-support/native-image/build-zinc-native-image.bash
Show resolved Hide resolved build-support/native-image/build-zinc-native-image.bash
function fetch_pants_zinc_wrapper_jars {
pants_zinc_compiler_version='0.0.15'
pants_underlying_zinc_dependency_version='1.1.7'
# NB: `native-image` emits a warning on later protobuf versions, which the pantsbuild

This comment has been minimized.

Copy link
@stuhood

stuhood Jul 19, 2019

Member

To minimize the differences between the native image and the JVM version, we should probably do this in our wrapper, here:

jar_library(
name='zinc',
jars=[
scala_jar(org='org.scala-sbt', name='zinc', rev=zinc_version,
excludes=[
scala_exclude(org='org.scala-sbt', name='io'),
scala_exclude(org='org.scala-sbt', name='util-logging'),
]),
],
dependencies=[
':io',
':util-logging',
],
)

Ok to do as a followup after this lands.

This comment has been minimized.

Copy link
@cosmicexplorer

cosmicexplorer Jul 19, 2019

Author Contributor

I've been doing that locally, but removed it because it seemed like a hack. Will definitely add in a followup!

}

# generate the deps jar and reflect config at known locations!!!

This comment has been minimized.

Copy link
@stuhood

stuhood Jul 19, 2019

Member

It feels like this script needs a bit more baking. Is it worth extracting this file out of this PR, finishing the design for how the native image will be maintained, and then revisiting it?

This comment has been minimized.

Copy link
@cosmicexplorer

cosmicexplorer Jul 19, 2019

Author Contributor

I think that this script should absolutely be considered a prototype given:
(1) it will be significantly less complex once we can upstream our native-image fixes into oracle/graal#1448 via #7955
(2) it's difficult to write reproducible bash scripts, so it would probably need breaking down into component rules anyway

And (3) I think "here is a script that works, with clear instructions on how to run it from any repo" feels like a great way to decouple "being able to generate native-images" from "here is a working implementation in pants".

This comment has been minimized.

Copy link
@stuhood

stuhood Jul 19, 2019

Member

I'm not sure that answers my question... do you think it would be reasonable to break this out and finish the UX design before landing this particular file? I'm just talking about deferring generate-native-image-for-pants-targets.bash, not about build-zinc-native-image.bash: the latter should land here I think.

Show resolved Hide resolved src/python/pants/backend/jvm/tasks/jvm_compile/zinc/zinc_compile.py Outdated
Show resolved Hide resolved src/python/pants/backend/jvm/tasks/jvm_compile/zinc/zinc_compile.py Outdated

@cosmicexplorer cosmicexplorer force-pushed the cosmicexplorer:zinc-native-image-agent-auto-macro branch 5 times, most recently from 96ee232 to 354e480 Jul 23, 2019

cosmicexplorer added some commits Jun 28, 2019

add native-image build script!!!
add auto native-image gen script

fix scripts

update script

update script with coursier
deeply expand the native-image readme
also, move the readme!

@cosmicexplorer cosmicexplorer force-pushed the cosmicexplorer:zinc-native-image-agent-auto-macro branch from 354e480 to e9078cb Jul 23, 2019

@stuhood
Copy link
Member

left a comment

Thanks!

@cosmicexplorer cosmicexplorer merged commit b76a9d2 into pantsbuild:master Jul 24, 2019

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.