Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

graalvm{8,11}-ce: init at 20.2.0 #99631

Merged
merged 1 commit into from Nov 3, 2020
Merged

graalvm{8,11}-ce: init at 20.2.0 #99631

merged 1 commit into from Nov 3, 2020

Conversation

@glittershark
Copy link
Member

@glittershark glittershark commented Oct 5, 2020

Motivation for this change

See #93139 - multiple packages require graalvm, but it requires significant system resources to build so it's disabled in Hydra - unfortunately as a result it is both unavailable in a pre-built form to users and it keeps breaking (see eg #98256). This packages the pre-built community edition of graalvm.

Supersedes #93139, as the original author wanted to hand it off.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
@glittershark
Copy link
Member Author

@glittershark glittershark commented Oct 5, 2020

@hlolli
Copy link
Member

@hlolli hlolli commented Oct 5, 2020

I'm relieved that this works. The top-level binaries work at first glance and seem to runs just fine from what I've tested. I don't know how the 3rd party plugin development story is with this. But knowing that we should be able the override the truffle paths.

I wonder how we should handle the llvm toolchain as is documented here https://www.graalvm.org/reference-manual/llvm/Compiling/

[nix-shell:~/.cache/nixpkgs-review/pr-99631]$ results/graalvm11-ce/bin/lli --print-toolchain-path
/nix/store/5qgfk0bp9byfs80vkhqsqbpfarng6cpf-graalvm11-ce-20.2.0/languages/llvm/native/bin

with the printed toolchain path I try to compile the hello.c

/nix/store/5qgfk0bp9byfs80vkhqsqbpfarng6cpf-graalvm11-ce-20.2.0/languages/llvm/native/bin/clang hello.c -o hello
Tool execution failed. Are you sure the toolchain is available at /nix/store/5qgfk0bp9byfs80vkhqsqbpfarng6cpf-graalvm11-ce-20.2.0/lib/llvm
You can install it via GraalVM updater: `gu install llvm-toolchain`

More infos: https://www.graalvm.org/docs/reference-manual/languages/llvm/

I doubt we are able to use the graal updater with nixos.

Could you have a look if you are able to compile to llvm bytecode with the bundled llvm toolchain?

Getting this to work would be the easiest solution by far. And since it's broken on hydra atm, just merge it for now until we have better, or we can improve this and forget about compiling graalvm from sources.

@glittershark
Copy link
Member Author

@glittershark glittershark commented Oct 7, 2020

Could you have a look if you are able to compile to llvm bytecode with the bundled llvm toolchain?

I'm not sure how to do this really - I'm only using this for native-image compilation of clojure jars, which is working perfectly off of this branch.

@hlolli
Copy link
Member

@hlolli hlolli commented Oct 8, 2020

I will try to see if I can find a solution. Not having the llvm toolchain working would break the sulong support of graalvm.

@hlolli
Copy link
Member

@hlolli hlolli commented Oct 11, 2020

I was able to fix one error but I need to find out why the -fuse-ld flag is always paassed

env NIX_CFLAGS_LINK= NIX_CFLAGS= C_FLAGS= /nix/store/x9j3vxmad2zxi4q7hnn168j25j7wyhzz-graalvm11-ce-20.2.0/lib/llvm/bin/clang --graalvm-print-cmd test.c -o test
GraalVM wrapper script for clang
GraalVM version: 20.2.0
running: /nix/store/x9j3vxmad2zxi4q7hnn168j25j7wyhzz-graalvm11-ce-20.2.0/lib/llvm/bin/clang -I/nix/store/x9j3vxmad2zxi4q7hnn168j25j7wyhzz-graalvm11-ce-20.2.0/languages/llvm/native/include -I/nix/store/x9j3vxmad2zxi4q7hnn168j25j7wyhzz-graalvm11-ce-20.2.0/languages/llvm/include -stdlib=libc++ -Wno-unused-command-line-argument -flto=full -g -O1 -L/nix/store/x9j3vxmad2zxi4q7hnn168j25j7wyhzz-graalvm11-ce-20.2.0/languages/llvm/native/lib -fuse-ld=/nix/store/x9j3vxmad2zxi4q7hnn168j25j7wyhzz-graalvm11-ce-20.2.0/lib/llvm/bin/ld.lld -Wl,--mllvm=-lto-embed-bitcode,--lto-O0 --graalvm-print-cmd test.c -o test

But it seems this could be a graalvm thing and not nix https://github.com/oracle/graal/blob/3cd4bda0868432f72cc33203db598ff88eedc558/sulong/projects/com.oracle.truffle.llvm.toolchain.launchers/src/com/oracle/truffle/llvm/toolchain/launchers/common/ClangLikeBase.java#L174-L180

(help possibly needed):
glittershark#1

@glittershark
Copy link
Member Author

@glittershark glittershark commented Oct 22, 2020

@hlolli I'm currently depending on this in my project, and am aware of some other people who are using my fork - is there any reason we couldn't just get this merged as is and use it as a launching-off point for future work?

@hlolli
Copy link
Member

@hlolli hlolli commented Oct 22, 2020

No fine for me, I'm not using graalvm these days. So I'd not block this at this stage. Approving.

hlolli
hlolli approved these changes Oct 22, 2020
@glittershark
Copy link
Member Author

@glittershark glittershark commented Oct 22, 2020

/marvin opt-in

@marvin-mk2
Copy link

@marvin-mk2 marvin-mk2 bot commented Oct 22, 2020

Hi! I'm an experimental bot. My goal is to guide this PR through its stages, hopefully ending with a merge. You can read up on the usage here.

@marvin-mk2 marvin-mk2 bot added the marvin label Oct 22, 2020
@glittershark
Copy link
Member Author

@glittershark glittershark commented Oct 22, 2020

/status needs_reviewer

@bennyandresen
Copy link
Contributor

@bennyandresen bennyandresen commented Oct 23, 2020

Could we unite it with graalvm{8,11}-ee into one nix file?

It seems that it is basically the same code and differences only in url and sha256.

Then they will be better in sync in the future.
`

I tried to do this at first but I reached my limits of Nix programming abilities. I wanted to parametrize them and then dispatch based on but I couldn't get the scoping and/or casing to work as I wanted them to.
I'm basically just stumbling from package to package just to get them to work and have them not look terrible.

I'm also in favor of this getting merged. I would love to update babashka and clj-kondo again for everyone using Nix and not continue using my private -bin packages.

@glittershark
Copy link
Member Author

@glittershark glittershark commented Oct 23, 2020

I'll give it a shot

@marvin-mk2
Copy link

@marvin-mk2 marvin-mk2 bot commented Oct 26, 2020

Reminder: Please review!

This Pull Request is awaiting review. If you are the assigned reviewer, please have a look. Try to find another reviewer if necessary. If you can't, please say so. If the status is not accurate, please change it. If nothing happens, this PR will be put back in the needs_reviewer queue in one day.

@glittershark
Copy link
Member Author

@glittershark glittershark commented Oct 26, 2020

/status awaiting_changes

@bqv
Copy link
Contributor

@bqv bqv commented Oct 31, 2020

Keen to see this happen!

@glittershark
Copy link
Member Author

@glittershark glittershark commented Oct 31, 2020

@conferno

There is 20.2.1 already (#101438 )

So it doesn't look like there's a CE of 20.2.1 yet - given that, can we hold off on deduping this and graalvm-ce? I have a WIP of that up at glittershark@59d76a1 if people wanna take a look, but until CE 20.2.1 is out I vote we just merge this and then do the deduping later.

cc @volth @hlolli

@marvin-mk2
Copy link

@marvin-mk2 marvin-mk2 bot commented Nov 3, 2020

Reminder: Please review!

This Pull Request is awaiting review. If you are the assigned reviewer, please have a look. Try to find another reviewer if necessary. If you can't, please say so. If the status is not accurate, please change it. If nothing happens, this PR will be put back in the needs_reviewer queue in one day.

@bqv
Copy link
Contributor

@bqv bqv commented Nov 3, 2020

/status needs_merger

@marvin-mk2 marvin-mk2 bot requested a review from kevincox Nov 3, 2020
@kevincox kevincox merged commit dc79731 into NixOS:master Nov 3, 2020
17 checks passed
glittershark added a commit to glittershark/nixpkgs that referenced this issue Nov 4, 2020
Now that we have community builds of graalvm landed in NixOS#99631, both
clj-kondo and babashka can depend on those versions of graalvm rather
than the one that requires building from source - this can be built in
hydra, and generally is much easier to build and test.
jonringer added a commit that referenced this issue Nov 4, 2020
Now that we have community builds of graalvm landed in #99631, both
clj-kondo and babashka can depend on those versions of graalvm rather
than the one that requires building from source - this can be built in
hydra, and generally is much easier to build and test.
bennyandresen added a commit to bennyandresen/nixpkgs that referenced this issue Nov 5, 2020
with NixOS#99631 and NixOS#102693 merged, it's possible to bump the babashka
version again.

However recent versions of babashka depend on java11 features and I
spoke in Slack with the project lead and this java11 dependency will
exist going forward.
@bennyandresen bennyandresen mentioned this pull request Nov 5, 2020
10 tasks
bennyandresen added a commit to bennyandresen/nixpkgs that referenced this issue Nov 11, 2020
Now that we have community builds of graalvm landed in NixOS#99631, both
clj-kondo and babashka can depend on those versions of graalvm rather
than the one that requires building from source - this can be built in
hydra, and generally is much easier to build and test.
bennyandresen added a commit to bennyandresen/nixpkgs that referenced this issue Nov 11, 2020
with NixOS#99631 and NixOS#102693 merged, it's possible to bump the babashka
version again.

However recent versions of babashka depend on java11 features and I
spoke in Slack with the project lead and this java11 dependency will
exist going forward.
bennyandresen added a commit to bennyandresen/nixpkgs that referenced this issue Nov 11, 2020
Now that we have community builds of graalvm landed in NixOS#99631, both
clj-kondo and babashka can depend on those versions of graalvm rather
than the one that requires building from source - this can be built in
hydra, and generally is much easier to build and test.
bennyandresen added a commit to bennyandresen/nixpkgs that referenced this issue Nov 11, 2020
with NixOS#99631 and NixOS#102693 merged, it's possible to bump the babashka
version again.

However recent versions of babashka depend on java11 features and I
spoke in Slack with the project lead and this java11 dependency will
exist going forward.
bennyandresen added a commit to bennyandresen/nixpkgs that referenced this issue Nov 12, 2020
Now that we have community builds of graalvm landed in NixOS#99631, both
clj-kondo and babashka can depend on those versions of graalvm rather
than the one that requires building from source - this can be built in
hydra, and generally is much easier to build and test.

(cherry picked from commit bb885bc)
bennyandresen added a commit to bennyandresen/nixpkgs that referenced this issue Nov 12, 2020
with NixOS#99631 and NixOS#102693 merged, it's possible to bump the babashka
version again.

However recent versions of babashka depend on java11 features and I
spoke in Slack with the project lead and this java11 dependency will
exist going forward.

(cherry picked from commit 43ae904)
bennyandresen added a commit to bennyandresen/nixpkgs that referenced this issue Nov 12, 2020
Now that we have community builds of graalvm landed in NixOS#99631, both
clj-kondo and babashka can depend on those versions of graalvm rather
than the one that requires building from source - this can be built in
hydra, and generally is much easier to build and test.

(cherry picked from commit 3341f6c)
bennyandresen added a commit to bennyandresen/nixpkgs that referenced this issue Nov 12, 2020
with NixOS#99631 and NixOS#102693 merged, it's possible to bump the babashka
version again.

However recent versions of babashka depend on java11 features and I
spoke in Slack with the project lead and this java11 dependency will
exist going forward.

(cherry picked from commit 172cbb8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants