Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

graalvm{8,11}-ce: init at 20.1.0 #93139

Closed
wants to merge 2 commits into from

Conversation

@bennyandresen
Copy link
Contributor

@bennyandresen bennyandresen commented Jul 14, 2020

Motivation for this change

babashka requires graalvm20 and building graalvm20 from source seems rather difficult and the extremely long compile times and other issues that @hlolli is experiencing are rather draining.

For now I would propose that we add graalvm8-ce and graalvm11-ce

I will add a PR to babashka and clj-kondo that make them depend on graalvm11-ce until we have a better graalvm "built from source" story.

for more information see #83080

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

cc: @volth @jlesquembre

@hlolli
Copy link
Member

@hlolli hlolli commented Jul 14, 2020

Not a bad idea. Tough I have graalvm core working in my PR, holding back because I still don't have the graalpython/graal-js and other polyglots working yet. Can you confirm if the polyglot binaries work? If so this will be a good interim solution. One of my biggest problem in my PR is lack of response and comments from maintainers. I think I will continue on my PR next month using newer version, many of the problems I was hitting have been resolved in newer releases after I reported them upstream.

Alternatively we can say, screw the polyglots and let's get the core working properly, hence why I'm trying to make the libraries independent from the rest (perhaps an overly-abitious pursuit).

@bennyandresen
Copy link
Contributor Author

@bennyandresen bennyandresen commented Jul 14, 2020

@hlolli
Yeah the polyglot binaries work. I've been able to run simple programs in truffleruby and graalpython.

the wasm binary doesn't crash, but I'm unable to confirm that it runs any wasm file.
I've never worked with wasm before and my minimal emscripten compiled example doesn't run.

@jlesquembre
Copy link
Contributor

@jlesquembre jlesquembre commented Jul 14, 2020

@bennyandresen thanks for working on this, I like the idea. In fact, I was thinking on opening a similar PR myself, but I didn't find time to work on it. I'll try it tomorrow

@hlolli
Copy link
Member

@hlolli hlolli commented Jul 14, 2020

same here I'll give this PR a try tomorrow. The fact that the polyglots langs are working is a seller to me.

@jlesquembre
Copy link
Contributor

@jlesquembre jlesquembre commented Jul 15, 2020

@jlesquembre
Copy link
Contributor

@jlesquembre jlesquembre commented Jul 15, 2020

@GrahamcOfBorg build graalvm11-ce

@bennyandresen
Copy link
Contributor Author

@bennyandresen bennyandresen commented Jul 15, 2020

@jlesquembre I already had them downloaded and therefor didn't realize that only wasm is included in the core graalvm release. graalpython and truffleruby are their own projects but are vm-compatible on their own release page.

Fixed in the commit I just updated.

@jlesquembre
Copy link
Contributor

@jlesquembre jlesquembre commented Jul 15, 2020

@GrahamcOfBorg build graalvm11-ce

@jlesquembre
Copy link
Contributor

@jlesquembre jlesquembre commented Jul 15, 2020

@bennyandresen thanks, looks better now, I was able to build it locally, still need to test the executables

@jlesquembre
Copy link
Contributor

@jlesquembre jlesquembre commented Jul 15, 2020

@GrahamcOfBorg build graalvm8-ce

@jlesquembre
Copy link
Contributor

@jlesquembre jlesquembre commented Jul 15, 2020

I was able to compile a clojure project, looks good to me

@jlesquembre
Copy link
Contributor

@jlesquembre jlesquembre commented Jul 16, 2020

I found an error:

/nix/store/1kx44zs01d9ksibdxk3gi15akq5i8nfg-graalvm11-ce-20.1.0/lib/installer/bin/gu --help
/nix/store/1kx44zs01d9ksibdxk3gi15akq5i8nfg-graalvm11-ce-20.1.0/lib/installer/bin/gu: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

@bennyandresen
Copy link
Contributor Author

@bennyandresen bennyandresen commented Jul 16, 2020

I found an error:

/nix/store/1kx44zs01d9ksibdxk3gi15akq5i8nfg-graalvm11-ce-20.1.0/lib/installer/bin/gu --help
/nix/store/1kx44zs01d9ksibdxk3gi15akq5i8nfg-graalvm11-ce-20.1.0/lib/installer/bin/gu: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

thanks! fixed in latest push.

@ludovicc
Copy link

@ludovicc ludovicc commented Jul 18, 2020

Hello, I don't know if you can add support for Darwin. I did a quick search and found that GHC supports Darwin using dynamic libraries (https://github.com/NixOS/nixpkgs/blob/dbb7aaef62bb6e7cece233111914879c46d593fa/pkgs/development/compilers/ghc/8.6.5-binary.nix)
I can help testing on Darwin, but I cannot figure out how to make Graal work on this platform.

@bennyandresen
Copy link
Contributor Author

@bennyandresen bennyandresen commented Jul 22, 2020

Hello, I don't know if you can add support for Darwin. I did a quick search and found that GHC supports Darwin using dynamic libraries (https://github.com/NixOS/nixpkgs/blob/dbb7aaef62bb6e7cece233111914879c46d593fa/pkgs/development/compilers/ghc/8.6.5-binary.nix)
I can help testing on Darwin, but I cannot figure out how to make Graal work on this platform.

I tried for 30 minutes to integrate both platforms into a single file, but I wasn't successful.
Maybe you can fork the file I provided and change all the linux strings to darwin and see if that works for you? Then I can try again to see if I can incorporate it.

@jlesquembre
Copy link
Contributor

@jlesquembre jlesquembre commented Jul 27, 2020

I was using graalvm11-ce to compile more clojure projects, and looks good to me, I didn't find any issues.

I'd like to get this merged and update clj-kondo and babashka to use this version until we fix the version built from the sources. Darwin users, in the meantime, can use overlays to build that packages from a different graalvm version. That's anyways necessary since the current graalvm version is broken on master. (https://hydra.nixos.org/job/nixpkgs/trunk/graalvm8.x86_64-darwin)

@thenonameguy
Copy link

@thenonameguy thenonameguy commented Aug 18, 2020

@bennyandresen can you please update your maintainer references so the build completes/this PR can get merged? I'm eager to use this and use the resulting binary cached derivation.

@seylerius
Copy link
Contributor

@seylerius seylerius commented Sep 2, 2020

What's missing here to make this buildable? Do we need to rerun the tests? It looks like the error listed in the failing check has been resolved.

@ncryptid
Copy link
Contributor

@ncryptid ncryptid commented Sep 7, 2020

Something I noticed using this: If I try to compile a static native-image (with the "--static" flag), I get some weird errors with Clojure related to using the clj-http-lite library trying to make simple http requests. The executable will run fine from the computer I compiled it on, but if I run it from a different one, it fails. I can compile it dynamically but then it's guaranteed to not work pretty much anywhere else because of how the paths are in nix.

The same exact code works fine anywhere (or at least on Debian, Ubuntu, and NixOS) if I compile it on Ubuntu by installing the latest version of graalvm-ce using the instructions in the docs. I'm guessing this would have something to do with how the static zlib and glibc stuff is being handled in the nix derivation but I don't know anywhere near enough about graalvm or java to say for sure.

(ns gema.main
  (:gen-class)
  (:require [clj-http.lite.client :as client]))

(defn test-get []
  (println (client/get "http://example.org")))
Exception in thread "main" java.net.UnknownHostException: example.org
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at java.net.Socket.connect(Socket.java:556)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
        at sun.net.www.http.HttpClient.New(HttpClient.java:339)
        at sun.net.www.http.HttpClient.New(HttpClient.java:357)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
        at clj_http.lite.core$request.invokeStatic(core.clj:76)
        at clj_http.lite.core$request.invoke(core.clj:42)
        at clojure.lang.Var.invoke(Var.java:384)
        at clj_http.lite.client$wrap_query_params$fn__996.invoke(client.clj:167)
        at clj_http.lite.client$wrap_user_info$fn__1006.invoke(client.clj:191)
        at clj_http.lite.client$wrap_url$fn__1023.invoke(client.clj:213)
        at clj_http.lite.client$wrap_redirects$fn__940.invoke(client.clj:47)
        at clj_http.lite.client$wrap_decompression$fn__947.invoke(client.clj:63)
        at clj_http.lite.client$wrap_input_coercion$fn__962.invoke(client.clj:115)
        at clj_http.lite.client$wrap_output_coercion$fn__953.invoke(client.clj:72)
        at clj_http.lite.client$wrap_exceptions$fn__933.invoke(client.clj:33)
        at clj_http.lite.client$wrap_basic_auth$fn__1001.invoke(client.clj:181)
        at clj_http.lite.client$wrap_accept$fn__974.invoke(client.clj:135)
        at clj_http.lite.client$wrap_accept_encoding$fn__980.invoke(client.clj:146)
        at clj_http.lite.client$wrap_content_type$fn__969.invoke(client.clj:127)
        at clj_http.lite.client$wrap_form_params$fn__1018.invoke(client.clj:208)
        at clj_http.lite.client$wrap_method$fn__1013.invoke(client.clj:196)
        at clj_http.lite.links$wrap_links$fn__901.invoke(links.clj:65)
        at clj_http.lite.client$wrap_unknown_host$fn__1028.invoke(client.clj:219)
        at clj_http.lite.client$get.invokeStatic(client.clj:273)
        at clj_http.lite.client$get.doInvoke(client.clj:270)
        at clojure.lang.RestFn.invoke(RestFn.java:410)
        at gema.main$test_get.invokeStatic(main.clj:43)
        at gema.main$_main.invokeStatic(main.clj:67)
        at gema.main$_main.doInvoke(main.clj:67)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at gema.main.main(Unknown Source)

@glittershark
Copy link
Member

@glittershark glittershark commented Oct 5, 2020

if @bennyandresen is busy and it feels kosher to people I'd love to take this and rebase it to fix the eval error

@bennyandresen
Copy link
Contributor Author

@bennyandresen bennyandresen commented Oct 5, 2020

if @bennyandresen is busy and it feels kosher to people I'd love to take this and rebase it to fix the eval error

Please go ahead. If there is a way for me to transfer this PR over to you, I'll do that.

@glittershark
Copy link
Member

@glittershark glittershark commented Oct 5, 2020

I'll just open a new one. thanks 😄

@glittershark glittershark mentioned this pull request Oct 5, 2020
10 tasks
@glittershark
Copy link
Member

@glittershark glittershark commented Oct 5, 2020

Done, along with a version bump. This can safely be closed now.

@bennyandresen
Copy link
Contributor Author

@bennyandresen bennyandresen commented Oct 5, 2020

Closing in favor of #99631

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

8 participants