Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

julia: 1.1 -> 1.3 #79174

Merged
merged 4 commits into from Feb 18, 2020
Merged

julia: 1.1 -> 1.3 #79174

merged 4 commits into from Feb 18, 2020

Conversation

@cstich
Copy link
Contributor

@cstich cstich commented Feb 3, 2020

This updates julia to version 1.3.1. I couldn't get some of the dependencies as defined in the original build script to work, so this is building the source binary that includes all dependencies as provided by upstream. Over time I guess it would make sense to work on replacing the bundled dependencies with native ones.

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • [x ] NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • [x ] Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
@7c6f434c
Copy link
Member

@7c6f434c 7c6f434c commented Feb 4, 2020

Hm, what happenned to all-packages.nix change adding julia_1_3 = … ?

Added all-packages.nix
@cstich cstich force-pushed the cstich:julia-1.3.1 branch from 3268a61 to 194f9ce Feb 4, 2020
@cstich
Copy link
Contributor Author

@cstich cstich commented Feb 4, 2020

Hm, what happenned to all-packages.nix change adding julia_1_3 = … ?

Short answer, I looked for the place where top-level packages are defined but could not find it. So thanks for pointing that out. I have added it now to all-packages.nix. Also this might possibly be a very beginner question but how do I actually test the package definition in all-packages.nix?

@7c6f434c
Copy link
Member

@7c6f434c 7c6f434c commented Feb 4, 2020

Hm, your file misses the Darwin-only arguments, as revealed by ofBorg failure.

To test that your checkout where you prepare the PR has a usable Julia 1.3 package, do nix-build -o julia-1.3 /path/to/nixpkgs-checkout -A julia_1_3 to get a julia-1.3 symlink to the built package.

In general, I guess the Nix manual could be useful (note that Nix, Nixpkgs and NixOS have their own different manuals)

@cstich
Copy link
Contributor Author

@cstich cstich commented Feb 4, 2020

I'll look into this. Thanks for the pointers.

@ofborg ofborg bot requested review from rbvermaa, garrison and 7c6f434c Feb 13, 2020
@7c6f434c 7c6f434c merged commit 2a52e5e into NixOS:master Feb 18, 2020
15 checks passed
15 checks passed
julia_13 on aarch64-linux No attempt
Details
julia_13 on x86_64-linux Timed out, unknown build status
Details
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Feb 18, 2020
julia: 1.1 -> 1.3
(cherry picked from commit 2a52e5e)
@GTrunSec
Copy link
Contributor

@GTrunSec GTrunSec commented Feb 20, 2020

julia test failed

│ %5 = Test.Expr(:comparison, %2, ==, %4) [20/1955] │ %6 = Test.Expr(:comparison, $(QuoteNode(:(test12806()))), :(==), $(QuoteNode(:((true, true, false, [1]))))) │ @_2 = Test.eval_test(%5, %6, $(QuoteNode(:(#= /build/source/test/core.jl:951 =#))), $(QuoteNode(false))) └── $(Expr(:leave, 1)) 3 ─ goto #9 4 ┄ $(Expr(:leave, 1)) 5 ─ #4111#_e = $(Expr(:the_exception)) │ %12 = #4111#_e isa Test.InterruptException └── goto #7 if not %12 6 ─ Test.rethrow() └── goto #8 7 ─ false 8 ┄ %17 = #4111#_e │ %18 = Base.getproperty(Test.Base, :catch_stack) │ %19 = (%18)() │ @_2 = Test.Threw(%17, %19, $(QuoteNode(:(#= /build/source/test/core.jl:951 =#)))) └── $(Expr(:pop_exception, :(%1))) 9 ┄ %22 = @_2 │ Test.do_test(%22, $(QuoteNode(:(test12806() == (true, true, false, [1]))))) └── return ), 0x0000000000000001) … Ptr{Nothing} @0x00007fffc9b84b25, Ptr{Nothing} @0x00007ffff7aeb8a1, Ptr{Nothing} @0x00007fffc9b849d8, Ptr{Nothing} @0x00007ffff7aeb8a1, Ptr{Nothing} @0x00007fffc9b8487e, Ptr{Nothing} @0x00007ffff7aeb8a1, Ptr{Nothing} @0x00007fffc9b846f8, Ptr{Nothing} @0x00007fffc9b81c35, Ptr{Nothing} @0x00007ffff7af78ee, Ptr{Nothing} @0x0000000000000000] == Base.StackTraces.StackFrame[] Error in testset REPL: Test Failed at /build/source/usr/share/julia/stdlib/v1.3/REPL/test/replcompletions.jl:107

), 0x0000000000000001) … Ptr{Nothing} @0x00007fffc9b84b25, Ptr{Nothing} @0x00007ffff7aeb8a1, Ptr{Nothing} @0x00007fffc9b849d8, Ptr{Nothing} @0x00007ffff7aeb8a1, Ptr{Nothing} @0x00007fffc9b8487e, Ptr{Nothing} @0x00007ffff7aeb8a1, Ptr{Nothing} @0x00007fffc9b846f8, Ptr{Nothing} @0x00007fffc9b81c35, Ptr{Nothing} @0x00007ffff7af78ee, Ptr{Nothing} @0x0000000000000000] == Base.StackTraces.StackFrame[]
Error in testset REPL:
Test Failed at /build/source/usr/share/julia/stdlib/v1.3/REPL/test/replcompletions.jl:107
Expression: count(isequal("REPL"), c) == 1
Evaluated: 2 == 1
Error in testset precompile:
Error During Test at none:1
Test threw exception
Expression: precompile

Error in testset stress:
Error During Test at none:1
Test threw exception
Expression: stress

ERROR: LoadError: Test run finished with errors
in expression starting at /build/source/test/runtests.jl:62
make[1]: *** [Makefile:25: all] Error 1
make[1]: Leaving directory '/build/source/test'
make: *** [Makefile:584: testall] Error 2
builder for '/nix/store/az57bwbvh5fd346i29q4cwkqcnxsmdm3-julia-1.3.1.drv' failed with exit code 2
error: build of '/nix/store/az57bwbvh5fd346i29q4cwkqcnxsmdm3-julia-1.3.1.drv' failed

@cstich
Copy link
Contributor Author

@cstich cstich commented Feb 20, 2020

julia test failed

This looks like it this issue: JuliaLang/julia#32377
The workaround seems to be to use multiple threads to build julia. How many threads are you using to build it?
Alternatively we can think about skipping that particular test, although I don't know how from the top of my hat.

EDIT: Grammar

@GTrunSec
Copy link
Contributor

@GTrunSec GTrunSec commented Feb 23, 2020

julia test failed

This looks like it this issue: JuliaLang/julia#32377
The workaround seems to be to use multiple threads to build the julia. How many threads are you using to build it?
Alternatively we can think about skipping that particular test, although I don't know who from the top of my hat.

I didn't change the default setting of julia_1.3. So I guess the default thread number was one. Do I need
modify julia_1.3.nix to set multiple threads ? or which arguments I can set to skip the particular test?

@cstich
Copy link
Contributor Author

@cstich cstich commented Feb 24, 2020

Parallel builds are enabled for julia_1.3. Can you check the environment variable NIX_BUILD_CORES and see how many cores you have enabled for building nix expressions?

And I unfortunately do not know how to skip tests.

@GTrunSec
Copy link
Contributor

@GTrunSec GTrunSec commented Feb 24, 2020

Parallel builds are enabled for julia_1.3. Can you check the environment variable NIX_BUILD_CORES and see how many cores you have enabled for building nix expressions?

And I unfortunately do not know how to skip tests.

before building LLVM(it did my cup core number), when compiling Julia that did a single thread.

/build/source/src/codegen.cpp:6415:34: warning: 'retval' may be used uninitialized in this function [-Wmaybe-uninitialized]              
 6415 |             ctx.builder.CreateRet(retval);                                                                                       
      |             ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~                                                                                        
    LINK usr/lib/libjulia.so.1.3                                                                                                         
Warning: git information unavailable; versioning information limited                                                                     
    CC ui/repl.o                                                                                                                         
    LINK usr/bin/julia                                                                                                                   
Warning: git information unavailable; versioning information limited                                                                     
    JULIA usr/lib/julia/corecompiler.ji    


///////////////       it's going to  single thread                                                                                  
essentials.jl                                                                                                                            
ctypes.jl                                                                                                                                
generator.jl                                                                                                                             
reflection.jl                                                                                                                            
options.jl                                                                                                                               
promotion.jl                                                                                                                             
tuple.jl                                                                                                                                 
pair.jl                                                                                                                                  
traits.jl                      
@GTrunSec
Copy link
Contributor

@GTrunSec GTrunSec commented Feb 24, 2020

I just overrode the argument of checkTarget ="testall" to none

julia_13 = pkgs.julia_13.overrideAttrs(oldAttrs: {checkTarget = "";});  

it passed the build.

@twhitehead
Copy link
Contributor

@twhitehead twhitehead commented Feb 26, 2020

I believe the issue is that the test system doesn't truly isolate tests. Specifically, the Main module space is shared by the test running system and the tests themselves. This has bad effects in the REPL module which has several implicit and explicit references to Main.

What is screwing up this particular test is that test/runtests.jl imports REPL to get REPL.Terminal. This adds the module REPL to the Main namespace and causes the using REP completion test in stdlib/REPL/test/replcompletions.jl to return the module REPL brought into scope by the test runner in addition to the expected package REPL.

I tried to see if this issue could be worked around by wrapping replcompletions.jl in a module Main statement to hide the global one. While it did resolve this issue, it causes a few of the other later tests to stop working for reasons I haven't fully figured out. My wild guess would be that it might have to do with my Main not having the magic global Main export everything automatically property.

A horrible hack to resolve the issue is to just change import REPL in test/runtests.jl to import REPL.Terminal and all REPL.Terminal. call prefixes to just Terminal. I only mention this to prove the point though, not to suggest it be done.

@twhitehead
Copy link
Contributor

@twhitehead twhitehead commented Feb 26, 2020

Sorry for the last post. Had intended to post that on the julia issue tracker. Didn't realize I had jumped repositories when clicking links.

As it happens though, I have been actually working on this in order to get Julia 1.3 compiling under Nix. The other issue I've ran into is the exceptions test set can explode rather spectacularly due to an earlier core module test exposing JuliaLang/julia#34579.

My personal recommendation would be to just disable both this one test and the core module test (test12806).

@twhitehead
Copy link
Contributor

@twhitehead twhitehead commented Feb 26, 2020

Forgot to mention, you will want to set USER (why is this set to root in the build environment?) and HOME as well or you will have a bunch of other test failures.

HOME=$NIX_BUILD_TOP
USER=$(whoami)
@cstich
Copy link
Contributor Author

@cstich cstich commented Feb 27, 2020

@twhitehead How do I selectively disable tests for julia?
Also HOME is set to "$NIX_BUILD_TOP". And I do not know why USER is set to root, I just copied things from version 1.1 over to this. At some point this whole expression could probably use a re-write but I unfortunately do not have time for this right now.

@twhitehead
Copy link
Contributor

@twhitehead twhitehead commented Mar 2, 2020

You either have to do some text substitution on the test file (e.g., use sed to put a # in front of the line that runs the test) or remove or zero the test file itself. I'll submit my suggested cleanups when I get the chance (currently just having problems with one more test that I'm tracking down).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.