Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Win32 segfault in numbers test with precompiled sys.dll #8895

Closed
tkelman opened this issue Nov 3, 2014 · 15 comments
Closed

Win32 segfault in numbers test with precompiled sys.dll #8895

tkelman opened this issue Nov 3, 2014 · 15 comments
Labels
system:windows Affects only Windows system:32-bit Affects only 32-bit systems

Comments

@tkelman
Copy link
Contributor

tkelman commented Nov 3, 2014

Looks like it was caused by the call overloading merge (#8712):

    JULIA test/all
     * linalg1
     * linalg2
     * linalg3
     * linalg4
     * linalg/triangular
     * linalg/tridiag
     * core
     * keywordargs
     * numbers

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x1 -- unknown function (ip: 1)
unknown function (ip: 1)
unknown function (ip: 104293888)
unknown function (ip: 1668356486)
unknown function (ip: 1668379800)
unknown function (ip: 1967365175)
unknown function (ip: 1668307051)
unknown function (ip: 1668356486)
unknown function (ip: 1668379274)
unknown function (ip: 1668356486)
unknown function (ip: 63554567)
unknown function (ip: 1668379274)
unknown function (ip: 1999495203)
unknown function (ip: 1668625304)
unknown function (ip: 1668620938)
unknown function (ip: 4202922)
unknown function (ip: 4199408)
unknown function (ip: 1961112458)
unknown function (ip: 1999544178)
unknown function (ip: 1999544133)
/bin/sh: line 1:   928 Segmentation fault      /home/MPC_CAR/julia32/usr/bin/julia.exe --check-bounds=yes -f ./runtests.jl all
Makefile:9: recipe for target 'all' failed
make[1]: *** [all] Error 139
Makefile:413: recipe for target 'testall1' failed
make: *** [testall1] Error 2

This may be a Win32-specific problem, would need someone with 32 bit Linux to test and confirm.

$ usr/bin/julia -e 'versioninfo()'
Julia Version 0.4.0-dev+1385
Commit 27f1b81* (2014-11-03 19:19 UTC)
Platform Info:
  System: Windows (i686-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-4702MQ CPU @ 2.20GHz
  WORD_SIZE: 32
  BLAS: libopenblas (DYNAMIC_ARCH NO_AFFINITY Nehalem)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

Here's the bisect log, the skips were from failures to compile or otherwise different-looking failures, all from within the same call-overload PR I think.

$ git bisect log
git bisect start
# bad: [27f1b8154873a25492482d7856e775f027dcb0c0] Fix #8874
git bisect bad 27f1b8154873a25492482d7856e775f027dcb0c0
# bad: [90094210bfab81dd74323a58c145722329a98b78] Merge pull request #8790 from JuliaLang/jcb/rm_evaluate
git bisect bad 90094210bfab81dd74323a58c145722329a98b78
# good: [dee2cf4ab74fe352773c0c6e14a85bbffa2f566f] Merge pull request #8658 from JuliaLang/cjh/doc-arrays
git bisect good dee2cf4ab74fe352773c0c6e14a85bbffa2f566f
# good: [e22f90edc81a19b42b12997cbf3bc3938b743215] fix for linking embedding example on windows
git bisect good e22f90edc81a19b42b12997cbf3bc3938b743215
# bad: [ff88a00c04ec003b74605cc395b76ed385c39b59] avoid overflowing environment in type intersection when a typevar meets the same type repeatedly
git bisect bad ff88a00c04ec003b74605cc395b76ed385c39b59
# bad: [0f67c6925ddb21c86733e855d38d77870e386b6c] remove old ASCIIString constructor code from dump.c
git bisect bad 0f67c6925ddb21c86733e855d38d77870e386b6c
# bad: [24637047860ab7f4cfc0c0d51dd523f6aeca3dc1] fix codegen for changes to apply() for call overloading
git bisect bad 24637047860ab7f4cfc0c0d51dd523f6aeca3dc1
# good: [b430c48e46c875306c4baaf00b44d991a9fe1fe3] get basic call overloading working with 1 bootstrap stage
git bisect good b430c48e46c875306c4baaf00b44d991a9fe1fe3
# skip: [a4dcc697c9f2279167c397d0807e3e04efb199a8] Merge branch 'master' of github.com:JuliaLang/julia into call_overload
git bisect skip a4dcc697c9f2279167c397d0807e3e04efb199a8
# skip: [bd955f2d08dfe295056c55aa9677d1cc7039188a] update emit_call for jl_fptr_llvmt => jl_pfptr_llvmt change
git bisect skip bd955f2d08dfe295056c55aa9677d1cc7039188a
# bad: [aeae50f4e84870020bb4d65d8a26d7ba7e08cb69] some work on call overloading
git bisect bad aeae50f4e84870020bb4d65d8a26d7ba7e08cb69
# skip: [27e476c2116d2f16bdb85a3ce75a95580da9389d] Merge branch 'master' into call_overload
git bisect skip 27e476c2116d2f16bdb85a3ce75a95580da9389d
# only skipped commits left to test
# possible first bad commit: [aeae50f4e84870020bb4d65d8a26d7ba7e08cb69] some work on call overloading
# possible first bad commit: [a4dcc697c9f2279167c397d0807e3e04efb199a8] Merge branch 'master' of github.com:JuliaLang/julia into call_overload
# possible first bad commit: [bd955f2d08dfe295056c55aa9677d1cc7039188a] update emit_call for jl_fptr_llvmt => jl_pfptr_llvmt change
# possible first bad commit: [27e476c2116d2f16bdb85a3ce75a95580da9389d] Merge branch 'master' into call_overload
@tkelman tkelman added the system:32-bit Affects only 32-bit systems label Nov 3, 2014
@jakebolewski
Copy link
Member

Tests pass on Linux 32 bit, so I believe this is a Win32 problem.

@jakebolewski jakebolewski added the system:windows Affects only Windows label Nov 3, 2014
@tkelman tkelman changed the title 32 bit segfault in numbers test Win32 segfault in numbers test with precompiled sys.dll Nov 3, 2014
@tkelman
Copy link
Contributor Author

tkelman commented Nov 3, 2014

Thanks Jake. Deleting sys.dll makes the problem go away, so this does look very similar to the Heisenbug I saw on that branch, just more repeatable now. Doesn't happen if I run tests in parallel, or using julia-debug. Can you try make testall1 on 32 bit Linux just to be sure?

@jakebolewski
Copy link
Member

Tests still pass with make testall1

@tkelman
Copy link
Contributor Author

tkelman commented Nov 4, 2014

This might not be in the numbers test at all - if I add a bunch of printlns, even on the last line of test/numbers.jl, the segfault happens after all of them execute. So maybe it's actually in serial pmap or the runtests function?

This is probably another "hope it goes away when we upgrade LLVM" issue.

@nalimilan
Copy link
Member

Doesn't happen on my Linux nightlies either, FWIW.

@mlubin
Copy link
Member

mlubin commented Nov 4, 2014

We're seeing segfaults on Win32 with JuMP also: https://ci.appveyor.com/project/mlubin/jump-jl/build/1.0.592/job/g3m4lk4d48lfxxr5

@tkelman
Copy link
Contributor Author

tkelman commented Nov 4, 2014

@mlubin interesting, but probably unrelated as the Windows binaries shouldn't be including sys.dll right now

@tkelman tkelman mentioned this issue Nov 12, 2014
Closed
staticfloat added a commit that referenced this issue Nov 23, 2014
@staticfloat
Remove sys.dll on windows when running make dist, until related iss…
02e57db 
…ues are solved (#8895, among others)
staticfloat added a commit that referenced this issue Nov 23, 2014
@staticfloat
Remove sys.dll on windows when running make dist, until related iss…
c9da7cb 
…ues are solved (#8895, among others)
@vtjnash
Copy link
Sponsor Member

vtjnash commented Nov 23, 2014

this is strange, since the sys.dll file should be fine to include for win32

@tkelman
Copy link
Contributor Author

tkelman commented Nov 23, 2014

I'll note that I haven't had this happen as repeatably for the last week or so. Not sure what changed, or if that's conclusive enough to close the issue as the underlying cause might still be present?

@vtjnash
Copy link
Sponsor Member

vtjnash commented Nov 23, 2014

its always possible that it got fixed along the way, but seems somewhat unlikely. it'll be hard to fix if it isn't reproducible however – sometimes it's easier to just close the issue and wait to see if it comes up again

@tkelman
Copy link
Contributor Author

tkelman commented Nov 23, 2014

It should be reproducible at any of the commits that are listed above as bad:, but sure I'll close for now and reopen if it comes back.

@tkelman tkelman closed this as completed Nov 23, 2014
waTeim pushed a commit to waTeim/julia that referenced this issue Nov 23, 2014
@staticfloat @waTeim
Remove sys.dll on windows when running make dist, until related iss…
a8806c1 
…ues are solved (JuliaLang#8895, among others)
@tkelman tkelman mentioned this issue Nov 25, 2014
Merged
@tkelman
Copy link
Contributor Author

tkelman commented Nov 30, 2014

This is back on master at 84612d4, but now with a backtrace thanks to @vtjnash's awesome recent work.

     * numbers

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x620a0afe -- jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_profile_is_running at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_gc_collect at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
alloc_4w at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_exprn at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_copy_ast at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_copy_ast at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
anonymous at no file:1945
jl_eval_with_compiler_p at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_parse_eval_all at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_load_ at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
runtests at D:\cygwin64\home\Tony\julia32\test\testdefs.jl:5
jl_apply_generic at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
RtlFreeHeap at C:\Windows\SysWOW64\ntdll.dll (unknown line)
free at C:\Windows\syswow64\msvcrt.dll (unknown line)
ht_keyindex2 at dict.jl:539
utf8proc_NFKC at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
SetLocalTime at C:\Windows\syswow64\KERNELBASE.dll (unknown line)
SetLocalTime at C:\Windows\syswow64\KERNELBASE.dll (unknown line)
jl_alloc_tuple_uninit at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_f_apply at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
jl_f_apply at D:\cygwin64\home\Tony\julia32\usr\bin\libjulia.dll (unknown line)
/bin/sh: line 1: 14100 Segmentation fault      /home/Tony/julia32/usr/bin/julia.exe --check-bounds=yes -f ./runtests.jl all
Makefile:9: recipe for target 'all' failed
make[1]: *** [all] Error 139
Makefile:433: recipe for target 'testall1' failed
make: *** [testall1] Error 2

I see signs of gc and utf8proc in there?

@tkelman tkelman reopened this Nov 30, 2014
@vtjnash
Copy link
Sponsor Member

vtjnash commented Nov 30, 2014

Looks like a missing gc root. The sys.dll thing is probably just a red herring, caused by altering the allocation/gc order

@vtjnash
Copy link
Sponsor Member

vtjnash commented Nov 30, 2014

although, it also looks like a corrupted stack.

i wonder if backtraces would be even better if we disabled FPO (-fno-omit-frame-pointer), at least until we switch to using link.exe (supported by clang3.5+ and msvc). some of those names can't be correct.

@tkelman tkelman mentioned this issue Dec 16, 2014
Closed
@tkelman
Copy link
Contributor Author

tkelman commented Dec 17, 2014

I think the fixes for #9366 may have solved this. I'll reopen if I see it come back.

@tkelman tkelman closed this as completed Dec 17, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
system:windows Affects only Windows system:32-bit Affects only 32-bit systems
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@vtjnash @nalimilan @jakebolewski @mlubin @tkelman