RFC: MSVC continued

[tkelman]

Following up on #6230, #6349, and others. This gets past the stack overflow when compiling inference.jl into the system image, but craps out at pcre.jl because the preprocessed PCRE headers don't come out correctly from the MSVC preprocessor (it doesn't understand -dM). Putting this up so people can look at it, I'll keep thinking about workarounds for the preprocessor issues - can probably copy some of them straight out of the MinGW binaries.

How to run this, assuming you have Visual Studio 2013 (express should work) installed:

• start a command prompt, cd "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC"
• vcvarsall x86_amd64
• cd to your MSYS installation, whichever folder sh.exe is located in (either bin or usr/bin depending whether you're using MSYS1 or a recent MSYS2 - best to use msysgit here, otherwise you'll likely get an error message due to the MSYS version of link.exe showing up first in the path)
• set MSYSTEM=MINGW32
• sh --login
• cd to your Julia directory, checkout this branch latest master
• make sure msys can find 7z on the path
• ARCH=x86_64 USEMSVC=1 contrib/windows/msys_build.sh

This downloads the latest Windows nightly binary and uses as many dependencies from it as possible, and downloads an MSVC build of LLVM that I made in early April. I should update that with our latest patches, and build a 32-bit version with VS 2013 at some point.

[StefanKarpinski]

The preprocessor thing is kind of a hack. We could check in the generated file if necessary.

[tkelman]

I think I tracked it down once via git blame to some time in 2010 (#6230 (comment)). MSVC also can't compile/preprocess from stdin, so errno_h.jl doesn't come out right either.

[vtjnash]

I think I tracked it down once via git blame to some time in 2010 (#6230 (comment)). MSVC also can't compile/preprocess from stdin, so errno_h.jl doesn't come out right either.

From Google, it seems that it read from stdin, but it doesn't need the - flag to do so.

The preprocessor thing is kind of a hack. We could check in the generated file if necessary.

Might not want to do that, since they can vary by platform. But we could just use the normal preprocessor capabilities directly (like we do with uv_constants.h and build.h)

[tkelman]

I updated the build script (to copy pcre_h.jl from the nightly) and pushed another commit that actually gets the sysimg to finish building. Making sys.dll with MSVC won't work right now, since sys.o doesn't have COMDAT stuff that the MSVC linker wants.

And the newlib version of getopt can't handle options with = settings like --check-bounds=yes. The musl one calls write for error output, I'll try refactoring it to use fprintf instead.

Trying to run the MSVC Julia gives some messages 'x86_64' is not a recognized processor for this target (ignoring processor) and quits immediately after showing the banner, so something's still not right. Probably LLVM misconfigured in some way. But if I take the sys.ji generated by MSVC Julia, and move that over to replace the system image of a MinGW Julia, then the latter appears to still pass tests okay.

(Unfortunately running this same setup on AppVeyor gives a whole bunch of access violations at the same place this used to hit a stack overflow https://ci.appveyor.com/project/tkelman/julia/build/1.0.489, I suspect the VM's might not have enough memory to handle 64-bit Julia codegen)

[vtjnash]

I think we can avoid this complicating ifndef if we write this as:

#define CNAME jl_longjmp
#include "./../deps/openlibm/i387/ENTRY.h"
<function definition here>
#include "./../deps/openlibm/i387/END.h"

where ENTRY.h is:

#define _START_ENTRY .p2align 2,0x90
#if defined(__linux__) || defined(__FreeBSD__) || defined(__ELF__)
#ifndef __APPLE__
#define EXT(csym)     csym
#endif
#define HIDENAME(asmsym)    .asmsym
_START_ENTRY
.globl EXT(CNAME)
.type EXT(CNAME),@function
EXT(CNAME):

#elif defined(_WIN32)
#define EXT(csym)     _##csym
#define HIDENAME(asmsym)    .asmsym

#ifndef _MSC_VER
.intel_syntax
.text
_START_ENTRY
.globl EXT(CNAME)
.section .drectve
.ascii " -export:" #CNAME
.section .text
.def EXT(CNAME)
.scl 2
.type 32
.endef
EXT(CNAME):
#else
.586
.model small,C
CNAME proc
#endif

#endif

and END.h is:

#if defined(__linux__) || defined(__FreeBSD__) || defined(__ELF__)
.size CNAME, . - CNAME
#else
#ifndef _MSC_VER
.end
#else
CNAME endp
#endif
#endif

#undef CNAME
#undef CNAME
#undef _START_ENTRY
#ifndef __APPLE__
#undef EXT
#endif

[tkelman]

I was wanting to avoid having to add extra little files over in a different submodule, but yeah this version should work and might be step one of eventually attempting openlibm with MSVC. Though I think most of the assembly in openlibm is ATT syntax, not Intel, right?

[vtjnash]

Trying to run the MSVC Julia gives some messages 'x86_64' is not a recognized processor for this target (ignoring processor) and quits immediately after showing the banner

that can happen if you are forcing ARCH (which you are), but not also setting MARCH to a reasonable value. we could probably also just convert x86_64 into the valid x86-64 setting

[tkelman]

Adding override MARCH = x86-64 to Make.user gets rid of the warnings, but doesn't change the REPL exiting immediately after showing the banner. Non-interactive hello world usr/bin/julia -e "println(\"hello\")" works, but not much else.

[tkelman]

@vtjnash reminded me of this in a different issue - I think codegen might be nonfunctional with these MSVC experiments. Do we need to JIT code to a different ABI if LLVM and Julia are built with MSVC?

Also, llvmcall (specifically size_t nargt = jl_tuple_len(tt); Value *argvals[nargt];) doesn't compile in MSVC.

d:\code\msys64\home\tony\julia\src\ccall.cpp(639) : error C2057: expected consta
nt expression
d:\code\msys64\home\tony\julia\src\ccall.cpp(639) : error C2466: cannot allocate
 an array of constant size 0
d:\code\msys64\home\tony\julia\src\ccall.cpp(639) : error C2133: 'argvals' : unk
nown size

Any ideas?

[tkelman]

I switched from newlib's version of getopt to modifying musl's slightly, it handles --check-bounds=yes better. I think this is merge-able at the moment if no objections to the changes. Codegen still appears broken with MSVC-compiled Julia, might be worth revisiting after LLVM 3.5.

[tkelman]

Any other comments on the minor tweaks here? Suggestions for things to try to get codegen to work? musl's MIT licensed so I don't think there should be any concern adding 2 files from it, right?

[vtjnash]

A link to musl's LICENSE file should probably go into LICENSE, then this looks good to merge

[vtjnash]

note, your instructions in the first post are missing references to 7z and MARCH which are needed

[vtjnash]

it would also be preferable for your msys_build.sh script to not replace binaries in /usr/bin, but simply to shadow them by downloading them to dist-extras, and pushing that onto $PATH

[tkelman]

A link to musl's LICENSE file should probably go into LICENSE, then this looks good to merge

Done. We aren't linking to it so much as copying and modifying a couple small pieces of its source, but whatever.

note, your instructions in the first post are missing references to 7z and MARCH which are needed

7z added. MARCH gets set in Make.user by the script, doesn't need to be set on the command line.

it would also be preferable for your msys_build.sh script to not replace binaries in /usr/bin, but simply to shadow them

Done.

Any ideas on what's broken about codegen?

[tkelman]

Maybe it isn't codegen but something repl-related? Or both...
I can generate vc120.pdb files by adding -Zi to CC, but not sure how to get WinDbg to use them.

[tkelman]

Interesting, the REPL actually starts and I can do some very basic things if I comment out

julia/src/jl_uv.c

Line 526 in a08ae29

free(req);

(probably not a good idea to do this though?)

It looks like I'm getting undefined variable errors but libuv isn't displaying them correctly for some reason?

[vtjnash]

it seems like we are trying to free a junk req? or perhaps it was allocated by a different msvcrt? (not sure how, it was allocated here:

julia/src/jl_uv.c

Line 633 in a08ae29

uv_write_t *uvw = (uv_write_t*)malloc(sizeof(uv_write_t)+n);

)

but you mentioned that a sys.ji file taken from the nightlies doesn't have this issue? so it sounds more like memory corruption than a libuv or malloc issue

[tkelman]

but you mentioned that a sys.ji file taken from the nightlies doesn't have this issue?

Err, not exactly. I can try that. What I had tried earlier was taking sys.ji created by MSVC Julia and used it with a nightly, and the nightly still seemed to work fine.

[vtjnash]

ah, gotcha, so reverse my statements: It's probably not a memory issue or Julia issue, but might be happening inside libuv. it would be good to see what req pointers are going into and coming out of uv_write

[tkelman]

A milestone, huh?

it would be good to see what req pointers are going into and coming out of uv_write

Suggestions for how I might go about doing that?

[vtjnash]

is the msvc llvm built with assertions. if not, could you put one up with assertions enabled?

also, add USEMSVC=1 to the Make.user file and -DEBUG to LD and -Z7 to CC and -EHsc to the CXX target and -Fo instead of -o (to fix debug target)

[tkelman]

is the msvc llvm built with assertions.

No idea. Is that default with the cmake build? It was many months ago and I don't remember what options I used. (Part of why I like autoconf - I can at least open up config.log and reconstruct exactly what I did. Unfortunately I don't think I kept the built sources around, if there's an equivalent log file for cmake.)

if not, could you put one up with assertions enabled?

I guess. LLVM takes many hours to build with MSVC, I've been avoiding doing it again.

also, add USEMSVC=1 to the Make.user file and -DEBUG to LD and -Z7 to CC and -EHsc to the CXX target and -Fo instead of -o (to fix debug target)

Slow down. Are these things you're asking me to improve about the (currently hacky) way this builds, or instructions for making a debug build, or a combination of both? -o shows up in a lot of places, it would take a lot of digging to find them all. scratch that this one's easy

[vtjnash]

all of the above. plus suggestions to try while i wait for my build to finish.

instead of -Z7, it may be better to look at the set of flags used by libuv in debug configuration and see if those are better

[mossr]

@tkelman When running msys_build.sh, I'm getting this error when trying to extract the mingw64-prec RPM file:

Processing archive: mingw64-pcre-8.34-2.fc21.noarch.rpm

Extracting  mingw64-pcre-8.34-2.fc21.noarch     Data Error

Sub items Errors: 1

[tkelman]

@mossr That may be a network issue corrupting the file. However that particular file shouldn't be needed any more, are you working from latest master or one of my branches?

[mossr]

@tkelman I'm working from your 9b687fa... commit. Should I be working from the latest master?

[tkelman]

Yeah, no reason not to, this was all merged into master a couple months ago and should be working fine. We don't have the MSVC build "officially" documented or given first-class fully-functional support (so this merged PR is still the main point of information for it), but it should compile to a REPL at least. We also aren't continuously testing against MSVC right now to save time on appveyor, but if someone commits some non-MSVC-friendly C it should usually be simple to fix.

[mossr]

@tkelman What commit was this last confirmed to work?

[tkelman]

@mossr sounds like I last checked the build about 4 days ago, not sure on the exact commit though. I know I checked at 28c4357 since that was an explicit MSVC compatibility change I asked Jeff to make in #9255.

I'm running some other builds right now so I can't check at today's master, but I don't expect any recent commits to have caused any problems. If they did it's usually a simple fix, so let me know what kind of errors you're hitting if any.

[tkelman]

Okay, looks like #9251 did actually break the MSVC build (https://ci.appveyor.com/project/tkelman/julia/build/1.0.572 cc @twadleigh), I'll see what needs to be done to fix it.

[mossr]

@tkelman It's important to mention that I'm trying to use the 2010 version of MSVC. Besides adding "msvcr100" to dlload.c and init.c, is there anything else that would be required to change?

I've ran into some issues building libuv because of a non-2013 compiler. Most of them were easy to fix, but this may result in future issues.

[twadleigh]

@tkelman : looks like MSVC can't find _open and _dup2. A google search turns up that MSVC expects to find them declared in io.h. Should be enough to (conditionally?) include it in init.c to get past that error.

[tkelman]

@mossr Yeah you should've said that earlier. MSVC 2010 isn't good enough. You'll need to recompile your own LLVM, and various other C99 bits and pieces aren't going to work. If you can get MSVC 2010 to work I'd entertain a pull request, but I'm not going to work too hard to maintain it myself. I think LLVM trunk only supports 2013 at the moment, so trying to keep old MSVC versions working is probably a futile endeavor long-term.

@twadleigh I fixed it with that include in c88315e, it doesn't seem to hurt the mingw case so I just did it for all windows builds. Granted there's a suitesparse change that's still awaiting a new nightly so at the moment an MSVC 2013 build will need to use a version of master before #9251 was merged.

[joa-quim]

Well, I don't know what you guys do to build with MSVC but I simply don't manage. I hided all directories that might interfere in the path. None of them have gcc, but still the build is done with a gcc (not msvc). I can confirm that by observing the processes running in the task manager (I see Gcc, G++, cc1)
I can only conclude that what is used is the gcc that lives in the mingw64 that is downloaded by msys_build.sh

[tkelman]

@joa-quim you're apparently going down the wrong branch of

julia/contrib/windows/msys_build.sh

Lines 86 to 119 in f140ff2

	if [ -z "$USEMSVC" ]; then
	if [ -z "`which ${CROSS_COMPILE}gcc 2>/dev/null`" ]; then
	f=mingw-w$bits-bin-$ARCH-20140102.7z
	if ! [ -e $f ]; then
	echo "Downloading $f"
	$curlflags -O $mingw-w64-dgn/files/mingw-w64/$f
	fi
	echo "Extracting $f"
	$SEVENZIP x -y $f >> get-deps.log
	export PATH=$PATH:$PWD/mingw$bits/bin
	# If there is a version of make.exe here, it is mingw32-make which won't work
	rm -f mingw$bits/bin/make.exe
	fi
	export AR=${CROSS_COMPILE}ar
	f=llvm-3.3-$ARCH-w64-mingw32-juliadeps.7z
	# The MinGW binary version of LLVM doesn't include libgtest or libgtest_main
	mkdir -p usr/lib
	$AR cr usr/lib/libgtest.a
	$AR cr usr/lib/libgtest_main.a
	else
	echo "override USEMSVC = 1" >> Make.user
	echo "override ARCH = $ARCH" >> Make.user
	echo "override XC_HOST = " >> Make.user
	export CC="$PWD/deps/libuv/compile cl -nologo -MD -Z7"
	export AR="$PWD/deps/libuv/ar-lib lib"
	export LD="$PWD/linkld link"
	echo "override CC = $CC" >> Make.user
	echo 'override CXX = $(CC) -EHsc' >> Make.user
	echo "override AR = $AR" >> Make.user
	echo "override LD = $LD -DEBUG" >> Make.user
	f=llvm-3.3-$ARCH-msvc12-juliadeps.7z
	fi

- however you're setting USEMSVC=1 it isn't working properly.

This could be improved by migrating from a shell script to cmake for MSVC support, ref #1832

[joa-quim]

Yes, a cmake solution would be ideal.

[tkelman]

Note that there are nmake files in the repository, but they're incomplete and currently unused and unmaintained. nmake is pretty limited in syntax and annoyingly incompatible with GNU make, but the fact that there's at least something to start from whereas cmake would require porting things from scratch means someone might want to go that route instead. I'd advise against it since cmake offers advantages across all platforms (#1832 (comment)), but I wouldn't oppose pull requests to update the nmake files.

[mossr]

@tkelman I built LLVM-3.3 with MSVC10 and that worked (using "Visual Studio 10 Win64").

I now get a Julia seg. fault from loading.jl when trying to build sys0.ji/.dll

    JULIA usr/lib/julia/sys0.o
exports.jl
base.jl
....
poll.jl
loading.jl

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x0 -- unknown function (ip: 0)
unknown function (ip: 0)
colon at range.jl:126
match at regex.jl:124
splitdir at path.jl:31
dirname at path.jl:39
jl_apply at c:\julia_nightly\build\julia-master\src\julia.h:983
jl_trampoline at c:\julia_nightly\build\julia-master\src\builtins.c:826
jl_apply at c:\julia_nightly\build\julia-master\src\julia.h:983
jl_apply_generic at c:\julia_nightly\build\julia-master\src\gf.c:1643
anonymous at sysimg.jl:155
jl_apply at c:\julia_nightly\build\julia-master\src\julia.h:983
jl_trampoline at c:\julia_nightly\build\julia-master\src\builtins.c:826
jl_apply at c:\julia_nightly\build\julia-master\src\julia.h:983
do_call at c:\julia_nightly\build\julia-master\src\interpreter.c:70
eval at c:\julia_nightly\build\julia-master\src\interpreter.c:210
jl_interpret_toplevel_expr at c:\julia_nightly\build\julia-master\src\interpreter.c:26
jl_toplevel_eval_flex at c:\julia_nightly\build\julia-master\src\toplevel.c:498
jl_eval_module_expr at c:\julia_nightly\build\julia-master\src\toplevel.c:148
jl_toplevel_eval_flex at c:\julia_nightly\build\julia-master\src\toplevel.c:390
jl_parse_eval_all at c:\julia_nightly\build\julia-master\src\toplevel.c:546
jl_load at c:\julia_nightly\build\julia-master\src\toplevel.c:585
exec_program at c:\julia_nightly\build\julia-master\ui\repl.c:272
true_main at c:\julia_nightly\build\julia-master\ui\repl.c:328
julia_trampoline at c:\julia_nightly\build\julia-master\src\init.c:1030
wmain at c:\julia_nightly\build\julia-master\ui\repl.c:389
__tmainCRTStartup at f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crtexe.c:552
BaseThreadInitThunk at C:\Windows\system32\kernel32.dll (unknown line)
RtlUserThreadStart at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
RtlUserThreadStart at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
make[1]: *** [/c/julia_nightly/build/julia-master/usr/lib/julia/sys0.o] Error 1
make: *** [release] Error 2

[tkelman]

Hey, not bad, you got pretty far then. For reference, which commit are you building? Presumably with some local patches on top I'm guessing? Master just got some new inline assembly that'll have to be cleaned up for MSVC compatibility eventually, but I'm more concerned with just getting it to pass tests with mingw64 right now.

Anyway, if you build with VERBOSE=1 you should see the build command that's generating the system image there. Can you try running that same build command in a debugger, or attaching one to the julia process before it crashes?

[vtjnash]

I think that's symptomatic of trying to use one of the non-existent compiler intrinsics, eg for i128 operations

[tkelman]

Entirely possible. I think the last time I tried with Visual Studio 2012 there were problems with infinity and nan, but those are probably fixable with smallish patches instead of switching to a newer compiler version like I did.

dirname also sounds familiar, I think it needed to be tweaked in a compiler-specific way.

[mossr]

@tkelman I'm currently on master b5f94b6. And yes, I have a few patches in my working directory (the infinity/nan definitions being one of them).

I'm spending today debugging this crash. @vtjnash is the missing compiler intrinsic associated with LLVM?

[vtjnash]

Yes, it's expecting the compiler-rt library. I think tkelman was working on building that?

[tkelman]

Last time I looked (pretty sure I tried both 3.3 and 3.5.0), the cmake+msvc build system wasn't fully hooked up for the builtins part of compiler-rt. You might check LLVM trunk to see if anything's any different now (but breakage for other API reasons is really common there, so beware).

[vtjnash]

I updated the build script (to copy pcre_h.jl from the nightly) and pushed another commit that actually gets the sysimg to finish building. Making sys.dll with MSVC won't work right now, since sys.o doesn't have COMDAT stuff that the MSVC linker wants.

@tkelman fwiw, i think I've successfully enabled the COMDAT stuff that the MSVC link.exe needed, in bfa51bb (for llvm3.5+ / MCJIT)

[tkelman]

i think I've successfully enabled the COMDAT stuff that the MSVC link.exe needed

Neat. I'll try to play with that once the LLVM upgrade is done. I feel really iffy about trying to mix MSVC link.exe with the usual MinGW toolchain - there's going to be nasty path-shadowing with the posix link in either Cygwin or MSYS2 and I don't want to use the newer MSVC runtime while dependencies are using the MinGW one. And it's unusable in cross-compilation.

For an MSVC build of Julia itself, we should perhaps open a new issue to bring back partial support for it. You mentioned doing something with longjmp instead of the unsupported inline assembly that's now on master, but I didn't understand what you meant.

[vtjnash]

the linking stage (ld/lld/link.exe) is the one stage where it really doesn't matter if you're mixing tools. the linker doesn't know or care anything about the platform ABI. all it needs to know how to do is dependency resolution and the structure of the binary output. these things are the same, whether you use binutils or msvc. however, link.exe has the additional ability of converting CodeView information (which is documented, and generated by llvm -- see ldc-developers/ldc#167 for example) to the proprietary pdb format (which does not seem to be documented, although, of course, seems to have been reverse-engineered to a large extent) that is cross-compatible with msvc code and windows debuggers.

the assembly i wrote is essentially an optimized version of jl_longjmp. thus, you could get the same result by creating a jmpbuf, manually setting the registers that matter (RBP=0 and RSP=jl_stackbase), and then calling jl_longjmp

[tkelman]

As we saw over in #9376 (comment), we do need a static lib of the stack-checker function, and I'm pretty sure link.exe won't understand MinGW's libgcc.a for a MinGW-built Julia. Not sure where it would come from in an MSVC-built Julia. I can give it a try when the rest of the LLVM issues are sorted.

[vtjnash]

msvc distributes chkstk.obj. it's not particularly difficult to tell llvm to target it instead of libgcc.a. however, libgcc.a should be usable with link.exe (http://stackoverflow.com/questions/2096519/from-mingw-static-library-a-to-visual-studio-static-library-lib)

[tkelman]

@mossr you may want to dust off your MSVC 2010 patches while we're still on LLVM 3.3, in #11776 I now have a working repl with 2013 but as long as we're using a pre-C++11 version of LLVM then supporting a few slightly older versions might be tractable.