RFC: Windows CI

[tkelman]

Travis can't run on Windows yet, and the MinGW cross-compilers in the Ubuntu repository are a few versions old. AppVeyor recently launched a beta for a Windows-based CI system very similar in functionality to Travis. It's pretty straightforward to get Cygwin and the necessary MinGW compiler packages installed from the appveyor.yml file. MSYS2 could be tried here instead of Cygwin, I don't know whether there's a non-interactive version of the MSYS2 installer.

The difficult part is getting a Windows build of Julia to run in a reasonable amount of time. The script cygwin_build.sh downloads binaries for as many dependencies as I could find. A decent number have mingw64 rpm's available, but for the largest two (LLVM and OpenBlas) I had to dig around a little. Any volunteers want to try putting together a mingw64-llvm or mingw64-openblas (or suitesparse, arpack, etc) rpm, either in Fedora or the OpenSuse build service?

The initial time limit for free plans in AppVeyor is only 10 minutes per build, and it takes almost half that time just to install Cygwin and packages, and download dependency binaries. If you hit the time limit a few times it looks like they bump you up to 30 minutes. Here is the build result in the current state, some errors are happening related to version_git at the end, and key not found: "HOMEDRIVE" whatever that means. Any suggestions?

Edit: they increased the time limit to 30 minutes for everyone, I just happened to notice it immediately as they made that change.

[ivarne]

It looks like git describe is failing in https://github.com/JuliaLang/julia/blob/master/base/version_git.sh#L25. The quick fix is to build with NO_GIT=1

[tkelman]

Tried that, see https://ci-beta.appveyor.com/project/tkelman/julia/build/1.0.96 - still "key not found: HOMEDRIVE", but a less-threatening warning about boilerplate git version info rather than "fatal".

[ivarne]

I did not expect it to to fix the main problem, but I thought it was nice to not have useless errors. The version_git.sh file there to create the GIT_VERSION_INFO struct that store information from the git repository. That information is used in the REPL banner, and the versioninfo() function. Nothing that you really need for CI testing.

This might be a error that should be fixed anyway. The problem is that I do not have a Windows computer.

[tkelman]

Sure, though that git info would be useful if you do both CI and automatic deployment of successful builds with the same setup. It doesn't seem as though the Windows binaries are automated right now, as it can be several weeks between updates.

It's something having to do with the CI environment. The version of Git they have installed isn't from Cygwin, so it has some configuration that I'm not familiar with. I tried installing Cygwin's git, but that was giving errors about not finding some shared libraries.

When I run my cygwin_build.sh script in Cygwin locally I don't see anything about HOMEDRIVE, but I am getting a barrage of access violations. It may be that the binary version of LLVM or OpenBlas that I'm using has some problems. Compiling both from source avoids the access violations but is much too time-consuming for CI purposes.

This same script and downloaded binaries might work for a Linux-to-Windows cross compile too, if you can get recent-enough MinGW compilers and Wine?

[ivarne]

There is also #5629 where I use libgit2 to create the version_info.jl file. It is not finished, because the it took longer to execute than the normal git version on my machine. We also have a sourcedist make target that creates a tarball that can be built without git and internett access, but that does not solve your problem with binary dependencies.

[ihnorton]

Wow, it's awesome when something magically shows up the next day!

I don't know whether there's a non-interactive version of the MSYS2 installer.

Yes, the download here is a tgz.

want to try putting together a mingw64-llvm or mingw64-openblas

this would be great.

The initial time limit for free plans in AppVeyor is only 10 minutes per build

seems rather limiting...

Thanks for taking this on!

(we should have Windows nightlies going again tonight or tomorrow, but I don't think it's feasible to do CI on that build system).

[mlubin]

Very cool. How difficult would it be to set up windows CI for packages as well? Can this be done now using the 0.2 binaries?

[tkelman]

Yes, the download here is a tgz.

Good to know. Worth looking at then, though the recent guidance by @cmundi in #6019 concerns me because of the need to restart MSYS2 multiple times.

(we should have Windows nightlies going again tonight or tomorrow, but I don't think it's feasible to do CI on that build system).

Is that cross-compiling in Jenkins or a cron job somewhere?

Had a thought this morning that maybe the most-recent binary Julia download is the best place to get Julia's dependencies, assuming they change more slowly than Julia itself. Then an MSVC build might also be feasible, though I have very little experience using MSVC with a no-bash, no-gmake, non-autotools build system. There's a src/Windows.mk file that @vharavy and @tknopp have worked on that might work with nmake?

How difficult would it be to set up windows CI for packages as well? Can this be done now using the 0.2 binaries?

Probably, worth a try. Wouldn't need to bother with Cygwin/MinGW if you're not compiling any Fortran dependencies. Have a guinea-pig package with good tests you'd want to play with first?

[aviks]

It'd be great to have windows tests for packages

Have a guinea-pig package with good tests you'd want to play with first?

https://github.com/JuliaLang/JSON.jl would be good candidate. No dependencies, native or julia. and decent test coverage.

[cmundi]

@tkelman - The need to restart MSYS[2] shell can be eliminated. Here's what I think it takes to support continuous integration:

1. Maintain a clear distinction between the build environment and the build. The shell restarts are all part of setting up the environement.
2. Install MSYS2 as part of the build environment; this includes killing and restarting the shell after pacman -Su. Note I do mean kill, because exit is unreliable if the update changes msys*<version>*.dll
3. Within the actual build of Julia, we need to modify the OpenBLAS build to not attempt to use Window's lib.exe. This might be done in any number of ways, some quite hackish, but it would be nice to do a way likely to be accepted upstream.

[tkelman]

Thanks for the input @cmundi.

Install MSYS2 as part of the build environment

Do you mean pre-installed on AppVeyor's VM's? I don't think this is feasible, since they're primarily targeting .NET developers rather than cross-platform projects. If I knew more about NuGet or Chocolatey then maybe MSYS2 could be set up via those package managers in a ready-to-go way? Could contact AppVeyor and make a feature request to support MSYS2, but I suspect they'd respond by saying to make a package for it.

Within the actual build of Julia, we need to modify the OpenBLAS build to not attempt to use Window's lib.exe. This might be done in any number of ways, some quite hackish, but it would be nice to do a way likely to be accepted upstream.

This won't show up if you're using a binary openblas. I have yet to see this one in my Cygwin-to-MinGW cross compiles even with openblas from source, otherwise I could help find where in the openblas build it happens. Is there more info on this in a separate issue, either here or on openblas' tracker?

[cmundi]

AppVeyor is definitely .NET centtic. That's not a bad thing; I'm just not
sure I'd use it for this use case.

I have not had time to chase down the exact location of the OpenBLAS build
problem. When I do I will report upstream.
On Mar 3, 2014 12:32 PM, "Tony Kelman" notifications@github.com wrote:

Thanks for the input @cmundi https://github.com/cmundi.

Install MSYS2 as part of the build environment

Do you mean pre-installed on AppVeyor's VM's? I don't think this is
feasible, since they're primarily targeting .NET developers rather than
cross-platform projects. If I knew more about NuGet or Chocolatey then
maybe MSYS2 could be set up via those package managers in a ready-to-go
way? Could contact AppVeyor and make a feature request though?

Within the actual build of Julia, we need to modify the OpenBLAS build to
not attempt to use Window's lib.exe. This might be done in any number of
ways, some quite hackish, but it would be nice to do a way likely to be
accepted upstream.

This won't show up if you're using a binary openblas. I have yet to see
this one in my Cygwin-to-MinGW cross compiles even with openblas from
source, otherwise I could help find where in the openblas build it happens.
Is there more info on this in a separate issue, either here or on openblas'
tracker?

Reply to this email directly or view it on GitHubhttps://github.com//pull/6028#issuecomment-36549177
.

[tkelman]

AppVeyor is definitely .NET centtic. That's not a bad thing; I'm just not sure I'd use it for this use case.

What else is there? Wercker had a beta on Windows that actually started up inside Cygwin, but they closed it down for now. Jenkins or TeamCity take a lot of work to get set up and need dedicated hardware (AFAIK).

Running a cross-compile build + Wine in the Travis matrix should also work, but it might miss some things that only happen on an actual Windows box? Dunno.

[cmundi]

All of the CI tools I've see require significant effort. Some, like
Jenkins, can take as much effort to maintain as to set up.

One thought which might be helpful or not is separating testing from
building. Building could be done as a cross-compile on whatever Un*x is
convenient, with a remote shell-out to a Windows test farm. The test farm
could be Windows VMs hosted anyplace. No cygwin or mingw needed for CI. Or
am I missing something?
On Mar 3, 2014 4:41 PM, "Tony Kelman" notifications@github.com wrote:

AppVeyor is definitely .NET centtic. That's not a bad thing; I'm just not
sure I'd use it for this use case.

What else is there? Wercker had a beta on Windowshttp://blog.wercker.com/2013/10/25/introducing_wercker-labs_windows.htmlthat actually booted into Cygwin, but
they closed it down for nowhttp://wercker.uservoice.com/forums/181077-general/suggestions/5398537-create-official-windows-box-for-net.
Jenkins or TeamCity take a lot of work to get set up and need dedicated
hardware (AFAIK).

Reply to this email directly or view it on GitHubhttps://github.com//pull/6028#issuecomment-36575375
.

[tkelman]

One thought which might be helpful or not is separating testing from
building. Building could be done as a cross-compile on whatever Un*x is
convenient, with a remote shell-out to a Windows test farm. The test farm
could be Windows VMs hosted anyplace. No cygwin or mingw needed for CI.

Sounds complicated. I personally like the "CI as blueprint for dependencies and how to build" philosophy, but maybe others differ.

https://github.com/JuliaLang/JSON.jl would be good candidate. No dependencies, native or julia. and decent test coverage.

Thanks for the suggestion. Here's what I put together as a package-level appveyor.yml. I ran into ERROR: key not found: "HOMEDRIVE" here too, turns out the CI environment doesn't set HOMEDRIVE or HOMEPATH environment variables (ref. #6042). And you need to have administrator privileges to make a symlink in Windows. Not sure how to run a command as admin in AppVeyor, but changing ln -s to cp -r worked: https://ci-beta.appveyor.com/project/tkelman/json-jl/build/1.0.14

[tkelman]

The build for Julia itself is looking much better, now using the most recent Julia binary download as the source for almost all of the dependencies. It's getting to the tests, failing due to #6016. https://ci-beta.appveyor.com/project/tkelman/julia/build/1.0.130

[aviks]

Thanks @tkelman, that's brilliant. (Although the tgbkchaag5.cmd error looks scary!)

I'll try to enable this in the JSON.jl repo. I don't think many Julia packages use OS specific functionality, but at the very least, testing a broad set of packages will provide a decent smoke test of the Julia windows build.

Should we try to set up a chocolatey/NuGet package for julia nightlies and releases, like we do for ubuntu? Is that easy to do? Might make it easier to write and maintain the appveyor scripts.

[aviks]

Trying to run a slightly more complex package on AppVeyor seems to produce a crash inside LLVM : Instruction does not dominate all uses!.

https://ci-beta.appveyor.com/project/aviks/ito-jl/build/1.0.4

[tkelman]

(Although the tgbkchaag5.cmd error looks scary!)

I think that's just PowerShell being weird. Lots of *nix commands (and Pkg in Julia) tend to print informational messages to stderr, but still return success. PowerShell seems to catch everything that goes to stderr as a NativeCommandError. Redirecting stderr to stdout 2>&1 helps for things that run in bash, not sure if that translates into Windows commands.

at the very least, testing a broad set of packages will provide a decent smoke test of the Julia windows build.

Agreed.

Should we try to set up a chocolatey/NuGet package for julia nightlies and releases, like we do for ubuntu?

Either that or a mingw64 rpm. Chocolatey or NuGet wouldn't (AFAIK) help people who are cross-compiling from Linux, an rpm would.

Is that easy to do?

No idea, haven't tried it.

Trying to run a slightly more complex package on AppVeyor seems to produce a crash inside LLVM : Instruction does not dominate all uses!

Don't know enough about LLVM to have any idea on that one, can you replicate it outside of AppVeyor? It might be a case of LLVM trying to use an instruction that AppVeyor's VM can't handle.

[tkelman]

Looks like 5809f6f resolved the test failure from #6016, but now it's failing in the numbers tests, most likely due to #3081. There's a patch for this problem at deps/win64-int128.llvm-3.3.patch that Julia has to apply to LLVM 3.3 on win64, and the downloaded version I found doesn't have that patch applied. I might have to go searching for a different LLVM binary to use.

[aviks]

can you replicate it outside of AppVeyor?

Nope, the tests run successfully on Windows 7 with the latest Julia 0.3 binary snapshot.

[vtjnash]

@tkelman that patch doesn't exist outside of Julia's codebase. you would need to compile your own (adding encouragement to get this merged on upstream issue might also help)

[tkelman]

@vtjnash Right, I can see that. The upstream issue is http://llvm.org/bugs/show_bug.cgi?id=16168? Was the patch submitted via Phabricator as suggested there? I attempted a build using LLVM 3.4 but ran into build errors that look like API changes https://gist.github.com/tkelman/9405781, and I'm building with 3.5svn right now.

I also tried a 32-bit build here https://ci-beta.appveyor.com/project/tkelman/julia/build/1.0.176 but ran into one of the linear algebra tolerance issues addressed in #5902.

[vtjnash]

You'll lose progressively more Windows features if you try upgrading to a newer version of LLVM.

Phabricator link: http://llvm-reviews.chandlerc.com/D1998

[cmundi]

@vtjnash : Are you saying Julia is locked into an old release/fork of llvm? If so, is this just a temporary fact while the core devs focus on building out the language? Or is this something which requires a fix in llvm? kind of looks that way...

(please pardon my naivety, I've just started studying the pipeline for julia and have only ever dabbled with llvm before)

[vtjnash]

Only on Windows, where we have tended to only support one version of LLVM at a time due to the complexity of the patches required for error handling and i128 support.

[cmundi]

Thanks for your explainng about the LLVM patches. Everything is harder on
Windows. :)
On Mar 7, 2014 8:08 AM, "Jameson Nash" notifications@github.com wrote:

Only on Windows, where we have tended to only support one version of LLVM
at a time due to the complexity of the patches required for error handling
and i128 support.

Reply to this email directly or view it on GitHubhttps://github.com//pull/6028#issuecomment-37026646
.

[tkelman]

I clearly have no idea what I'm doing when it comes to rebasing...

[tkelman]

Once you've signed up you go to https://ci.appveyor.com/projects/new, hover over JuliaLang/julia and click add. Then we merge this and see what happens.

[ViralBShah]

@StefanKarpinski or @alanedelman will have to do the signup and put the credit card details, since I do not have them. Hopefully will happen soon, or ping by email or on this thread.

[FeodorFitsner]

You don't need credit card to do Pro trial.

-Feodor

On Sun, Nov 23, 2014 at 2:16 PM, Viral B. Shah notifications@github.com
wrote:

@StefanKarpinski or @alanedelman will have to do the signup and put the credit card details, since I do not have them. Hopefully soon, or ping by email or on this thread.

Reply to this email directly or view it on GitHub:
#6028 (comment)

[tkelman]

But we will need a credit card to upgrade from the trial, the only part we still need to evaluate is setting it up on a more permanent basis and seeing how well it works when running on every commit and pull request.

It should probably be Stefan not Alan to sign up, since it looks like Alan doesn't even have commit credentials here.

[tkelman]

@ViralBShah I added all the necessary hooks here and on all other JuliaLang packages for which I might at some point want to turn on AppVeyor, so you may as well remove my admin privileges before I misclick something. Unless @StefanKarpinski can get to activating this soon since there's an off chance I may need to adjust something once this is turned on.

[StefanKarpinski]

What do I need to do to get this paid for?

[tkelman]

Once you've signed up via https://ci.appveyor.com/signup for a Pro trial, log in either via github auth or create a new appveyor-only account, then under your name in the upper right click Plan. @FeodorFitsner how do we request the open-source discount?

I think you can add me as a collaborator to your appveyor account so I can modify the settings, under Team. Feodor can you confirm whether I should be added as User or Collaborator, in order to be able to modify CI settings but not access any billing info?

[FeodorFitsner]

Sure, let me guys configure this plan for you. So, it's Pro with 50% off, right?

[tkelman]

Yes, Pro with open-source. No need for private projects but we do want the high-performance build environment.

[FeodorFitsner]

Right. While logged into AppVeyor use this link (https://ci.appveyor.com/plan/order/oss_pro_monthly) to purchase "Pro FOSS" plan. It's on dedicated Hyper-V.

[StefanKarpinski]

Ok, I did that for my AppVeyor account linked to my GitHub account. There's a bunch of options under https://ci.appveyor.com/team and unclear which I should choose. There's also the tantalizing "GitHub teams" option, which seems like it might make it easy to give all JuliaLang contributors the ability to mess with our AppVeyor setup, which would be quite handy.

[StefanKarpinski]

I added the JuliaLang/Contributors team so maybe that gives people who can push to Julia access.

[tkelman]

Yay! It sounds like we're in business then, I see it going at https://ci.appveyor.com/project/stefankarpinski/julia/history

I do not see the ability to adjust the settings though, so that part didn't quite work yet.

[StefanKarpinski]

I added you as an Administrator using your kelman.net email address.

[tkelman]

Ah, there it is, I just need to log out then log back in and select which account to log in under. Slightly odd system.

[vtjnash]

Yay! Thanks everyone for making this happen.

[tkelman]

Let's see how long the queue gets. We could turn it off for PR's and only run it on master if the wait gets too long. What would be nifty is an @AppVeyorBot we could ping on pr's to selectively run only on the pr's where it makes sense to.

[nalimilan]

Pretty cool!

[timholy]

This is great!

[tkelman]

cc @StefanKarpinski @FeodorFitsner I'm seeing a warning banner on AppVeyor today:

Trial period expires in 4 days. Select plan to keep running your builds. After trial period is over only projects with public repositories will run.
If you have any questions or need to extend your trial feel free to contact us.

Can we make sure over the next couple days that the subscription is set up properly and we stay on the fast build environment? Assuming people are happy enough to continue with AppVeyor, I think despite the occasional timeout it's definitely been worth having enabled.

[FeodorFitsner]

Just in case I've sent Stefan the link to re-subscribe to the same plan. It's holidays right now, so if you need more days to sort it out let me know.

Merry Christmas and Happy New Year! :)

[tkelman]

Merry Christmas and Happy New Year! :)

Thanks so much Feodor, you too!

[ViralBShah]

I would like to say that despite the issues we have seen, appveyor is great to have and we will be certainly staying with it.

[vtjnash]

That's an odd way to say it, since "I would like to say" usually ends with a "but". However, I agree that AppVeyor has been an extremely crucial for improving the quality of the Windows side of Julia, even just in the short time that it's been active.

[StefanKarpinski]

I'll deal with this shortly – we're in the middle of switching bank accounts.

Does it make sense to switch to a JuliaLang account on AppVeyor? It feels a little weird to have everyone using my personal account. Not that I mind, it's just confusing.

[tkelman]

Does it make sense to switch to a JuliaLang account on AppVeyor?

If AppVeyor supports organization-named accounts, that would make sense to me.