Improve Travis CI testing speed

[kmod]

@dagar First of all, thanks for getting it working, it's definitely a huge help :)

I wonder if there's something we can do about the testing time though, which I assume comes mostly from having to compile LLVM. I don't think we can easily get rid of our source dependency on LLVM, but we end up building the same stuff every time so maybe there's some way to cache it? We could probably come up with some hash of {llvm_rev, patchset}, and map that to the final build products that we use. A simpler option could be to enable ccache for the Travis build and have Travis cache the .ccache directory.

What do you think? Not hugely important at this stage but could be nice.

[dagar]

I looked into this initially. I was worried about exceeding the travis-ci.org fair use. The current travis free tier doesn't support caching between builds. Their new docker based infrastructure does, but doesn't let you install packages or use your own containers yet.

The travis build could try to retrieve a prebuilt llvm+clang from something like s3 using the llvm rev + patchset hash. At the end of the build after testing it could upload to s3 if it built from source. CMake needs a few small changes to work with either a prebuilt llvm or from source.

Another option you may want to consider is providing a machine. I believe it's fairly easy to replicate the github + travis integration with jenkins. Builds would be significantly faster with ccache plus not having to install packages and full clone every time. The same jenkins build could also update speed.pyston.org.

[dagar]

Another plus for the bring your own machine idea is that we could easily extend the build matrix to include Release, Release+Asserts, Debug with both clang and gcc or different modes (GRWL, GIL, etc). We could even add builds that run under valgrind or with the sanitizers (asan, tsan, etc).

[marscher]

at least receiving the sources of llvm via a tarball or zip saves ~1.5 min despite cloning their whole (really huge) history ;)

[dagar]

@kmod - if you're interested in the bring your own hardware option I could take care of the setup and integration. You'd basically just need to provide a machine with ubuntu + jenkins + pyston packages and make the jenkins web interface publically accessible (probably github auth).

coreclr is using jenkins with github integration like travis.

[kmod]

I'm not very psyched about sysadmining another machine :) The 20 minute Travis-CI testing time is kind of annoying right now but isn't too high up on the priority list; I'm sure we'll set something like this up eventually but it might be a while before it makes sense to learn about jenkins and what not.

[kmod]

An interesting idea from @denji: we currently clone the whole llvm+clang repositories, but we only want a specific revision. We can use the --depth=1 argument to only fetch a single revision, but the issue is that our revision is currently specified as an SVN commit number, and we determine the git revision by looking at the commit logs.

The savings of only checking out a single revision are pretty huge (11x speedup, or a savings of about 2 minutes), but probably block on doing one of the other changes to how+where we get the llvm repositories (ex making them submodules).

[denji]

$ llvm (master) > git log --grep="llvm/trunk@230300" | head -1
commit aecbb87ee8036ba8d2732d5c9fcc2c83a8c8b8d7

[kmod]

Yes but you can't do that until you've checked out the repository :(

[dagar]

I think it depends how you want to handle being able to modify llvm going forward and how applying patches (git_svn_gotorev.py) should work. Or should all of that stay the same, but for the sake of travis speed just shallow clone and manually apply each patch?

[denji]

Of all DVCS systems, only supports the Veracity revision:hash at the same time that saddens me as user git :(

[marscher]

non-linear history makes me also sad sometimes :D Why do you need all revisions, if you know latest commit in git refers to the last svn commit? If the distance is farther, then depth clone a nice cutoff to catch it.

[kmod]

Well, we don't want the last revision, we want a specific revision of LLVM since the API changes pretty rapidly -- you can check out the commits that touch llvm_revision.txt to get a sense of how quickly the API moves on and what kinds of changes we need to make. We're currently specifying the revision as a SVN revision; I guess we could try specifying it as a git revision, or specify both the svn and git revisions.

[denji]

@kmod we don't mean trank (master), branch already made stable releases https://github.com/llvm-mirror/llvm/branches

[dagar]

Working on travis-ci using a prebuilt LLVM in PR #400.

[dagar]

I think with pyston's current usage of travis-ci the build is now about as fast as it's going to get.

After adding ccache and moving to travis-ci's new infrastructure a build is typically 10-15 minutes, and 50-75% of that time it spent running tests. The actual compile is 1-2 minutes and the rest of the time is spent installing deps and cloning llvm, clang, and submodules.
If they readd apt caching or allow custom containers we might be able to save another few minutes per build.

[dagar]

One idea to consider once travis-ci stops scaling is to split off the integration and extra tests into another repo and travis-ci instance. This could run nightly with the last good build (stored as a github release). This has the side benefit of getting the test/integration submodules out of the main repo which makes for a lengthy clone (and it's kind of unusual).