llvm sources keep breaking

[jfirebaugh]

TLDR: Does anyone have a working build on Travis CI using clang >= 3.5?

As far as I can tell, the whitelisted llvm sources stopped working a couple days ago, when llvm released a new version.

I know this may be out of Travis's control, but bear with me: I'd like to use this ticket reach out to other folks and brainstorm solutions. I'm CC'ing some people who have opened or commented on related tickets because they may be facing similar issues right now: @JesseTG, @ldionne, @gnzlbg, @bkentel, @prudhomm.

@gnzlbg summed this up well in travis-ci/apt-package-safelist#561 (comment):

Every time LLVM makes a new release branch the same thing happens, forcing us to disable TravisCI for 1-2 months. This happens twice a year so that's 2-4 months without travis, every year. It is the third time this happens so I'm going to explain the problem with the hopes that somebody at travis cares enough about C++ to fix this forever (or at least ensure that it happens as pain free as possible next time, for this time we are already 27 days without travis).

In a nutshell, LLVM releases twice a year.

1-2 months before a release, a release branch for the new release version is forked, and the master branch version number is bumped.

For example: in July the master branch of LLVM had version 3.7. Since 3.7 is going to be released somewhere in August, a branch for llvm 3.7 was made, and the master branch version was bumped to 3.8.

This means that the Ubuntu Precise build bots for the master branch switch from building LLVM 3.7 to building LLVM 3.8 breaking all travis scripts that work with master. Since LLVM 3.7 has not been released yet there are no buildbots for ubuntu precise that build 3.7, so users cannot move to 3.7 release yet. Since 3.8 is not white listed, users cannot move to 3.8 dev (it's been one month since this PR was sent...).

That basically leaves going back to llvm 3.6 as the only option, which is very bad since LLVM encourages those using it to follow master as close as possible.

This will happen again twice in 2016, in both January and July.

The only solution I can see is actually pretty simple: white list the LLVM dev versions as soon as they fork a release branch. It is a very small PR (like this one). Otherwise you are forcing all of us that follow LLVM dev per LLVM recommendations to disable travis-ci testing, for 27 in a row already and counting...

There have been 5 closed PRs to fix this so people do care about it.

A release just happened, and just as predicted, all of our builds broke. We were using clang 3.5; I haven't been able to get an upgrade to 3.6 working. I posted to the llvm-dev mailing list asking what's up:

Hi,

At Mapbox we have been using the llvm-toolchain-precise-3.5 package
source to install clang-3.5 on Travis CI VMs for testing purposes
(Travis VM images are Ubuntu 12.04). This has worked fairly well for
us; thanks for providing it. However, the 3.5 source seems to have
recently disappeared, breaking our builds. No problem, we thought,
we'll update to 3.6. However, this doesn't seem to work either. When
using either the llvm-toolchain-precise or llvm-toolchain-precise-3.6
sources, no clang-3.6 package can be found:

$ sudo -E apt-add-repository -y "deb http://llvm.org/apt/precise/
llvm-toolchain-precise main"
Installing APT Packages (BETA)
$ export DEBIAN_FRONTEND=noninteractive
$ sudo -E apt-get -yq update &>> ~/apt-get-update.log
$ sudo -E apt-get -yq --no-install-suggests --no-install-recommends
--force-yes install gdb clang-3.6 libstdc++-4.9-dev libstdc++6
libllvm3.4 xutils-dev libxxf86vm-dev x11proto-xf86vidmode-dev
mesa-utils
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package clang-3.6

Full output here: https://travis-ci.org/mapbox/mapbox-gl-native/jobs/81888110
Similar output when using the llvm-toolchain-precise-3.6 source here:
https://travis-ci.org/mapbox/mapbox-gl-native/jobs/81865117

Are there packages missing from these sources? What version is the
unsuffixed llvm-toolchain-precise source expected to contain? And for
the future avoidance of unexpected build breakage, is there a setup we
can use where packages won't disappear from underneath us?

Thanks!

So far no response.

Anyone know what's up here?

[jfirebaugh]

I just discovered that my issues seem to be entirely due to llvm.org taking extremely long to connect (30+ seconds), such that apt-get update just skips the sources:

W: Failed to fetch http://llvm.org/apt/precise/dists/llvm-toolchain-precise-3.6/Release.gpg  Connection failed
W: Failed to fetch http://llvm.org/apt/precise/dists/llvm-toolchain-precise-3.6/Release  
W: Failed to fetch http://llvm.org/apt/precise/dists/llvm-toolchain-precise-3.6/main/binary-amd64/Packages  Unable to connect to llvm.org:http:
W: Failed to fetch http://llvm.org/apt/precise/dists/llvm-toolchain-precise-3.6/main/binary-i386/Packages  Unable to connect to llvm.org:http:

@travis-ci: Could you please not redirect this output to ~/apt-get-update.log? Seeing it in the build log would have saved me a lot of time.

Original question still applies: how do we get a recent version of clang on Travis VMs if llvm.org continues to time out?

[dirk]

I ran into this problem last night as well. Thank you for doing some investigating and getting the conversation started, @jfirebaugh!

I'm wondering if there's a flag that can be passed to apt-get update to get it to exit with a non-zero status if it fails to fetch from one of the sources. I scanned over the manpage for apt-get but couldn't find anything to that effect. 😕

Given that having sources be up-to-date is necessary for many builds, I'd say that failing to fetch sources should fail the build. What do you think? Should build-failure-on-source-fetch-failure be a default and/or an optional flag?

[jfirebaugh]

llvm.org is down again today, breaking builds. Still looking for a solution here if anyone has one.

[dirk]

I believe Travis mirrors/caches some APT stuff already (see here). Would it be unduly difficult to get llvm.org's APT source somehow included in that?

[jfirebaugh]

@dirk -- APT caching "is available only for private repositories."

[jfirebaugh]

Someone deleted the entire http://llvm.org/apt/ subdirectory today, again breaking all Travis builds that use the whitelisted packages.

[jfirebaugh]

It looks like llvm.org is no longer going to host packages: http://lists.llvm.org/pipermail/llvm-dev/2016-May/100303.html

[gyorb]

Is there any alternative apt source for llvm/clang or building from source is the only solution?

[gyorb]

Until there is no better solution, I found an llvm-backport ppa created by Alexander Drozdov to download clang 3.7 for trusty. Here is our modified .travis.yml if you are interested.

[Kicer86]

what about mirroring useful repos by travis itself?

[sylvestre]

This is now a fixed issue I think