Travis CI

[bjourne]

This site offers free continuous integration testing for projects hosted on github: https://travis-ci.org/ It would be pretty cool to setup Factor to use it as a supplement to our own build bots.

[mrjbq7]

Good idea. I use it for another project, maybe I can queue that up for later.

[jonenst]

I'm having the tests run at https://travis-ci.org/jonenst/factor
You can see the trial and error writing the .travis.yml but it turned out ok !

[jonenst]

after 49 minutes, all jobs ended with
The job exceeded the maximum time limit for jobs, and has been terminated.

[bjourne]

👍 Perhaps try running the tests without load-all? Then only the vocabs compiled into the image are tested.

[mrjbq7]

The compiler deploy tests are some of the worst offenders, maybe exclude those.

We could make a fast-tests-only? flag that slips a few long ones for Travis?

[jonenst]

https://travis-ci.org/jonenst/factor/builds/188540475
Still not passing all tests.

I basically removed generate-help and do-benchmarks (20 minutes each according to http://builds.factorcode.org/dashboard ) and split the tests in 3 so that they would last about 20 minutes. So we spend 20 minutes bootstrapping/loading-all, then 20 other minutes testing. That's a bit wasteful but I wanted a proof of concept. Splitting more would be even more wasteful...

So now there are many questions..

• How would we want to use travis ? Replace the current build bots ? Maintain 2 test environments ?
• How do we do dashboards and reports ?
• How much do we test ? What do we test ? How do we optimize the tests for travis ?

[mrjbq7]

Can we pay for more CPU time to run all tests?

[mrjbq7]

Or maybe we make the tests take less time. I'll look at that a little today.

[jonenst]

• Travis CI tests exceed 50 minute ytdl-org/youtube-dl#5289 says the build is 120 minutes for containers. Using them raises questions that need to be solved (moving the deps from build.sh to hardcoded in .travis.yml, using ubuntu12.04 on which the standard gcc is too old. Also what happens to macos).
• https://travis-ci.com/plans and https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts says the timeout is 120 minutes for paid plans.
• Support configurable job timeouts travis-ci/travis-ci#2736 says they have options if we contact them. Maybe that's longer than 120 minutes ?

But I think we should focus on running more intelligent (ie shorter) tests ?

[mrjbq7]

As a rough guide to long tests, this is the percentage of time that test-all takes (excluding some stuff like postgres and memcached that I'm not running locally so those tests just fail).

You can read this that tools.deploy takes 70% of the overall test time, and math.vectors.simd takes 3.4%, etc.

    { "tools.deploy" 70.208579206439 }
    { "math.vectors.simd" 3.432642933546964 }
    { "graphviz" 3.019179750454758 }
    { "classes.tuple" 2.560411919890502 }
    { "db.postgresql" 1.959483386449348 }
    { "compiler" 1.873638688005764 }
    { "regexp" 1.234512415763741 }
    { "memory" 0.9378004941618001 }
    { "tools.profiler.sampling" 0.8370876228996605 }
    { "timers" 0.8250592030795237 }
    { "specialized-arrays" 0.8240954700942372 }
    { "hashcash" 0.5640703138775561 }
    { "io.launcher.unix" 0.5089367087099408 }
    { "tools.disassembler" 0.4870729506429094 }
    { "cuckoo-filters" 0.4781854242262294 }
    { "classes.struct" 0.4492532768623572 }
    { "parser" 0.3936029020888083 }
    { "persistent.vectors" 0.3392758395439747 }
    { "delegate" 0.3388011978388816 }
    { "peg.ebnf" 0.2985104585520624 }
    { "concurrency.locks" 0.2734425398671684 }
    { "curses" 0.2681655444981742 }
    { "io.files" 0.2246254839028875 }
    ...

[bjourne]

Cool! Looks like excluding tools.deploy would be reasonable?

[jonenst]

https://travis-ci.org/jonenst/factor/builds/189845398

Got it to almost replicate the build bots results. Still need to make the imap and forestdb tests work...

[bpollack]

If our motivation is just that we wanted to get off relying on Mason, would looking at drone.io make sense here? Very similar idea to Travis CI, but they (currently?) are way faster, and can be self-hosted if we wanted to hybridize what we were up to. I'm only asking because this seems to have been a lot more effort than I think we were anticipating; I'm not otherwise interested in bikeshed colors.

[mrjbq7]

This is pretty awesome, we might need to make some changes because of how we use mason to update our build dashboard, email reports, benchmark, and upload new images...

[jonenst]

https://travis-ci.org/jonenst/factor/jobs/190092435

It passed for linux. It's running load-all test-all but not help generation and benchmarks.
I think it should be possible to update the dashboard at http://builds.factorcode.org/dashboard because the child sends things over http (some things wouldn't work, like the heartbeat).

Do you want to take it from here ? Or I can run my own mason instance and test it there.

[mrjbq7]

Any thoughts on getting Windows to work?

[mrjbq7]

Also, of having both 32-bit and 64-bit builds?

[jonenst]

• a quick google search suggests https://www.appveyor.com/ for windows
• travis ci doesn't provide 32bits VMs. I don't know how much work it is to test a 32bit factor on a 64bit machine..

[mrjbq7]

It's pretty easy to build 32-bit on 64-bit, I do that on OS X all the time:

$ ./build.sh update macosx-x86-32

Note: that requires universal 32/64 builds of whatever library dependencies exist.

Also, linux is a bit different.

Ahh, I didn't realize Travis didn't have a Windows option.

[bjourne]

Pretty, pretty good. So as a next step we can make it so the Travis CI machine reports its build status to http://builds.factorcode.org/dashboard? There is a feature in mason for slaves to send their build status and logs to the master but I don't know how it works.

Linux is totally horrible for cross-running binaries. You can easily built a 32bit Factor vm, but then you also want it to dlopen 32bit libraries like sqlite.so and so on. Some libraries can be made to work by setting LD_LIBRARY_PATH to point to the directories containing the 32bit builds of them, but getting it to work really well I don't think is possible.

[jonenst]

Pretty, pretty good. So as a next step we can make it so the Travis CI machine reports its build status to http://builds.factorcode.org/dashboard? There is a feature in mason for slaves to send their build status and logs to the master but I don't know how it works.

Yes, that's what I was saying when I said "I think it should be possible to update the dashboard at http://builds.factorcode.org/dashboard because the child sends things over http [...] Do you want to take it from here ?"
because there is a shared secret for to be allowed to update the dashboard

[mrjbq7]

How do you make shared secrets available to Travis but not just anyone?

[jonenst]

You have multiple options : https://docs.travis-ci.com/user/environment-variables/ " if it does contain sensitive information, and might be different for different branches – encrypt it and add it to your .travis.yml if it does contain sensitive information, but is the same for all branches – add it to your Repository Settings " So for the gmail password of the test imap user I use repository settings. For the mason secret that would also be the correct thing to use. Le 11 janv. 2017 15:32, "John Benediktsson" <notifications@github.com> a écrit :

…

How do you make shared secrets available to Travis but not just anyone? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1760 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFceNZmfTmEjDYkMqb8s1q6i3kEa2o9ks5rROgZgaJpZM4LJnSJ> .

[jonenst]

Does the mac build test libgdk_pixbuf-2.0.dylib (resource:basis/images/loader/gtk/gtk-tests.factor ) and libcairo.dylib ( resource:basis/cairo/cairo-tests.factor ) ? Maybe we should add a way to get the full log from builds, eg http://builds.factorcode.org/report?os=macosx&cpu=x86.64.

Looking into getting the mac build to work on travis : https://s3.amazonaws.com/archive.travis-ci.org/jobs/191101478/log.txt

[erg]

We had a couple clean Linux clang/gcc builds! However, macOS times out and has never built clean. I disabled fftw tests since installing it takes a long time and we don't get anything out of the tests imo.

[bjourne]

Perhaps we should just let Travis compile and bootstrap? Then at least we could get it to display green. :) Imo, Travis can not replace our current build farm because it has no support for Windows and 32bit architectures and those are our mot problematic targets. And it doesn't seem like Travis will add support for those anytime soon. :/ I don't know what we can do here. Maintaining the build farm is a lot of work but without it, a lot of architectures that people rarely use would break.

[erg]

We could have it test the vocabs that matter most. Maybe core/, io, compiler, ui?

[bjourne]

I removed the testing in: c3f04e7 The idea is to just get Travis green. We can then incrementally return the testing code.

[erg]

Travis is green! Let's test some of core too?

[erg]

We're now testing all pull requests against all vocabs that changed from origin/master, and I just pushed a patch to test core-vocabs too.

To be thorough we should load-all help-lint-all, and zealot vocab can test several things at once if we figure out how to integrate it with TravisCI.

[erg]

Ok, this issue is done imo. We are testing vocabs that changed since last clean build or the diff of the PR and origin/master, help-lint-all, and all of core-vocabs tests. Closing.