Travis CI #1760

Open
bjourne opened this Issue Dec 10, 2016 · 23 comments

Projects

None yet

4 participants

@bjourne
Member
bjourne commented Dec 10, 2016

This site offers free continuous integration testing for projects hosted on github: https://travis-ci.org/ It would be pretty cool to setup Factor to use it as a supplement to our own build bots.

@mrjbq7
Member
mrjbq7 commented Dec 10, 2016

Good idea. I use it for another project, maybe I can queue that up for later.

@jonenst
Contributor
jonenst commented Jan 3, 2017 edited

I'm having the tests run at https://travis-ci.org/jonenst/factor
You can see the trial and error writing the .travis.yml but it turned out ok !

@jonenst
Contributor
jonenst commented Jan 3, 2017

after 49 minutes, all jobs ended with
The job exceeded the maximum time limit for jobs, and has been terminated.

@bjourne
Member
bjourne commented Jan 3, 2017

👍 Perhaps try running the tests without load-all? Then only the vocabs compiled into the image are tested.

@mrjbq7
Member
mrjbq7 commented Jan 3, 2017

The compiler deploy tests are some of the worst offenders, maybe exclude those.

We could make a fast-tests-only? flag that slips a few long ones for Travis?

@jonenst
Contributor
jonenst commented Jan 3, 2017

https://travis-ci.org/jonenst/factor/builds/188540475
Still not passing all tests.

I basically removed generate-help and do-benchmarks (20 minutes each according to http://builds.factorcode.org/dashboard ) and split the tests in 3 so that they would last about 20 minutes. So we spend 20 minutes bootstrapping/loading-all, then 20 other minutes testing. That's a bit wasteful but I wanted a proof of concept. Splitting more would be even more wasteful...

So now there are many questions..

  • How would we want to use travis ? Replace the current build bots ? Maintain 2 test environments ?
  • How do we do dashboards and reports ?
  • How much do we test ? What do we test ? How do we optimize the tests for travis ?
@mrjbq7
Member
mrjbq7 commented Jan 3, 2017

Can we pay for more CPU time to run all tests?

@mrjbq7
Member
mrjbq7 commented Jan 3, 2017

Or maybe we make the tests take less time. I'll look at that a little today.

@jonenst
Contributor
jonenst commented Jan 3, 2017 edited

But I think we should focus on running more intelligent (ie shorter) tests ?

@mrjbq7
Member
mrjbq7 commented Jan 3, 2017

As a rough guide to long tests, this is the percentage of time that test-all takes (excluding some stuff like postgres and memcached that I'm not running locally so those tests just fail).

You can read this that tools.deploy takes 70% of the overall test time, and math.vectors.simd takes 3.4%, etc.

    { "tools.deploy" 70.208579206439 }
    { "math.vectors.simd" 3.432642933546964 }
    { "graphviz" 3.019179750454758 }
    { "classes.tuple" 2.560411919890502 }
    { "db.postgresql" 1.959483386449348 }
    { "compiler" 1.873638688005764 }
    { "regexp" 1.234512415763741 }
    { "memory" 0.9378004941618001 }
    { "tools.profiler.sampling" 0.8370876228996605 }
    { "timers" 0.8250592030795237 }
    { "specialized-arrays" 0.8240954700942372 }
    { "hashcash" 0.5640703138775561 }
    { "io.launcher.unix" 0.5089367087099408 }
    { "tools.disassembler" 0.4870729506429094 }
    { "cuckoo-filters" 0.4781854242262294 }
    { "classes.struct" 0.4492532768623572 }
    { "parser" 0.3936029020888083 }
    { "persistent.vectors" 0.3392758395439747 }
    { "delegate" 0.3388011978388816 }
    { "peg.ebnf" 0.2985104585520624 }
    { "concurrency.locks" 0.2734425398671684 }
    { "curses" 0.2681655444981742 }
    { "io.files" 0.2246254839028875 }
    ...
@bjourne
Member
bjourne commented Jan 4, 2017

Cool! Looks like excluding tools.deploy would be reasonable?

@jonenst
Contributor
jonenst commented Jan 7, 2017

https://travis-ci.org/jonenst/factor/builds/189845398

Got it to almost replicate the build bots results. Still need to make the imap and forestdb tests work...

@bpollack
Member
bpollack commented Jan 7, 2017

If our motivation is just that we wanted to get off relying on Mason, would looking at drone.io make sense here? Very similar idea to Travis CI, but they (currently?) are way faster, and can be self-hosted if we wanted to hybridize what we were up to. I'm only asking because this seems to have been a lot more effort than I think we were anticipating; I'm not otherwise interested in bikeshed colors.

@mrjbq7
Member
mrjbq7 commented Jan 7, 2017

This is pretty awesome, we might need to make some changes because of how we use mason to update our build dashboard, email reports, benchmark, and upload new images...

@jonenst
Contributor
jonenst commented Jan 9, 2017

https://travis-ci.org/jonenst/factor/jobs/190092435

It passed for linux. It's running load-all test-all but not help generation and benchmarks.
I think it should be possible to update the dashboard at http://builds.factorcode.org/dashboard because the child sends things over http (some things wouldn't work, like the heartbeat).

Do you want to take it from here ? Or I can run my own mason instance and test it there.

@mrjbq7
Member
mrjbq7 commented Jan 9, 2017

Any thoughts on getting Windows to work?

@mrjbq7
Member
mrjbq7 commented Jan 9, 2017

Also, of having both 32-bit and 64-bit builds?

@jonenst
Contributor
jonenst commented Jan 9, 2017
  • a quick google search suggests https://www.appveyor.com/ for windows
  • travis ci doesn't provide 32bits VMs. I don't know how much work it is to test a 32bit factor on a 64bit machine..
@mrjbq7
Member
mrjbq7 commented Jan 9, 2017 edited

It's pretty easy to build 32-bit on 64-bit, I do that on OS X all the time:

$ ./build.sh update macosx-x86-32

Note: that requires universal 32/64 builds of whatever library dependencies exist.

Also, linux is a bit different.

Ahh, I didn't realize Travis didn't have a Windows option.

@bjourne
Member
bjourne commented Jan 10, 2017

Pretty, pretty good. So as a next step we can make it so the Travis CI machine reports its build status to http://builds.factorcode.org/dashboard? There is a feature in mason for slaves to send their build status and logs to the master but I don't know how it works.

Linux is totally horrible for cross-running binaries. You can easily built a 32bit Factor vm, but then you also want it to dlopen 32bit libraries like sqlite.so and so on. Some libraries can be made to work by setting LD_LIBRARY_PATH to point to the directories containing the 32bit builds of them, but getting it to work really well I don't think is possible.

@jonenst
Contributor
jonenst commented Jan 11, 2017

Pretty, pretty good. So as a next step we can make it so the Travis CI machine reports its build status to http://builds.factorcode.org/dashboard? There is a feature in mason for slaves to send their build status and logs to the master but I don't know how it works.

Yes, that's what I was saying when I said "I think it should be possible to update the dashboard at http://builds.factorcode.org/dashboard because the child sends things over http [...] Do you want to take it from here ?"
because there is a shared secret for to be allowed to update the dashboard

@mrjbq7
Member
mrjbq7 commented Jan 11, 2017

How do you make shared secrets available to Travis but not just anyone?

@jonenst
Contributor
jonenst commented Jan 11, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment