Scheduled automated testing & notification emails

[jonathanolson]

It would be great to have testing run nightly, and send out an email if there are any failures?

Who should get sent emails?

[jonathanolson]

Probably using headless webkit, potentially on mendeleev (where phettest.colorado.edu lives)

[phet-steele]

@jonathanolson I'll take the emails if this ever happens. Maybe nightly is overkill though? We should give time for things to be fixed, or else we would be getting repetitive errors every night (assuming errors aren't fixed the same day).

[jonathanolson]

Nightly seems like it would work fine if it reports only new bugs (maybe a weekly email about current bugs).

[phet-steele]

Well certainly if it could manage to only report new ones, I'm all for nightly.

[samreid]

@andrewadare mentioned an interest in continuous integration services for automated testing.

[andrewadare]

@samreid I looked into two continuous integration options earlier this summer: Jenkins (self-hosted) and Travis-CI (remote). Jenkins turned out to be more complicated to configure than I anticipated. If people consider this a high priority, the topic can be revisited, but I have not gotten that impression. Since @jonathanolson has some experience in the area of automated testing, it may be valuable to confer with him.

[ariel-phet]

@andrewadare might be good to bring up again at an upcoming developer meeting. I would not say a "high priority" but I do want us to continue to work on it.

[samreid]

I'd recommend the following steps:

1. Choose a server which is always on which can devote resources to running our tests. This could begin life as a developer's computer (and local use for development), but we would need to have a plan for a server-based resource or the rest of this won't work so well.
2. Put it on a continuous loop: pull-all.js, clone-missing-repos.sh, run our existing tests (unit tests + sim fuzzer tests+built tests + lint tests) in browser and report issues somewhere (email/status page/create issues/nerf dart gun). We may use phantom for running the browser-side tests.
3. Then we can starting augmenting with additional tests.

[samreid]

By the way, I'd like to see the type checking tests into our normal tests, but it could take some work to get there (mainly telling it how our types are defined across files): #483

[pixelzoom]

@samreid What are "the type checking tests"?

[pixelzoom]

Ah, never mind. You're referring to flow-jsdoc (or something similar) as mentioned in #410.

[jonathanolson]

1. Choose a server which is always on which can devote resources to running our tests.

Figaro sounds potentially appropriate?

[samreid]

I tried running phantomjs on figaro by the following steps:
figaro:~/test-server # uname -a

Linux figaro.colorado.edu 2.6.18-411.el5 #1 SMP Thu Jun 2 02:56:21 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2

...
filename="phantomjs-2.1.1-linux-x86_64.tar.bz2"' saved [23415665/23415665]

tar -vxjf phantomjs-2.1.1-linux-x86_64.tar.bz2?Signature=pay7OdwEahqHoiYHg1JHp%2FZ6z4w=&Expires=1473898339&AWSAccessKeyId=AKIAIWY5XSVPZPDQYRQQ&versionId=null&response-content-disposition=attachment;\ filename="phantomjs-2.1.1-linux-x86_64.tar.bz2"

... unzipped

create hello.js with this content (from the Quick Start) http://phantomjs.org/quick-start.html

console.log('Hello, world!');
phantom.exit();

phantomjs-2.1.1-linux-x86_64/bin/phantomjs hello.js

phantomjs-2.1.1-linux-x86_64/bin/phantomjs: /lib64/libz.so.1: no version information available (required by phantomjs-2.1.1-linux-x86_64/bin/phantomjs)
phantomjs-2.1.1-linux-x86_64/bin/phantomjs: /usr/lib64/libstdc++.so.6: version GLIBCXX_3.4.9' not found (required by phantomjs-2.1.1-linux-x86_64/bin/phantomjs) phantomjs-2.1.1-linux-x86_64/bin/phantomjs: /usr/lib64/libstdc++.so.6: versionGLIBCXX_3.4.11' not found (required by phantomjs-2.1.1-linux-x86_64/bin/phantomjs)
phantomjs-2.1.1-linux-x86_64/bin/phantomjs: /lib64/libc.so.6: version GLIBC_2.9' not found (required by phantomjs-2.1.1-linux-x86_64/bin/phantomjs) phantomjs-2.1.1-linux-x86_64/bin/phantomjs: /lib64/libc.so.6: versionGLIBC_2.10' not found (required by phantomjs-2.1.1-linux-x86_64/bin/phantomjs)
phantomjs-2.1.1-linux-x86_64/bin/phantomjs: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by phantomjs-2.1.1-linux-x86_64/bin/phantomjs)

In the download docs: http://phantomjs.org/download.html It says:

Note: For this static build, the binary is self-contained. There is no requirement to install Qt, WebKit, or any other libraries. It however still relies on Fontconfig (the package fontconfig or libfontconfig, depending on the distribution). The system must have GLIBCXX_3.4.9 and GLIBC_2.7.

I wonder if we are missing some things?

[samreid]

We could build our own phantom but it seems somewhat invasive: http://phantomjs.org/build.html

[jonathanolson]

Just checked, but I can't 'sudo yum' to install packages globally. If we want more full control, we could go with something like phettest.colorado.edu, however it could slow down the normal operation (it's not terribly beefy).

[samreid]

I'm not sure what the status of figaro is, are we using it as a backup system? Will it still be managed by OIT? If not, perhaps we can have sudo. If it will be managed by OIT, then perhaps we just ask if they can install phantomjs (and other utilities as necessary).

[andrewadare]

A bit more background on my research from this summer: Travis CI and CircleCI are remotely-hosted options. For open-source repos, Travis is free, and I'm pretty sure that's also the case for CircleCI. But PhET-iO poses a problem, which is what put me off.

[samreid]

@ariel-phet says figaro is currently a failover machine. We would have to ask @kathy-phet when we might retire it.

[samreid]

But PhET-iO poses a problem, which is what put me off.

Can you elaborate?

Oh, is it because it is not open source?

[andrewadare]

It's not open source, so the licensing is different. We'd have to pay for CI services from Travis or CircleCI.

[samreid]

@ariel-phet said figaro may be retired, and we may use simian for testing? Let's talk with @kathy-phet about this.

[samreid]

We could run tests on simian, and just pause the tests if simian has to be used as a failover.

[samreid]

@mattpen is going to ping Doug Johnson about whether we can use simian for this.

[samreid]

OIT asked what we will need and I replied:

We would like to start with an installation of http://phantomjs.org/ and I don't know if there is already good software for scheduling recurrent tasks (could use cron but I heard there was something nicer). We will run a node program that uses phantomjs and sends out email notifications about failures. I suspect we already have node, not sure if we need something else to send emails from it. We'll also need to run git but I think that may already be installed.

[samreid]

I created repo "aqua" for Automatic QUality Assurance, and moved issues to that repo, closing this one and we will continue in phetsims/aqua#4