Configuring travis/buildbot testing to be more robust

[durack1]

@jbeezley @doutriaux1 @aashish24 this creates a dedicated issue to attempt to resolve a bunch of errors that continue to recur in the travis and buildbot build tests as discussed in #1381 and a number of other issues/PRs.
Specific changes to implement:

• Add redhat 6.6 platform (6.6 - crunchy.llnl.gov - ALL) to the buildbot suite
• Add redhat 6.7 platform (6.7 - oceanonly.llnl.gov - NOGUI) to the buildbot suite
• Add mac 10.10 platform (10.10 - macbot.llnl.gov - LEAN) to the buildbot suite
• Add mac 10.11 platform (10.11 - ALL) to the buildbot suite
• Add redhat 7.x platform (7.1 - pcmdi10.llnl.gov - ALL available) to the buildbot suite (Add RHEL7.2 to standard test and binary distribution platforms #457)
• Use travis_wait before the cmake call to prevent failures due to long waits between build steps (see here and here) - Is there a switch for cmake which will allow retrying dropped source download attempts too?
• Deal with timeouts during cdms tests to opendap servers (see here, here and here)
• Deal with race conditions in vtk image checks (see here, here, here and here - these errors show where the right-side image pixels are randomized on the canvas also here, an attempt to add a time.sleep(1) to prevent a race condition in another test was used in Issue788 durack1 remove deprecated folders and files #1341 here) - fixed by Added a sleep for onscreen rendered tests #1393

This might also be useful:
http://docs.travis-ci.com/user/build-timeouts/

[chaosphere2112]

@jbeezley It looks like all of the failures are on the same buildbot; the test-laptop one. Is that a particularly slow device?

[chaosphere2112]

Also, just submitted #1393 to help with some race conditions in my tests.

[durack1]

@potter2, @doutriaux1 and I were wondering if you may have any under-utilized mac hardware that would be suitable as a mac-buildbot candidate? We currently don't have any mac hardware available to run build tests, and a 10.9, 10.10 and possibly even a 10.11 machine would certainly be helpful if available..

[potter2]

I really don't have anything. Sorry

Jerry

Sent from my iPad

On Jun 22, 2015, at 11:25 AM, Paul J. Durack <notifications@github.commailto:notifications@github.com> wrote:

@potter2https://github.com/potter2, @doutriaux1https://github.com/doutriaux1 and I were wondering if you may have any under-utilized mac hardware that would be suitable as a mac-buildbot candidate? We currently don't have any mac hardware available to run build tests, and a 10.9, 10.10 and possibly even a 10.11 machine would certainly be helpful if available..

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1384#issuecomment-114208344.

[durack1]

@potter2 this is what @doutriaux1 and I spoke to you about last week, getting another mac to run ALL build tests.. @doutriaux1 already has a server configured, but we need more clients and you indicated you have a mac machine at home that could be used within the buildbot test suite..?

[potter2]

Paul,
I am at GSFC/NASA this week but I will be home next week. Maybe we can get it set up then. Will the build work on a home network?

Jerry

Gerald (Jerry) Potter
NASA Goddard Space Flight Center
E-mail gerald.potter@nasa.govmailto:gerald.potter@nasa.gov
Office 28:S241
Phone 707.315.8314
Analysis and Data Consultant
Department GSFC:606.2

On Sep 1, 2015, at 11:49 AM, Paul J. Durack <notifications@github.commailto:notifications@github.com> wrote:

@potter2https://github.com/potter2 this is what @doutriaux1https://github.com/doutriaux1 and I spoke to you about last week, getting another mac to run build tests.. @doutriaux1https://github.com/doutriaux1 already has a server configured, but we need more clients and you indicated you have a mac machine at home that could be used within the buildbot test suite..?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1384#issuecomment-136767023.

[durack1]

Great @potter2 yeah the build will work, as long as it is permanently connected.. @doutriaux1 can explain specifics, but simply when a pull request is issued through github, that new branch is built on the buildbot suite, in which we currently don't have a mac ALL test running.. If you have a fairly powerful machine that is always on, we could then use this to test an ALL mac build and improve the testing and build robustness on mac platforms..

[potter2]

I need to think about the mac connection. I’ll get back t you next week..

Jerry

Gerald (Jerry) Potter
NASA Goddard Space Flight Center
E-mail gerald.potter@nasa.govmailto:gerald.potter@nasa.gov
Office 28:S241
Phone 707.315.8314
Analysis and Data Consultant
Department GSFC:606.2

On Sep 1, 2015, at 1:44 PM, Paul J. Durack <notifications@github.commailto:notifications@github.com> wrote:

Great @potter2https://github.com/potter2 yeah the build will work, as long as it is permanently connected.. @doutriaux1https://github.com/doutriaux1 can explain specifics, but simply when a pull request is issued through github, that new branch is built on the buildbot suite, in which we currently don't have a mac ALL test running.. If you have a fairly powerful machine that is always on, we could then use this to test an ALL mac build and improve the testing and build robustness on mac platforms..

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1384#issuecomment-136809349.

[doutriaux1]

@durack1 no it is not a given, for example at my place ATT blocks me from talking over 9982... I tried to open the port but no luck so far, even tried using other ports. From talking to ATT they have no idea my port stays blocked even after I tell my router to open it... So it's not 100% sure.

[doutriaux1]

@potter2 any progress with ucdavis?

[durack1]

@doutriaux1 could a routing rule be setup?

[doutriaux1]

I tried that and talked with ATT they were baffled it didn't work...

[jbeezley]

There is always ngrok.

[doutriaux1]

oh that's looks nice, maybe i will try that.

[doutriaux1]

will add a mac 10.11 when it's out officially (and bug have been ironed out)

[doutriaux1]

i think we aare pretty much abandoning travis at that point @chaosphere2112 are we getting a pro licence or not? If not let's dump travis

[durack1]

@doutriaux1 Xcode 7 was released today which goes along with OS X 10.11, so probably worth dropping that into the buildbot suite as soon as an OS X machine becomes available..

[doutriaux1]

Once the lab has an approved image for the green network will do.

[doutriaux1]

http://github.com/UV-CDAT/uvbot