[BKR-358] [DO NOT MERGE] add acceptance tests for dsl::helper::host_helpers
#930
Conversation
This is definitely useful for development, where, running acceptance tests on our vsphere cluster with just a couple of hosts in the config results in 2+ minute test times. When running acceptance tests that don't use puppet, using this pre-suite drops the test time to 18 seconds.
|
@rick why the empty pre-suite instead of not having a pre-suite? also, why is the |
This 🐕 told me to do that. But you make an interesting point. I can drop that pre-suite and just not run one. ✨ 🤘
I can move it there. I think I was mis-reading what |
|
Refer to this link for build results (access rights to CI server needed): |
Basic acceptance tests to cover changes in behavior for the highly-used `on` host helper method.
rick
force-pushed
the
bkr-358/add-acceptance-tests-for-dsl-helper-host_helpers
branch
from
b3cfd58
to
ab9766f
Compare
This reverts commit 9315248.
|
Refer to this link for build results (access rights to CI server needed): |
|
ha, yeah, actually, I realized that I merged some changes that broke this pattern recently, 👊 🌴 (no face palm, really?). Of course the pattern wasn't documented as far as I'm aware, so it might as well not exist. The original distinction was between tests that can run without puppet ( I like the idea of having that separation where as soon as we see whether base or puppet is in the path, we know whether we're breaking core beaker vs some interaction with something else, but I can understand the want to have the paths match up perfectly rather than having another distinction get in the way. I would recommend making the paths match the pattern in your changes to fix this, but am open to hearing why changing it would be better. |
|
No, I dig that totally. Mostly my confusion was just coming in to the project fresh and not catching the distinction. I think another benefit of the sort of separation you're talking about is that everything that's not under Thanks! ⚡ |
These don't require puppet (or much of anything else) to be installed, so they should actually be living under base/. h/t @kevpl
|
Refer to this link for build results (access rights to CI server needed): |
This isn't the hugest of deals, but, from personal experience, when searching or scanning for files, it makes epsilon of difference if test files have a suffix that readily distinguishes them from library files or helpers, etc.
|
Refer to this link for build results (access rights to CI server needed): |
|
Refer to this link for build results (access rights to CI server needed): |
|
Refer to this link for build results (access rights to CI server needed): |
This is a work in progress commit. I want to push up state because it turns out there is an ordering dependency in these tests; if scp_to fails due to the remote path not existing, it throws a RuntimeError, which appears to disable further SSH connections to the remote host. Subsequent tests will just hang. Probably a bug in beaker.
|
Refer to this link for build results (access rights to CI server needed): |
Note that a number of these are commented out currently due to problems with handing Net::SCP::Error exceptions raised during @ssh.close.
Currently I'm bypassing tests that are known to fail because of mis-handling Net::SCP::Error exceptions raised in @ssh.close. This just cleans that up a bit.
I guess this is sane-ish behavior for scp_from with a list of hosts. It's very order-dependent, and later hosts' content overwrites earlier hosts' content.
The reboot host test which sometimes times out ends up generating a SignalException for SIGTERM. It's not clear to me if this can be successfully caught in a way which allows for the acceptance suite to move forward (is this jenkins timing out due to lack of activity, or is this beaker generating a timeout exception due to not being able to connect?), but adding support here to attempt to catch the exception.
|
Refer to this link for build results (access rights to CI server needed): |
Without this, the `host_test` file will not be able to find the `fails_intermittently` method.
|
It's interesting that the way that the reboot failure occurs is that somehow we're not getting any output from (or around) the reboot test for 10 minutes, as far as jenkins is concerned. I'm able to verify, via This looks like classic output buffering: the reboot test is after the |
|
I've seen the truncated output intermittently... forever? I have never been able to track how/when it occurs. |
|
Refer to this link for build results (access rights to CI server needed): |
|
Confirmed that intermittent test failures are not actually intermittent. Beaker has a problem on centos7 / debian7 where when a host reboots and comes back up with a different IP address (which is happening on those platforms on our vmpooler), that beaker continues to try to SSH to that host using the same IP. c.f. https://tickets.puppetlabs.com/browse/QENG-3119 and https://tickets.puppetlabs.com/browse/QENG-3063 |
|
Refer to this link for build results (access rights to CI server needed): |
|
retest this please |
|
Refer to this link for build results (access rights to CI server needed): |
|
Test failures due to genconfig2 config not being appropriate (does not include two agent hosts) ... @kevpl was the job script changed? I think I had set it to use: and now it shows: I'm presuming if it was changed it was (a) because the new config broke something; (b) because it got restored from someplace else, (c) the config for other jobs needed to be tweaked in an incompatible way, (d) ??? (Yet another argument for getting this node under jjb management) |
|
retest this please |
1 similar comment
|
retest this please |
|
Refer to this link for build results (access rights to CI server needed): |
|
Refer to this link for build results (access rights to CI server needed): |
|
retest this please |
|
Refer to this link for build results (access rights to CI server needed): |
This test has proven particularly troublesome and we have a separate effort to diagnose and deal with this issue. This has been holding up landing of these acceptance tests for over a month and there's no reason to keep this code on a branch. Will re-enable this test via resolving QENG-3063.
This adds acceptance tests, along with a configuration file suitable for running these tests in our jenkins instance, for the Beaker host helpers. This also temporarily disables the "reboot" host acceptance test, which has been intermittently failing, and will be re-enabled via closure of QENG-3063. This work is detailed in the GitHub Pull Request at: #930
This adds acceptance tests, along with a configuration file suitable for running these tests in our jenkins instance, for the Beaker host helpers. This also temporarily disables the "reboot" host acceptance test, which has been intermittently failing, and will be re-enabled via closure of QENG-3063. This work is detailed in the GitHub Pull Request at: #930
|
Refer to this link for build results (access rights to CI server needed): |
|
🚢 in #1004 |
This adds acceptance tests, along with a configuration file suitable for running these tests in our jenkins instance, for the Beaker host helpers. This also temporarily disables the "reboot" host acceptance test, which has been intermittently failing, and will be re-enabled via closure of QENG-3063. This work is detailed in the GitHub Pull Request at: voxpupuli#930
Adding acceptance tests for Beaker's
dsl::helper::host_helpers(see: https://github.com/puppetlabs/beaker/blob/master/lib/beaker/dsl/helpers/host_helpers.rb)remaining work
CURRENTLYin the step descriptions) have documented what is the expected behavior insteadhosts.first->defaultTEST_SCP_ERROR_ON_CLOSE->BEAKER_ TEST_SCP_ERROR_ON_CLOSEtestfile.sh(and friends) ?rsyncissues on CentOS (it doesn't come installed by default)confinesyntax instead ofifblocksSkipTestinconfine_block#944 otherwiseconfine_blockdoesn't even workrsync returned #<Rsync::Result:0x00000003a40be8 @raw="ssh: connect to host 10.32.122.240 port 22: Connection timed out\r\nrsync: connection unexpectedly closed (0 bytes received so far) [sender]\nrsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9]\n", @exitcode=255>rsync: localhost:/tmp/beaker20150910-5591-1ot7dbp to root@10.32.125.204:/tmp/.AxwlnP/testfile.txt {:ignore => } rsync returned #<Rsync::Result:0x00000004b35d18 @raw="ssh: connect to host 10.32.125.204 port 22: Connection timed out\r\nrsync: connection unexpectedly closed (0 bytes received so far) [sender]\nrsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9]\n", @exitcode=255>/cc @puppetlabs/beaker