forked from crowbar/crowbar
/
README.testing
498 lines (471 loc) · 25.1 KB
/
README.testing
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
Prerequisites:
* An x86_64 system with at least 16 GB of RAM, 8 cores, and 50 GB free
space in /home. The test framework deploys Crowbar to a set of
virtual machines, and you need a fairly beefy system to run them all.
* Bash 4 or higher.
The test framwork uses associative arrays to track the state of
the virtual machines it runs.
* cgroup support in the kernel.
The framework uses cgroups to ensure that it cleans up after itself
properly after a run.
* A dedicated user account for running tests in.
The framework expects to have a its own user account to store test
logs and hold the disk images.
* read/write access to /dev/kvm
The test framework runs its tests using KVM virtual machines.
* screen
The test framwork arranges to attach the KVM consoles to screen, and it
configures screen to capture all output from those consoles.
* brctl
The test framework arranges for the vms to talk over an isolated
network.
* ip tuntap
The test framework needs to set up and tear down tap interfaces,
and it relies on having a version of the ip command that includes
tuntap support.
* Two specific ruby gems:
* net-http-digest_auth
* json
Both of these gems should be installed systemwide on the test machine
using the gem install command before trying to run the framework.
* 192.168.124.0/24
The admin node expects to live on 192.168.124.10, and the framework
needs to be able to talk to the admin node using IP. It expects
to be able to bind 192.168.124.1 to the first bridge it creates.
* Passwordless sudo to root for the following commands:
* /bin/mount, /bin/umount
The test framework needs to be able to mount and umount the Crowbar
image that it will test and the cgroup filesystem that it will track its
pids.
* /bin/ip
The test framework needs to be able to set up and tear down tap interfaces
for the KVM virtual machines.
* /usr/sbin/brctl
The test framework needs to be able to set up and tear down bridges for
the KVM virtual machines to talk over.
* $CROWBAR_DIR/test_framework/make_cgroups.sh
make_cgroups.sh is the utility that we use to mount the cgroup filesystem
we track the pids we have spawned and that we use to kill all our
child processes when cleaning up.
Usage:
* Run the framework from the main Crowbar git checkout.
sudo uses absolute paths to determine whether a program can run under it,
and we need to run make_cgroups.sh as root to make cgroups for the overall
smoketest and for the individual test runs. The framework will check to
make sure it has all the access it needs before starting a test run,
and it will print a helpful error message if it is missing something.
* You have to pass the ISO you want to test with the use-iso parameter.
* Run test_crowbar.sh with the appropriate options.
If you run it with no options, it will run a scratch test (which spins up
the admin and compute nodes and ensures that it can deploy an OS on all the
nodes using Crowbar).
Primary Operating Modes:
* <barclamps> <test-options>
This instructs the test framework to deploy a cluster, and then test the
specified barclamps with the framework running with the specified options.
Framework options change over time -- run test_crowbar.sh help to see
the current ones.
Example: test_crowbar.sh nova
* cleanup
This instructs the test framework to clean up as much as it can from
the previous run if it did not get a chance to perform its usual cleanup.
This will try to deallocate any stray tap interfaces, tear down bridges,
kill any stray processes, and umount any stray mounts we may have left
behind.
Available Tests:
By default, you can pass the names of any barclamps as tests, and the
framework will try to deploy the barclamps (and their dependencies) and run
any tests that the barclamp has available.
Test Options:
Run test_crowbar.sh help to see a list of options for the framework.
Usage Hints:
* screen -r crowbar-test
The smoketest framework maintains a screen session named crowbar-test
that holds a shell session using the same environment the tests will run
with, a continually updated snaptshot of the state of the tests, and the
text-mode video console of all the virtual machines.
* (unset DISPLAY; $HOME/test_framework/test_crowbar.sh)
By default the test framework will let KVM use SDL video output if it
detects X. Unsetting DISPLAY before running test_crowbar.sh will force
it to not do that.
* xhost +local:; sudo su - crowbar-tester
Having to maintain two different X sessions to run things as crowbar-tester
(or whatever you want to name the user running the smoketests) is annoying.
xhost +local: will allow anyone connecting to the local Unix domain socket
for your X session to display applications.
Directory Layout:
The smoketest framework expects to be able to control the layout of its home
directory. It generates the following directory structure on its first run:
$CROWBAR_DIR/testing
|- cli
| \- (the last copy of the Crowbar CLI that we extracted)
\- crowbar-being-tested
|- admin.disk
|- admin.errlog
|- admin.pid
|- admin.status
|- (repeat for the other virtual machines)
|- image
| \- (contents of crowbar-being-tested.iso)
\- logs
|- admin.1
| |- ttyS0.log
| \- ttyS1.log
|- screenlog.admin.1
|- (repeat for each generation of each virtual machine)
\- (log snapshots of the cluster at various checkpoints)
Each of the above files has a specific type of information:
* cli
The cli directory holds the last copy of the Crowbar CLI that we extracted,
along with a couple of auxillary commands that the framework automatically
creates. The cli directory is added to $PATH so that the test hooks
can use the Crowbar CLI without needing to extract their own copy.
* crowbar-being-tested
This directory holds all of the test information that the test framework
uses and captures on any single run of tests.
* admin.disk
This is the disk image for the admin node. There will be at least one
disk image for each virtual machine that is part of the test.
* admin.errlog
When the test framework deploys the admin node, it captures all stderr
output to this file. There will be an errlog file for each virtual machine
that the framework deploys.
* admin.pid
This is the PID of the KVM instance that is currently running the admin
node. The actual PID of any given virtual machine will change each time
a VM has to restart, so we have KVM always log the correct PID here.
There will be a pid file for each virtual machine.
* admin.status
This file is where the test framework tracks the status of the admin
node through a run of the test framework. There is a status file for
each virtual machine.
* image
This is where the crowbar .iso we are currently testing is mounted.
* logs
This is where the logs from each individual KVM instance are captured to.
Since each virtual machine will be restarted at least once during a test
run, we capture the logs from KVM seperatly for each instance, and we group
the KVM runs into generations -- the first run if a given VM is generation
1, the seconnd is generation 2, and so on.
* admin.1
This is where the logs from the first generation of the admin VM
are saved. Each generation of each VM will have its own log directory.
* ttyS0.log
This is where anything that is output to ttyS0 from the VM is saved to.
The test framework configures syslog on each virtual machine to write
everything it captures to ttyS0, making this logfile a transcript
of everything logged to syslog. By default, Crowbar configures everything
that is capable of logging to syslog to do so.
* ttyS1.log
This is where anything written to ttyS1 in the VM is saved to. By default,
the test framework configures Crowbar to configure each machine it deploys
to use ttyS1 as a serial console, so everything written to the console
will be saved to this file.
* screenlog.admin.1
By default, we configure KVM run under screen using the curses-based text
console. As long as a VM does not switch to graphics mode, everything that
the VM displays to the video hardware will be saved in this log. This
is useful for debugging VM failures that happen before the system
switches over to using the serial console. This logfile has lots of
ANSI escape sequences because it is an exact transcript of what KVM is
writing to the screen -- you should run it through strings if you want
to get easier-to-parse information from it.
In addition to the above files, the test framework will gather a complete
log and config snapshot of the cluster from Crowbar at the following points
in a test run:
* When the admin node indicates that it has deployed and the admin node
sanity tests have passed.
* When a test has completed, whether that test passed or failed
* Just before the virtual machines are killed at the end of a test run.
These log bundles will be saved as a tarball in the directory that
the smoketest was executed from, and have names in the
format of checkpoint_name-checkpoint-status-YYYYMMDD-HHMMSS.tar, and
have the following directory layout when untarred:
YYYYMMDD-HHMMSS-five_random_characters
|- nodename
| \- nodename-YYYYMMDD-HHMMSS.tar.gz
\- (repeat for the rest of the nodes in the cluster that are up)
The tarfile for each node contains all of /etc and /var/log. The tarfile for
the admin node also contains /opt/dell/crowbar_framework/db (which is a dump
of the Crowbar-specific parts of the Chef database), and /install-logs (which
contains a capture of the logs the rest of the machines create in discovery
phase).
Example Test Run Transcript:
[crowbar-tester@testbox ~]$ (unset DISPLAY; test_crowbar.sh run-test scratch)
2011-07-16 11:32:29 -0500: admin - Mounting .iso
2011-07-16 11:32:29 -0500: admin - Hacking up kernel parameters
2011-07-16 11:32:29 -0500: admin - Creating disk image
2011-07-16 11:32:29 -0500: admin - Performing install from crowbar-dev.iso
2011-07-16 11:39:09 -0500: admin - Node exited.
2011-07-16 11:39:09 -0500: admin - Deploying admin node crowbar tasks
2011-07-16 11:44:55 -0500: admin - admin.smoke.test is discovered.
2011-07-16 11:47:01 -0500: admin - admin.smoke.test is hardware-installing.
2011-07-16 11:47:11 -0500: admin - admin.smoke.test is hardware-installed.
2011-07-16 11:47:32 -0500: admin - admin.smoke.test is installing.
2011-07-16 11:47:42 -0500: admin - admin.smoke.test is installed.
2011-07-16 11:47:52 -0500: admin - admin.smoke.test is readying.
2011-07-16 11:48:02 -0500: admin - admin.smoke.test is ready.
2011-07-16 11:48:02 -0500: admin - Daemonizing node with 1268 seconds left.
crowbar
crowbar_bios
crowbar_crowbar
crowbar_deployer
crowbar_dns
crowbar_ganglia
crowbar_glance
crowbar_hadoop
crowbar_logging
crowbar_machines
crowbar_nagios
crowbar_network
crowbar_node_state
crowbar_node_status
crowbar_nova
crowbar_ntp
crowbar_provisioner
crowbar_swift
crowbar_test
crowbar_watch_status
barclamp_lib.rb
barclamp_test.rb
blocking_chef_client.sh
gather_cli.sh
gather_logs.sh
install.sh
install_barclamp.sh
looper_chef_client.sh
single_chef_client.sh
transition.sh
update_hostname.sh
net-http-digest_auth-1.1.1.gem
PING 192.168.124.10 (192.168.124.10) 56(84) bytes of data.
64 bytes from 192.168.124.10: icmp_req=1 ttl=64 time=0.185 ms
--- 192.168.124.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.185/0.185/0.185/0.000 ms
apt_test: Success: empty list returned for /
apt_test: Success: empty list returned for /proposals
apt_test: Success: cli returned empty list for /
apt_test: Success: cli returned empty list for /proposals
apt_test: Success: create test_p1 proposal
apt_test: Success: create test_p1 proposal with bad url
apt_test: Success: list of proposals with test_p1
apt_test: Success: getting test_p1
apt_test: Success: editting test_p1
apt_test: Success: failed to edit test_p1
apt_test: Success: delete test_p1 proposal
apt_test: Success: not finding to delete test_p1 proposal
apt_test: Success: create test_p2 proposal
apt_test: Success: created committing test_p2
apt_test: Success: list of proposals with test_p2
apt_test: Success: list with test_p2
apt_test: Success: getting test_p2
apt_test: Success: delete test_p2
apt_test: Success: delete missing test_p2
apt_test: Success: delete test_p2 proposal
apt_test: Success: empty list returned for /
apt_test: Success: empty list returned for /proposals
apt_test: Success: cli returned empty list for /
apt_test: Success: cli returned empty list for /proposals
node_manipulate: Success: empty list returned for /
node_manipulate: Success: empty list returned for /proposals
node_manipulate: Success: cli returned empty list for /
node_manipulate: Success: cli returned empty list for /proposals
node_manipulate: Success transition dtest-machine-1.dell.com: result test
node_manipulate: Success transition dtest-machine-1.dell.com: name test
node_manipulate: Success: found dtest-machine-1.dell.com in node list
node_manipulate: Success: create test_p1 proposal
node_manipulate: Success creating proposal with test machine single
node_manipulate: Success creating proposal without test-multi-head
node_manipulate: Success creating proposal without test-multi-rest
node_manipulate: Success transition dtest-machine-2.dell.com: result test
node_manipulate: Success transition dtest-machine-2.dell.com: name test
node_manipulate: Success: found dtest-machine-2.dell.com in node list
node_manipulate: Success: queued committing test_p1
node_manipulate: Success: not finding active test_p1
node_manipulate: Success transition dtest-machine-1.dell.com: result test
node_manipulate: Success transition dtest-machine-1.dell.com: name test
node_manipulate: Success: found dtest-machine-1.dell.com in node list
node_manipulate: Success: found active test_p1
node_manipulate: Success creating role with test machine single
node_manipulate: Success creating role without test-multi-head
node_manipulate: Success creating role without test-multi-rest
node_manipulate: Success: delete test_p1 proposal
node_manipulate: Success: delete test_p1 active
node_manipulate: Success: empty list returned for /
node_manipulate: Success: empty list returned for /proposals
node_manipulate: Success: cli returned empty list for /
node_manipulate: Success: cli returned empty list for /proposals
node_manipulate: Success: create test_p1 proposal
node_manipulate: Success creating role without test machine single
node_manipulate: Success creating proposal with test-multi-head
node_manipulate: Success creating proposal with test-multi-rest
node_manipulate: Success: delete test_p1 proposal
node_manipulate: Success: empty list returned for /
node_manipulate: Success: empty list returned for /proposals
node_manipulate: Success: cli returned empty list for /
node_manipulate: Success: cli returned empty list for /proposals
Deleted node[dtest-machine-1.dell.com]!
Deleted node[dtest-machine-2.dell.com]!
2011-07-16 11:51:33 -0500: admin - Admin deploy tests passed
2011-07-16 11:51:33 -0500: admin - Node deployed.
2011-07-16 11:51:33 -0500: Gathering admin-deployed logs, please wait.
2011-07-16 11:51:35 -0500: Done gathering admin-deployed logs.
2011-07-16 11:51:35 -0500: virt-1 - Creating disk image
2011-07-16 11:51:35 -0500: virt-2 - Creating disk image
2011-07-16 11:51:35 -0500: virt-3 - Creating disk image
2011-07-16 11:51:36 -0500: virt-4 - Creating disk image
2011-07-16 11:51:45 -0500: virt-1 - PXE booting node (0)
2011-07-16 11:51:45 -0500: virt-2 - PXE booting node (0)
2011-07-16 11:51:46 -0500: virt-3 - PXE booting node (0)
2011-07-16 11:51:46 -0500: virt-4 - PXE booting node (0)
2011-07-16 11:54:38 -0500: virt-3 - d52-54-00-12-34-07.smoke.test is discovered.
2011-07-16 11:54:38 -0500: virt-2 - d52-54-00-12-34-05.smoke.test is discovered.
2011-07-16 11:54:39 -0500: virt-3 - allocate d52-54-00-12-34-07.smoke.test
2011-07-16 11:54:39 -0500: virt-2 - allocate d52-54-00-12-34-05.smoke.test
2011-07-16 11:54:48 -0500: virt-1 - d52-54-00-12-34-03.smoke.test is discovered.
2011-07-16 11:54:48 -0500: virt-1 - allocate d52-54-00-12-34-03.smoke.test
2011-07-16 11:54:50 -0500: virt-3 - d52-54-00-12-34-07.smoke.test is discovered.
2011-07-16 11:54:50 -0500: virt-2 - d52-54-00-12-34-05.smoke.test is discovered.
2011-07-16 11:54:50 -0500: virt-3 - allocate d52-54-00-12-34-07.smoke.test
2011-07-16 11:54:51 -0500: virt-2 - allocate d52-54-00-12-34-05.smoke.test
2011-07-16 11:54:51 -0500: virt-4 - d52-54-00-12-34-09.smoke.test is discovered.
2011-07-16 11:54:52 -0500: virt-4 - allocate d52-54-00-12-34-09.smoke.test
2011-07-16 11:54:59 -0500: virt-1 - d52-54-00-12-34-03.smoke.test is discovered.
2011-07-16 11:54:59 -0500: virt-1 - allocate d52-54-00-12-34-03.smoke.test
2011-07-16 11:55:01 -0500: virt-3 - d52-54-00-12-34-07.smoke.test is hardware-installing.
2011-07-16 11:55:01 -0500: virt-2 - d52-54-00-12-34-05.smoke.test is discovered.
2011-07-16 11:55:01 -0500: virt-2 - allocate d52-54-00-12-34-05.smoke.test
2011-07-16 11:55:02 -0500: virt-4 - d52-54-00-12-34-09.smoke.test is hardware-installing.
2011-07-16 11:55:10 -0500: virt-1 - d52-54-00-12-34-03.smoke.test is hardware-installing.
2011-07-16 11:55:11 -0500: virt-3 - d52-54-00-12-34-07.smoke.test is hardware-installed.
2011-07-16 11:55:12 -0500: virt-2 - d52-54-00-12-34-05.smoke.test is hardware-installing.
2011-07-16 11:55:12 -0500: virt-4 - d52-54-00-12-34-09.smoke.test is hardware-installed.
2011-07-16 11:55:20 -0500: virt-1 - d52-54-00-12-34-03.smoke.test is hardware-installed.
2011-07-16 11:55:22 -0500: virt-2 - d52-54-00-12-34-05.smoke.test is hardware-installed.
2011-07-16 11:55:58 -0500: virt-4 - Node exited.
2011-07-16 11:55:58 -0500: virt-4 - PXE booting node (1)
2011-07-16 11:55:59 -0500: virt-3 - Node exited.
2011-07-16 11:55:59 -0500: virt-3 - PXE booting node (1)
2011-07-16 11:56:06 -0500: virt-1 - Node exited.
2011-07-16 11:56:06 -0500: virt-1 - PXE booting node (1)
2011-07-16 11:56:07 -0500: virt-2 - Node exited.
2011-07-16 11:56:07 -0500: virt-2 - PXE booting node (1)
2011-07-16 12:00:06 -0500: virt-2 - d52-54-00-12-34-05.smoke.test is installing.
2011-07-16 12:00:06 -0500: virt-4 - d52-54-00-12-34-09.smoke.test is installing.
2011-07-16 12:00:07 -0500: virt-1 - d52-54-00-12-34-03.smoke.test is installing.
2011-07-16 12:00:07 -0500: virt-3 - d52-54-00-12-34-07.smoke.test is installing.
2011-07-16 12:00:36 -0500: virt-2 - d52-54-00-12-34-05.smoke.test is installed.
2011-07-16 12:00:37 -0500: virt-4 - d52-54-00-12-34-09.smoke.test is installed.
2011-07-16 12:00:37 -0500: virt-1 - d52-54-00-12-34-03.smoke.test is installed.
2011-07-16 12:00:38 -0500: virt-3 - d52-54-00-12-34-07.smoke.test is installed.
2011-07-16 12:01:05 -0500: virt-4 - Node exited.
2011-07-16 12:01:06 -0500: virt-4 - Booting node to disk (2)
2011-07-16 12:01:09 -0500: virt-1 - Node exited.
2011-07-16 12:01:09 -0500: virt-1 - Booting node to disk (2)
2011-07-16 12:01:11 -0500: virt-2 - Node exited.
2011-07-16 12:01:11 -0500: virt-3 - Node exited.
2011-07-16 12:01:11 -0500: virt-2 - Booting node to disk (2)
2011-07-16 12:01:11 -0500: virt-3 - Booting node to disk (2)
2011-07-16 12:01:18 -0500: virt-4 - d52-54-00-12-34-09.smoke.test is readying.
2011-07-16 12:01:28 -0500: virt-2 - d52-54-00-12-34-05.smoke.test is readying.
2011-07-16 12:01:28 -0500: virt-1 - d52-54-00-12-34-03.smoke.test is readying.
2011-07-16 12:01:29 -0500: virt-3 - d52-54-00-12-34-07.smoke.test is readying.
2011-07-16 12:03:53 -0500: virt-4 - d52-54-00-12-34-09.smoke.test is ready.
2011-07-16 12:03:53 -0500: virt-4 - Node deployed.
2011-07-16 12:03:53 -0500: virt-3 - d52-54-00-12-34-07.smoke.test is ready.
2011-07-16 12:03:53 -0500: virt-3 - Node deployed.
2011-07-16 12:03:53 -0500: virt-1 - d52-54-00-12-34-03.smoke.test is ready.
2011-07-16 12:03:53 -0500: virt-1 - Node deployed.
2011-07-16 12:04:13 -0500: virt-2 - d52-54-00-12-34-05.smoke.test is ready.
2011-07-16 12:04:13 -0500: virt-2 - Node deployed.
2011-07-16 12:04:31 -0500: scratch deploy passed.
2011-07-16 12:04:41 -0500: scratch tests passed.
2011-07-16 12:04:41 -0500: Gathering scratch-tests-passed logs, please wait.
2011-07-16 12:04:46 -0500: Done gathering scratch-tests-passed logs.
2011-07-16 12:04:46 -0500: virt-4 - Asking d52-54-00-12-34-09.smoke.test to shut down
2011-07-16 12:04:47 -0500: virt-2 - Asking d52-54-00-12-34-05.smoke.test to shut down
2011-07-16 12:04:48 -0500: virt-3 - Asking d52-54-00-12-34-07.smoke.test to shut down
2011-07-16 12:04:49 -0500: virt-1 - Asking d52-54-00-12-34-03.smoke.test to shut down
2011-07-16 12:04:49 -0500: Waiting for 2 minutes to allow nodes to power off
2011-07-16 12:04:50 -0500: virt-2 - Node exited.
2011-07-16 12:04:50 -0500: virt-3 - Node exited.
2011-07-16 12:04:51 -0500: virt-2 - Creating disk image
2011-07-16 12:04:51 -0500: virt-3 - Creating disk image
2011-07-16 12:05:02 -0500: virt-4 - Node exited.
2011-07-16 12:05:03 -0500: virt-4 - Creating disk image
2011-07-16 12:05:03 -0500: virt-1 - Node exited.
2011-07-16 12:05:04 -0500: virt-1 - Creating disk image
2011-07-16 12:06:49 -0500: virt-4 - Forcing d52-54-00-12-34-09.smoke.test to shut down
Deleted d52-54-00-12-34-09.smoke.test
2011-07-16 12:06:49 -0500: virt-2 - Forcing d52-54-00-12-34-05.smoke.test to shut down
Deleted d52-54-00-12-34-05.smoke.test
2011-07-16 12:06:50 -0500: virt-3 - Forcing d52-54-00-12-34-07.smoke.test to shut down
Deleted d52-54-00-12-34-07.smoke.test
2011-07-16 12:06:50 -0500: virt-1 - Forcing d52-54-00-12-34-03.smoke.test to shut down
Deleted d52-54-00-12-34-03.smoke.test
2011-07-16 12:06:51 -0500: Gathering final logs, please wait.
2011-07-16 12:06:54 -0500: Done gathering final logs.
2011-07-16 12:06:55 -0500: admin - Killing (try 1, signal TERM)
2011-07-16 12:06:55 -0500: admin - Killed with SIGTERM
2011-07-16 12:06:56 -0500: virt-2 - Asked to be killed, giving up.
2011-07-16 12:06:57 -0500: virt-3 - Asked to be killed, giving up.
2011-07-16 12:06:58 -0500: virt-4 - Asked to be killed, giving up.
2011-07-16 12:07:01 -0500: virt-1 - Asked to be killed, giving up.
deploy_admin: Passed
scratch: Passed
Deploy Passed.
Logs are available at /home/crowbar-tester/tested/crowbar-dev-20110716-120701-Passed.tar.gz.
[crowbar-tester@testbox ~]$
From this, you can see the process the smoketest uses to set up Crowbar:
* Mount the crowbar .iso we will test on the host machine.
* Inject the parameters that the smoketest frawework needs to facilitate
automated testing (the default hostname, serial logging, serial console
settings, and the SSH public key of the user running the test).
* Install the admin node from the crowbar ISO using the extra test
parameters
* Deploy Crowbar on the freshly installed admin node, cycling through the
normal node deploy phases as we go.
* Once the admin node is deployed, stop active status monitoring of it,
get a copy of the Crowbar CLI and install it on the host in testing/cli,
run some basic sanity tests and gather logs.
* Once the sanity tests have passed, spin up the rest of the virtual
machines and walk through their initial deploy process.
* Once the rest of the virtual machines have deployed, walk through the
tests we have asked the smoketest to run. In this instance, the test
framework was asked to run a scratch test, which just checks that it can
deploy nodes with just a basic OS install on them. If the smoketest
framweork was asked to test a different set of barclamps, it would have
tried to deploy them to the cluster as well.
Additionally, there are several environment variables that the test framework
sets:
* SMOKETEST_DIR
This environment variable points at the directory holds the test hooks
that are currently running. You can reference any other files in your test
via this variable.
* SMOKETEST_LOGDIR
This environment contains that directory that the hooks in the smoketest
should log to. The smoketest framework will automatically capture all
output from the tests into $SMOKETEST_LOGDIR/capture.log, and any proposals
that had to be created, modified, or committed will also be saved here.
* CROWBAR_IP
This is the IP address of the admin node.
* CROWBAR_KEY
This is the default username and password needed to perform digest auth
with Crowbar. It is normally crowbar:crowbar
Each smoketest has full access to the Crowbar CLI -- a fresh copy is downloaded
on every run of the framework, and $PATH is modified to contain it. The
following commands are also available:
* run_on
run_on uses passwordless SSH access to run commands as root on the deployed
VMs. run_on can handle IP addresses and hostnames -- if handed a hostname,
it looks it up using DNS from the admin node. The first argument should be
an IP address or a hostname, and the rest of the arguments are the
commandline that you want to run on the host.
* knife
This command uses run_on to invoke knife on the admin node. Any arguments
are passed unmodified to the knife instance on the admin node.
* knife_node_find
This is a thin wrapper around knife search node. It takes two arguments:
1: The search string to use. This is passed unchanged to knife node search
2: The section of the search results to print (such as IP or FQDN).