Plugin virt Tests
This page contains tests instructions for the virt plugin. This page is meant to aid in testing at release time and be a guide for developing automated tests. Please add any tests that you have run when testing this plugin.
NOTE:
- Tests cover CMT, CPU pinning info, CPU utilization, State metrics only.
- Several tests added for interaction coverage(restarting libvirtd, disabling metrics for VM, stopping VM, etc...)
- Added tests for MBM metric.
- Added 'sanity' tests for other metrics: CPU cycles/instructions, cache misses/references, interface statistic, disk and memory data.
- Added tests for disk errors, file system information and job statistics.
Test Environment details:
- Bare Metal, Ubuntu 16.04.1 LTS
- Kernel version: 4.4.0-43-generic
Repo/branch used:
- collectd/ feat_libvirt_upstreamed
Tests precondition:
- libvirt version used 2.4.0 (3.1.0)
- VM is started:
$ virsh start demo
$ root@silpixa00390838:~/orest/csv# virsh list
Id Name State
----------------------------------------------------
6 demo running
- Collectd is started with csv write plugin enabled and the following libvirt configuration:
Interval 2
LoadPlugin virt
<Plugin virt>
Connection "qemu:///system"
RefreshInterval 60
# Domain "demo"
# BlockDevice "name:device"
# BlockDeviceFormat target
# BlockDeviceFormatBasename false
# InterfaceDevice "name:device"
# IgnoreSelected false
# HostnameFormat name
# InterfaceFormat name
# PluginInstanceFormat name
Instances 1
#ExtraStats "cpu_util disk disk_err domain_state fs_info job_stats_background pcpu perf vcpupin memory_last_update"
</Plugin>
# |
Test summary |
Steps |
Expected Result |
---|---|---|---|
1 |
Verify that virt plugin dispatches CMT metrics. |
1. Start collectd with virt plugin and write plugin enabled(Interval is set to 1 second):
2. Get cmt metric from VM.
3. Wait 10 seconds(this is done in order to catch value written by collectd) 4. Stop collectd 5. Get collectd data:
6. Verify that value perf.cmt=196608 is present in collectd data. |
Since cmt performance metric is continuously changing it is complicated to catch equal values in virsh and collectd data. Therefore we are verifying that perf.cmt=196608 is present in collectd data. |
2 |
Verify that virt plugin does not dispatch CMT metrics when CMT has been disabled in VM |
1. Make sure that CMT metrics are dispatched by collectd:
2. Disable CMT metric in VM:
3. Verify that plugin stops dispatching cmt metrics:
|
Virt plugin should stop dispatching CMT metric when CMT is dynamically disabled for VM. |
3 |
Verify that virt plugin dispatches CPU pinning info metrics. |
1. Get CPU pinnning info using virsh tool:
2. Make sure that CPU pinning info metric is dispatched for all CPUs by collectd:
3.
|
|
4 |
Verify that virt plugin change CPU pinning info metric when its value is changed. |
1. Get CPU pinning info using virsh tool:
2. Make sure that pinning value for CPU-15 is equal to 1 in write collectd data:
3. Change CPU pinning using virsh tool:
4. Verify that CPU pinning info is changed to 0 in write collectd data:
|
CPU pinning info is changed to 0 in collectd data when its value was changed using virsh tool. |
5 |
Verify that virt plugin dispatches CPU utilization per VCPU in nanosecond format |
1. Start collectd with virt plugin and write plugin enabled. 2. Get vcpu metric from VM. 3. Wait collectd interval time for value update. 4. Stop collectd and get collectd data. 5. Compare utilizations for all vcpu in VM. |
|
6 |
Verify that notification raised when changed Virtual Machine state . |
1. Start VM: virsh start vm_name 2. Using exec plugin get notification message. 3. Reset VM: virsh reset vm_name 4. Using exec plugin get notification message. 5. Suspend VM: virsh suspend vm_name 6. Using exec plugin get notification message. 7. Resume VM: virsh resume vm_name 8. Using exec plugin get notification message. |
|
7 |
Verify that virt plugin starts dispatching data for newly created VM within RefreshInterval. |
1. One VM is in running state:
2. Set RefreshInterval to 10 seconds in collectd.conf:
3. Start collectd and immediately start second VM.
4. Make sure that data occurs after Refresh Interval. |
|
8 |
Verify that virt plugin stops dispatching data for deleted VM. |
1. Two VMs are running:
2. Set RefreshInterval to 10 seconds in collectd.conf:
3. Start collectd, immediately stop second VM and remove collectd data. 4. Make sure that some collectd metrics are still dispatched(State metrics)
5. Verify that collectd metrics stops dispatching after RefreshInterval |
|
9 |
Verify that virt plugin resumes dispatching data after libvirtd has been restarted |
1. Restart libvirtd service.
2. Wait until service is restarted.
3. Verify that virt plugin resumes collecting metrics. |
* Virt plugin resumes collecting metrics after libvirtd service has been restarted. |
10 |
Verify that virt plugin resumes dispatching data after VM has been restarted |
1. Restart VM:
2. Tail one of VM metrics(CPU total utilization):
3. Verify that virt plugin resumes collecting metrics. |
|
11 |
Verify that libvirt plugin correctly displays CPU utilization in percent in regular mode |
1. Start virt-top on server where VM is started 2. Open libvirt plugin file where CPU utilization is stored
3. Compare values NOTE: virt-top cuts output to 1 tenth symbol, and speed of values increment is different |
Note: In case of CPU is not loaded around zero values will be retrieved. |
12 |
Verify that libvirt plugin correctly displays CPU utilization in CPU load mode |
1. Start virt-top on server where VM is started 2. Open libvirt plugin file where CPU utilization is stored
3. Start stress tool on VM
4. Compare values NOTE: virt-top cuts output to 1 tenth symbol, and speed of values increment is different |
|
13 |
Verify that libvirt plugin correctly displays CPU utilization in percent upon VM restart |
1. Start virt-top on server where VM is started 2. Open libvirt plugin file where CPU utilization is stored:
3. Start stress tool on VM:
4. Start/Stop a VM within 60 seconds:
5. Compare values NOTE: virt-top cuts output to 1 tenth symbol, and speed of values increment is different |
* Values are pretty similar except 0 value every 10 sec |
14 |
Verify that libvirt plugin doesn't update CPU utilization if collectd is disabled |
1. Start virt-top on server where VM is started 2. Open libvirt plugin file where CPU utilization is stored
3. Compare values 4. Stop collectd 5. Compare values NOTE: virt-top cuts output to 1 tenth symbol, and speed of values increment is different |
* Update of percent-virt.. file has stopped |
15 |
Verify that libvirt plugin starts update CPU utilization if collectd is started |
1.Start virt-top on server where VM is started 2. Open libvirt plugin file where CPU utilization is stored
3. Stop collectd 4. Compare values 5. Start collectd 6. Compare values NOTE: virt-top cuts output to 1 tenth symbol, and speed of values increment is different |
* Update of percent-virt.. file has started NOTE: up to 10 sec before 1-st value appeared |
16 |
Verify that CPU utilization values are correct through at least 30-40 sec3 |
1. Start virt-top on server where VM is started 2. Open libvirt plugin file where CPU utilization is stored
3. Start stress tool on VM
4. Compare values NOTE: virt-top cuts output to 1 tenth symbol, and speed of values increment is different |
* Values are pretty similar except 0 value every 10 sec |
17 |
Verify libvirt collectd plugin MBM metric behavior upon enable/disable mbmt/mbml |
1. Start collectd with virt and csv (or other write) plugin enabled, libvirt. 2. Start vm and enable mbmt metric. Run some activity in VM:
3. Disable mbmt and enable mbml metric. 4. Disable mbml metric. 5. Enable both mbmt and mbml metric. |
|
18 |
Verify libvirt collectd plugin MBM metric updates every interval time set in collectd.conf. |
1. Start collectd with virt and csv (or other write) plugin enabled, libvirt. 2. Start vm and enable mbmt metric. Run some activity in VM: stress --cpu 2 --io 1 --vm 1 --vm-bytes 128M --timeout 20s 3. Change interval in collectd.conf. Restart collectd. 4. Repeat step#3 for different time intervals. |
|
19 |
Verify MBM metric upon collectd stop/start |
|
|
20 |
Verify libvirt collectd plugin MBM metric by un-/comment 'virt' in collectd.conf |
|
|
21 |
Verify MBM metric after libvirt service restart |
|
|
22 |
Verify MBM metric after VM start/destroy |
|
|
23 |
Verify libvirt collectd plugin MBM metric from two VMs |
|
|
24 |
Verify MBM metric after VM reboot, suspend, resume |
|
|
25 |
Verify zero disk errors are collected by virt plugin |
1. Start collectd with virt and wirte plugin enabled in collectd.conf.
2. Start VM 3. Get disk errors information using virsh tool and make sure no errors are present:
4. Start parsing syslog and make sure that plugin reportd zero disk errors. 5. Verify that plugin does not collectd any errors. |
* Plugin does not collectd any errors and syslog shows that zero disk errors are reported by collectd. |
26 |
Verify file system information reported by collectd corresponds to actual values of VM. |
1. Start collectd with virt and write plugin enabled in collectd.conf.
2. Make sure that exec plugin is enabled for capturing collectd notifications. exec_script:
collectd.conf for exec:
3. Get file system information using virsh utility:
4. Get Notification data reported by collectd:
5. Verify that notification data corresponds to data retrieved by virsh utility. |
Notification data corresponds to data retrieved by virsh utility. |
27 |
Verify job statistic is reported by virt plugin. |
1. Start collectd with virt and write plugin enabled in collectd.conf. 2. Set collectd read interval to 0.5 second in order to catch job statistics before VM exits. 3. Make sure that VM is in running state. 4. Perform virsh managedsave command and get job stat information using virsh in parallel.
5. Make sure that job information reported by collectd corresponds to values retrieved by virsh utility. |
* Job information reported by collectd corresponds to values retrieved by virsh utility. |
Test preconditions:
- Configured write, virt plugins in collectd.conf.
- Libvirtd, collectd services are running.
- perf metrics are enabled (either during VM definition or later on running VM).
# |
Test summary |
Steps |
Expected Result |
---|---|---|---|
1 |
Verify CPU cycles/instructions upon enable/disable of perf. |
1. Get CPU cycles metric from VM.
2. Compare CPU cycles values from virsh and write plugin (CSV). 3. Disable CPU cycles
4. Enable CPU cycles metric
5. Repeat same for CPU instructions metric. |
|
2 |
Verify CPU cycles/instructions upon collectd start/stop, change interval. |
1. Get CPU metrics from VM and collectd write plugin. 2. Stop collectd. Get CPU metrics from VM and collectd write plugin. 3. Start collectd. Get CPU metrics from VM and collectd write plugin. 4. Change interval in range 10-60 seconds (restart collectd). 5. Get CPU metrics from VM and collectd write plugin. |
|
3 |
Verify CPU cycles/instructions upon libvirtd start/stop, VM start/destroy. |
1. Get CPU metrics from VM and collectd write plugin. 2. Stop libvirtd. Get CPU metrics from VM and collectd write plugin. 3. Start libvirtd. Get CPU metrics from VM and collectd write plugin. 4. Stop VM
5. Start VM
|
|
4 |
Verify cache misses/references upon enable/disable of perf. |
1. Get cache misses metric from VM.
2. Compare cache misses values from virsh and write plugin (CSV).
3. Disable cache misses
4. Enable cache misses metric
5. Repeat same for cache references metric. |
|
5 |
Verify cache misses/references upon collectd start/stop, change interval. |
1. Get cache metrics from VM and collectd write plugin. 2. Stop collectd. Get cache metrics from VM and collectd write plugin. 3. Start collectd. Get cache metrics from VM and collectd write plugin. 4. Change interval in range 10-60 seconds (restart collectd). 5. Get cache metrics from VM and collectd write plugin. |
|
6 |
Verify cache misses/references upon libvirtd start/stop, VM start/destroy. |
1. Get cache metrics from VM and collectd write plugin. 2. Stop libvirtd. Get cache metrics from VM and collectd write plugin. 3. Start libvirtd. Get cache metrics from VM and collectd write plugin. 4. Stop VM
5. Start VM
|
|
7 |
Verify disk metrics upon collectd start/stop, change interval. |
1. Get disk number of operations/bytes read/write metrics from VM and collectd write plugin.
2. Stop collectd. Get disk metrics from VM and collectd write plugin. 3. Start collectd. Get disk metrics from VM and collectd write plugin. 4. Change interval in range 10-60 seconds. Get disk metrics from VM and collectd write plugin. |
|
8 |
Verify disk metrics upon libvirtd start/stop, VM start/destroy. |
1. Get disk number of operations/bytes read/write metrics from VM and collectd write plugin. 2. Stop libvirtd. Get disk metrics from VM and collectd write plugin. 3. Start libvirtd. Get disk metrics from VM and collectd write plugin. 4. Stop VM
5. Start VM
|
|
9 |
Verify interface metrics upon collectd start/stop, change interval. |
1. Get interface statistic from VM and collectd write plugin.
(files are:
) 2. Stop collectd. Get interface statistic from VM and collectd write plugin. 3. Start collectd. Get interface statistic from VM and collectd write plugin. 4. Change interval in range 10-60 seconds (restart collectd). 5. Get interface statistic from VM and collectd write plugin. |
|
10 |
Verify interface metrics upon libvirtd start/stop, VM start/destroy. |
1. Get interface statistic from VM and collectd write plugin. 2. Stop libvirtd. Get interface statistic from VM and collectd write plugin. 3. Start libvirtd. Get interface statistic from VM and collectd write plugin. 4. Stop VM
5. Start VM
|
|
11 |
Verify memory metrics upon collectd start/stop, change interval. |
1. Get memory metrics from VM and collectd write plugin.
(files are:
) 2. Stop collectd. Get memory metrics from VM and collectd write plugin. 3. Make changes to memory configuration
4. Start collectd. Get memory metrics from VM and collectd write plugin. 5. Change interval in range 10-60 seconds (restart collectd). 6. Get memory metrics from VM and collectd write plugin. |
NOTE:
|
12 |
Verify memory metrics upon VM start/destroy. |
1. Get memory statistic from VM and collectd write plugin. 2. Stop VM
3. Change max memory limit
4. Start VM
5. Get memory statistic from VM and collectd write plugin. |
|