TestProcedure

ToMaTo test procedure

Those tests should be performed for each release. Any failure should be documented and categorized:

• Occasional: Failures that only occur sometimes but not always.
• Non-critical: Failures that occur consistently but do not disable any main feature or for which a workaround exists.
• Critical: Failures that occur consistently and disable a main feature.

New releases must not have any known non-critical or critical failures. Critical failures in existing deployment should be fixed by a minor bugfix release.

User tests

These tests do not require administrative permissions.

Tutorials

• Chat tutorial
• OpenFlow tutorial

Editor tests

• Placing elements
• Connecting elements
• Changing attributes
• Interface auto-configuration

Topology design

• Scenario 1: 2 VMs (OpenVZ and KVM) and 1 switch
• Scenario 2: 2 VMs with 2 switches in series
• Scenario 3: 2 VMs with 2 switches in parallel

Topology actions

• Complete actions
  • Prepare
  • Destroy
  • Start (if unprepared)
  • Stop (if started)
  • Destroy (if started)
  • Remove (if prepared)
• Complete actions with different element states
• Single actions

Element types

• OpenVZ
• KVM
• Repy
• Switch
• External networks

Connection types

• Normal connection (with ping check)
  • Scenario 1: 2 VMs, 1 switch
  • Scenario 2: 2 VMs, 2 switches in a row
• To external network (with ping check)
  • Scenario 1: 1 VM, 1 external network
    • check Internet access
  • Scenario 2: 1 VM, 1 switch, 1 external network
    • check Internet access
  • Scenario 3: 2 VMs, 1 external network
    • check ping via external network

Link emulation

• Default values
  • Emulation on, bandwidth limit: 10 MBit/s, everything else off
• Latency (test via ping)
  • Test changes
• Loss ratio (test via ping)
• Duplication (test via ping)
• Bandwidth (test via iperf, or ping -s)

Packet capturing

• Capturing for download
  • Open in Wireshark
  • Open in Cloudshark
  • Size restrictions
• Live capture
• Filters work

Image upload/download

• Image upload works on all VM types
• Image download works on all VM types
• Creation with custom image works on all VM types

Executable archives

• Uploading executable archives works
• Archives execute when starting VMs
• Archives execute when uploaded to running VM
• Status is displayed correctly
• Archive download works

Monitoring

• Resource usage is tracked correctly
• Resource usage is updated after at most 10 minutes

Administrative tests

Those tests require additional permissions

User management

• Registering new accounts by anonymous users
• Registering new accounts by administrators
• Accepting new accounts
• Changing account details
  • By users
  • By administrators
• Changing password
• Resetting password
• Deleting user

Template management

• Creating a new template
• Template distribution works
• Modifying template works
• Deleting template works
• Downloading template works
  • Downloading torrent file
  • Download actual template

Host management

• Host installation howto works
• Adding new hosts works
• Problems on hosts are detected
  • High load
• Modifying hosts works
• Deleting hosts works
• Public key can be downloaded

Site/Organization management

• Adding organizations works
• Modifying organizations works
• Removing organizations works
• Adding sites works
• Modifying sites works
• Removing sites works
• Site map displays correctly

Permissions

• Restricted templates work
• Restricted profiles work
• Restricted external networks work

Other

Browser compatibility

• Current Chrome version
  • Chat tutorial works
• Current Firefox version
  • Chat tutorial works
• Current Internet Explorer version
  • Chat tutorial works

Templates

Tests for each template

• Boot works
• Keyboard language is correct
• Not outdated (< 1 year)
• Executable archive support flag is correct
• Executable archives supported (main templates only)
• No open ports
• Network configuration in editor works (where possible)
  • DHCP
  • IP/Netmask
  • Gateway
• Interfaces set to DHCP (no auto-config)

Templates to test

• Current Debian OpenVZ
• Current Debian KVM
• Current Ubuntu OpenVZ
• Current Ubuntu KVM

External networks (Internet)

• DHCP works if advertised
• Instances for each host
• Network is a single layer-2 network
• Blocked ports are documented
• Other restrictions are documented

Executable archives

• Installer archives exist for
  • Chat application (tutorial)
  • Python
  • Java
• All archives work on all main templates
• Archives are up-to-date (< 1 year)