Skip to content

QA Tools & Framework

lanken edited this page Mar 9, 2013 · 12 revisions

Testing (also known as Quality Assurance or QA) is an essential part of the software development process. While testing intermediate versions of products being developed, the Magento quality assurance team (QA team) needs to execute a number of tests. In addition, prior to publishing each new version of the Magento platform, it is mandatory that the version pass through a set of “regression” and “smoke” tests. These tests are standard for every new version of Magento products, and therefore can be automated to save human resources and time required for executing them.

The benefits of using automated testing are:

  • Simplified testing procedures which use existing automated tests
  • Reduced test execution time and human resource requirements
  • Complete control over the tests’ results (“actual results” vs. “expected results”)
  • Freedom to quickly change a test’s preconditions and input data, and re-run tests dynamically

What is MTAF?

The Magento Test Automation Framework (MTAF) is a system of software tools used for running repeatable functional tests against the Magento application being tested as well as other web-based applications.

MTAF is used for both writing test automation scripts and for performing the actual testing. Test automation scripts created within the framework can be used for testing most Magento functionality which does not relate to an external system. This is a cross-platform solution, so it does not depend on a specific operating system. MTAF allows QA specialists to quickly develop all kinds of tests for the current Magento version, and the tests can be reused at any time. Framework users can run a single test independently, a bunch of tests together (a test suite), or all available tests

How it works

How MTAF  works

1) Client / driver establishes connection with the Selenium RC

2) Selenium RC launches the browser (or reuses the old one) with URL that injects Selenium core's JavaScript into browser-loaded web page.

3) Client / driver passes a Selenese command to the server e.g.: open command

4) Server interprets the command and then triggers the corresponding JavaScript execution to execute that command within the browser (say open page in app under tests)

5) Requests to open the page is routed through proxy server

6) Proxy forwards request to app server

7) App server returns response.

MTAF Logical Structure

The following illustration explains the basics of MTAF logic.

MTAF Logical Structure

A UIMap is a concept for defining, storing, and serving UI elements of an application or a website. The UIMap file (YAML file) contains a set of ‘key-value’ pairs, where each key is the alias of a UI element, and the corresponding value is an Xpath locator. In terms of the UIMap, each page rendered by a browser is presented as a Document Object Model (DOM). Such a model includes every single UI element that can exist on a page. Xpath locators are paths to such UI element in a DOM model. Thus, using Xpath expressions, it is possible to link any UI element on a page (even a hidden one) with a method that imitates an action being executed upon it (a method in the MTAF library). For example, one can use UIMaps (Xpath expressions) and a “click” method from the MTAF library to reproduce clicking any UI element on the specified web page. An example of using such an approach can be the re-testing of the localized application.

A DataSet is a pool of YAML files (data storage) which describe the data main-space being referred to by the test case files. For search convenience purposes the data main-space is logically split into several data files. Each data file can be related to a specific functionality being tested (and thus, related to specific test case files).

A Test is, generally speaking, a single action or sequence of actions that evaluates whether a specific feature meets functional requirements. It can be said that a test is a method in a class which is inherited from the basic test case class. The test calls a library function (or functions), which in turn executes the actions required to check the application's expected business logic.

MTAF File Structure

As you can see from the image, all MTAF files are located in five main directories: config, fixture, framework, testsuite and var. MTAF File Structure

Preparing the Environment for Using MTAF

For the Magento Test Automation Framework to function properly, the following software must be installed and configured:

  • PHP 5.2.0 or later
  • PHPUnit 3.6.8 or later
  • Java Run-time Environment (JRE) 1.6 or later
  • Selenium Remote Control (RC) 1.0.3 or later, 2.0 rc2
  • TortoiseGIT (recommended, but optional)
  • Magento Community Edition 1.5 or later
  • Web browsers (Mozilla Firefox 3.x or later, Google Chrome, Internet Explorer, Apple Safari 2.x) Before installing and configuring the required software, ensure that the environment meets all the system requirements for Magento CE..
  • Working in IDE is recommended (PhpStorm, NetBeans 6.9.1, Zend, Eclipse, etc., will suffice)


1) Install and configure MySQL Server.

2) Install and configure Apache WEB Server 2.2.x

3) Install PHP.

4) Upgrade PEAR pear upgrade. If it Windows, don't forget to add environmental variables:

Additional System Variables
PHP_PEAR_DOC_DIR — C:\Programs\PHP\PEAR\docs
PHP_PEAR_PHP_BIN — C:\Programs\PHP\php.exe
PHP_PEAR_TEST_DIR — C:\Programs\PHP\PEAR\tests

5) Install PHPUnit

pear channel-discover
pear channel-discover
pear channel-discover
pear install phpunit/PHPUnit
pear install phpunit/PHPUnit_Selenium
pear install phpunit/DbUnit
pear install symfony/YAML

Note If you are getting an error no releases available for package or similar while your PHPUnit or PHPUnit_Selenium installation try to update pear channels with:

pear update-channels
...or to clear the pear cache:
pear clear-cache

6) Download Selenium Server 2.x (formerly Selenium RC Server) from the Selenium Downloads page, unpack it, copy to desired location and run

java -jar selenium-server.jar

7) Configuring Mozilla FireFox. It is strongly recommended that you create a pre-configured Firefox profile for running Selenium tests. To run selenium with specified firefox profile is used:

java -jar selenium-server.jar -firefoxProfileTemplate “<Selenium Profile Directory>”

It can also be helpful to do the following: Install useful add-ons (FireBug, Selenium IDE, ScreenGrab, Web Developer, Remember Certificate Exception). Accept SSL-certificates for working websites before using.

8) Selenium Server Configuration. Before you start using the MTAF, you need to properly configure the Selenium Server and the Selenium Client. To configure the necessary Selenium Server settings, simply run the proper set of commands in the command line. For example, it might be similar to the following:

java -jar selenium-server-standalone-2.4.0.jar -trustAllSSLCertificate

Where trustAllSSLCertificate specifies overriding all https warnings.

Selenium Client Configuration

To configure the Selenium Client, you should use either the config.yml file or the local.yml file. Both files are located in the config sub-folder of your local MTAF installation directory (for example, ..\magento\taf\config\config.yml).

Warning Be aware that the config.yml file (the original configuration template) is always under version control. This means that if you upload a new MTAF installation from the version controlrepository, all the settings you have previously made in this file will be overridden. On the other hand, if you commit changes made in the config.yml file to the repository, it may later affect other MTAF users. Therefore, it is strongly recommended to make local configuration changes ONLY within the local.yml file. The local.yml file is similar to the config.yml file and, when it exists, is used in its place. Moreover, this file is not managed by version control, so you can always perform your own configuration settings without effecting the original configuration template.

Browsers lists and configures the browsers that can be used to run both Magento client and admin branches.

  • browser: '*chrome' is the browser's identification accepted by Selenium RC. Keep in mind that the operating system, not Magento, is set up for a specific 'default' browser.
  • host is the identification of the machine where Selenium RC is installed.
  • port is used when one needs to support several servers on a single host, and thus needs to use different ports.
  • default identifies the default browser to be used for running a test or suite of tests. This 'browser identification' option is used only to specify the required browser for a single test run. Thus, you do not need to specify other browsers, simply classify the necessary one.


Selenium RC can work on remote computers, which means that the locations of the real tests, the Magento version being tested, and the testing environment are irrelevant . For example, it may happen that the three mentioned instances are located on three different virtual machines. Even so, if you properly configure the settings such as frontendUrl, adminUrl, adminLogin, adminPassword and browsers, the MTAF engine will work properly.

PHPUnit Configuration

phpunit.xml contains the list of test scripts (test cases) to be run. (..\magento\taf\phpunit.xml)

phpunit.xml is the standard PHPUnit configuration file. The file defines a test suite (the list of test scripts/test cases) to be run. In other words, it contains the suite’s name, a list of suffixes, and a list of directories that the necessary PHP files will be run from. For example:

        <testsuite name="All Tests">
            <directory suffix="Test.php">tests</directory>
        <log type="coverage-html" target="./tmp/report" charset="UTF-8" yui="true" highlight="false"
         lowUpperBound="35" highLowerBound="70"/>
        <log type="coverage-xml" target="./tmp/coverage.xml"/>
        <log type="graphviz" target="./tmp/"/>
        <log type="json" target="./tmp/logfile.json"/>
        <log type="metrics-xml" target="./tmp/metrics.xml"/>
        <log type="plain" target="./tmp/logfile.txt"/>
        <log type="pmd-xml" target="./tmp/pmd.xml" cpdMinLines="5" cpdMinMatches="70"/>
        <log type="tap" target="./tmp/logfile.tap" logIncompleteSkipped="true"/>
        <log type="junit" target="./tmp/logfile.xml" logIncompleteSkipped="false"/>
        <log type="testdox-html" target="./tmp/testdox.html"/>
        <log type="testdox-text" target="./tmp/testdox.txt"/>


  • is the name of the suite to be run
  • suffix="Test.php" is the suffix
  • tests is the name of the directory from which the tests will run F Based on these specific settings in the configuration file, the suite to be run will include all PHP files (tests) from the tests directory which have the Test.php suffix. If there are no files with the specified suffix, only the mentioned file will be run as a test.
Something went wrong with that request. Please try again.