AUTOMATED ROBUSTNESS TESTING AT THE UI LEVEL:
START TESTING INMEDIATELY: With TESTAR, you can start testing immediately!! TESTAR automatically generates and executes test sequences based on a structure that is automatically derived from the UI through the accessibility API. TESTAR can detect the violation of general-purpose system requirements through implicit oracles.
BUILD REQUIREMENTS INCREMENTALLY: While your application is automatically being tested for stability, crashes and undesired outputs, you can start adding more and more oracles that test more specific requirements of our application. This way we incrementally create the requirements, this is something that turns out to be very helpful when dealing with legacy systems.
NO TEST SCRIPTS: No test scripts are recorded!, so no test script maintenance! Tests are generated and executed on the fly! When your UI changes, so do your tests!
LICENSE: TESTAR is distributed FREE of charge as an open source project under the BSD-3 license
TESTAR v2.0.x is under development. It will work on Window 10. You can clone/download the latest version from our development branch: https://github.com/TESTARtool/TESTAR_dev
The software can be build with both ant and gradle.
Required tools to build the software
In order to build the native code, a view manual steps need to be executed;
- In order to build the windows native code, Nmake and the compile for Microsoft visual studio are required. These tools can be downloaded using the following link.
- Install the Visual Studio tools on your machine (remember the path where the tools are installed)
- Download compile_w10.bat and clean_w10.bat
- Copy clean.bat and compile.bat to the folder windows/native_src within the TESTAR project
- Adapt compile.bat and clean.bat. Set PATH to the installation folder used in step 2. CALL "C:<PATH>\2017\BuildTools\Common7\Tools\VsDevCmd.bat" -arch=x64
It should be possible to build the project using the instructions provided in the next section
To build the Gradle project and run it execute the following procedure;
.\gradlew installDistin the root of the project
- Change directory to testar/target/install/bin
- Run testar.bat
It is also possible to generate a zip file containing TESTAR. This zip can be extracted on any other machine that has a 64-bit Windows operating system and Visual Studio redistributable installed. A proper way of using TESTAR is to run the tool in a virtual-machine. To build the zip execute the following command.
.\gradlew distZipin the root of the project.
- Extract the zip on the machine where TESTAR is used.
NOTE: TESTAR requires Visual Redistributable which can be downloaded from the following link .Also a JAVA 1.8 JDK is required.
Running Gradle in Eclipse
The following procedure has been performed
- Create a new empty workspace for Eclipse in a folder which is not the folder that contains the source code.
- Select File -> Import to open the import dialog
- Select Gradle -> Existing Gradle project to open te import dialog
- Select the folder that contains the root of the source code and start the import
Running TESTAR from Gradle
TESTAR can be started using a gradle command from the root of the project.
- .\gradlew runTestar
Within Eclipse, TESTAR can be executed by running the task runTestar which is available in the map custom_testar. To debug the application with the runTestar task, provide your onw run configuration in which the option -DDEBUG is set.
Debug TESTAR from Gradle
In order to debug the TESTAR code, you must run;
- .\gradlew -DDEBUG=true runTestar.
Optionally you can build TESTAR (.\gradlew -DDBEBUG=true distZip ), copy the result to the machine where you want to run TESTAR and run TESTAR on the target machine. This allows the user to debug TESTAR from a different machine.
Running TESTAR binaries (obtained with gradlew build) from command line
TESTAR allow its execution and settings configuration from the command line. By default is executed with the selected protocol (.sse file) and the test.settings values of that protocol.
From the command line it is also possible to select the desired protocol to execute TESTAR and change the values of the test.settings.
The protocol to be executed can be selected using the "sse" parameter next to the name of the desired protocol. Ex: testar sse=desktop_generic
Other settings are input using the pairs "parameterX=valueX" separated by space. Ex: testar ShowVisualSettingsDialogOnStartup=false Mode=Generate
Certain characters such the slashes or the quotation marks must be entered in a double way to respect the treaty of special characters.
Some of the most interesting parameters that can help to integrate TESTAR as an CI tool are:
sse -> to change the protocol ShowVisualSettingsDialogOnStartup -> To run TESTAR without the GUI Mode -> TESTAR execution Mode (Spy, Generate, Record, Replay, View) SUTConnector & SUTConnectorValue -> The way to link with the desired application to be tested Sequences & SequenceLength -> The number of iterations and actions that TESTAR will execute SuspiciousTitles -> The errors that TESTAR will search in the execution
Example: testar sse=desktop_generic ShowVisualSettingsDialogOnStartup=false Sequences=5 SequenceLength=100 Mode=Generate SUTConnectorValue=" ""C:\Program Files\VideoLAN\VLC\vlc.exe"" " SuspiciousTitles=".[eE]rror.|.[eE]xcep[ct]ion."
- Currently, only the protocols
accessibility_wcag2ictsupport the graph database. Other protocols need to be adapted (see issue #52)
Graph database support
This version of TESTAR supports the output of data to an http://orientdb.com database. Currently Widget's, Actions, States and there relation are recorded to the graph database.
Restrictions on the use of the graph database
The model currently implemented consists of States, Actions and Widgets. The widgets are stored from
the method deriveActions. The protocols
contain an implementation which shows how the method storeWidget could be used.
When the storeWidget method is not called, TESTAR will raise an exception and stop working.
using graphdb on the local filesystem
The easiest way to use the graph database support is to write the database to the local file system After starting TESTAR, it's required to configure the database settings in the tab "GraphDB".
- url : plocal:<path> (for instance plocal:output/testdb)
- user: admin
- password: admin
- Check the checkbox.
using graphdb with orientdb server
In order to use the graphdb feature it's advised to install a graph database on your machine The current implementation of TESTAR has a backend for Orientdb. You can download the community edition from orientdb. Follow the installation instructions to install the database on your machine.
When orientdb is started the first time. The root password needs to be configured. Make sure you remember thes password.
In order to use the graphdb feature. A database must be created in Orientdb. To do this follow the following procedure;
- Start the database server (ORIENTDB_HOME/bin/server.bat)
- Start orientdb studio in a webbrowser http://localhost:2480
- Choose "New DB" and provide the name, root user and password. (The database will also get a default admin/adimin user/password).
After starting TESTAR, it's required to configure the database settings in the tab "GraphDB".
- url : remote:/<database name> (for instance remote:/localhost/demo)
- user: admin
- password: admin
- Check the checkbox.
When TESTAR finishes, the data be inspected in the database. The easiest way to see the complete graph is to type the following query in the graph tab "Select * From E". This will display the complete graph.
Requirements for a protocol when using GraphDB.
A part of the interaction with the graph database occurs in the method deriveActions. In this method, the available widgets are stored as they are derived within the protocol. When using the graph database extension, the user needs to be aware the widgets are stored. When this is not done, the model in the database is incomplete. A sample can be found in the protocol desktop_generic_graphdb.java.
Exploring the graph database with Gremlin.
Gremlin is a graph traversel engine which can be used to query a graph database. OrientDB supports Gremlin and provides an implementation of the traversal engine in it's community release.
To start Gremlin run the following command; orientdb-gremlin. This command is delivered with the community edition of Orientdb.
See the manuals of orientdb to learn about the possibilities to query the model.