Architectural enhancements (teamengine v5)

This enhancement proposal suggests an architecture refactoring and the adoption of a standard test reporting language.

Separate report presentation from test execution

Split the current web application into two components:

• a Testing Facility that:

  • uses the Test Execution Service
  • manages user test sessions (in a database rather than filesystem?)
  • parses the EARL results and generates human-readable reports (e.g. HTML5, PDF) to end users

• a Test Execution Service (TES) that provides a RESTful API for invoking (and managing) test suites

The testing facility would be a client of the TES, of course, but other software agents (including mobile clients) could do the same. The following table shows the types of resources exposed by the TES at the indicated endpoints.

TES endpoints

Path	Resource	Method(s)
/	Service description	GET
/ets	List of available test suites	GET
/ets/{ets-code}/{ets-version}	Test suite description	GET
/ets/{ets-code}/{ets-version}/run	Test run results	GET, POST

Adopt a standard test results language

I recommend adopting the W3C Evaluation and Report Language (EARL) 1.0 Schema, an RDF vocabulary for describing test results:

• Evaluation and Report Language (EARL) 1.0 Schema
• Developer Guide for Evaluation and Report Language (EARL) 1.0

The existing working draft hasn't been updated in a while, but it looks like the Evaluation and Repair Tools Working Group (ERT WG) is seeking to renew its charter. Note that advancing EARL to Recommendation status is one of its goals. The OGC could be an early adopter ;-)

Most test runners have their own custom result formats (usually based on XML), so it would be necessary to transform these to EARL test results (RDF/XML syntax). For example, using TestNG one could implement a custom listener or reporter to do this. The Validator front-end would then take these raw test results and produce something easy on the eyes for human readers :-)

The essential structure of the response message from the TES is shown below, where the enclosed entity is an RDF/XML representation of the test results.

HTTP/1.1 200 OK
Content-Type: application/rdf+xml
Content-Length: NNNNN

<?xml version="1.0" encoding="UTF-8"?>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:earl="http://www.w3.org/ns/earl#"
         xmlns:http="http://www.w3.org/2011/http#
         xmlns:ptr="http://www.w3.org/2009/pointers#"      
         xmlns:dct="http://purl.org/dc/terms/">

    <!-- EARL statements -->

</rd:RDF>

See sample EARL results.

Implementation notes

• Implement a CTLExecutor (use extension mechanism)
• Transform CTL results to EARL results
• Implement a custom TestNG listener (based on TestListenerAdapter) to create EARL results; use Apache Jena
• Generate HTML5 report from EARL results (RDF/XML)
• upgrade REST implementation to JAX-RS 2.0 (Jersey 2.x)