-
Notifications
You must be signed in to change notification settings - Fork 0
Snowballing: Data Extraction
Yury Alencar Lima edited this page Sep 16, 2019
·
90 revisions
To Extract data of the studies was used data extraction form present in [Data Extraction Form].
| Data | Response |
|---|---|
| Title of Study | Is business domain language support beneficial for creating test case specifications. |
| Authors | Florian Häser, Michael Felderer, Ruth Breu. |
| Year of publication | 2016. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | This study was conducted only graduate students with business domain familiar to the students. |
| DSL Name | This data not is presented, only are DSL represents the business rules. |
| Use or propose a DSL? | Use. |
| DSL Features | Others DSLs: Implements a minimalistic for BDD, how keywords, 'given', 'when', and 'then' to express tests in the form 'Given [initial context], when [event occurs], then[ensure some outcomes]'. The DSL includes domain concepts from various domains. When designing the language we have included domain concepts from the execution environment, in our case the web browser. The tester is supported by concepts like ‘open’, ‘go to’, ‘fill in’ etc. |
| Which DSL focus domain? | This DSL presented is for Web applications, but several DSLs are presented. |
| Technologies and Notations used in DSL development | This data not is presented in this study. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | This study not presented this data. |
| System Under Testing (SUT) data are represented in the models and how they are represented | The first DSL: The supported concepts, include the keywords ‘story’, ‘in order to’, ‘as a’, ‘I want to’, ‘scenario’, ‘given’, ‘and’, ‘when’, and ‘then’. It allows the tester to express tests in the form ‘Given [initial context], when [event occurs], then [ensure some outcomes]’ including a description, postcondition and a requirement that is to be verified. Besides the keywords, the tester is allowed to write everything in the form of free text. The second DSL: created by authors for the experiment, the DSL only consists of a ‘step’ (denoted by ‘–’) and a ‘check’ step language concept. Limiting the DSL has no impact on the expressibility of the test scenario. Only the support a tester gets by the tool environment when creating the test is limited. The final DSL presented: SUT data are inserted using the concepts like 'open', 'go to' with presented in DSL features. |
| Benefits in using DSL | The final DSL: The inclusion of business domain concepts allows significantly faster creation of test case specifications, while document length and quality are not significantly affected. |
| Drawbacks in using DSL | This data not is presented in this study. |
| Trends in using DSL | The integration of business domain concepts to implement ubiquitous language features is a promising issue. |
| Others comments about study | This article is an experiment using several DSLs for acceptance tests. This paper presents an empirical study based on a controlled student experiment that analyzes the perceived quality, the creation time and length of test case specifications, as well as support by the language as perceived by testers. |
| Data | Response |
|---|---|
| Title of Study | Automated Test Case Generation for Web Applications from a Domain Specific Model. |
| Author | Arne-Michael Torsel. |
| Year of publication | 2011. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | Not contains a evaluate of the DSL. |
| DSL Name | This study not presents this data. |
| Use or propose a DSL? | This article PROPOSE a prototype. |
| DSL Features | The generation of test oracles from model information and the transformation of abstract test cases to executable test scripts. |
| Which DSL focus domain? | Web Applications. |
| Technologies and Notations used in DSL development | Using the Eclipse Xtext framework, an editor was generated from the meta-model that enables comfortable model editing in the common Eclipse environment. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | Generator algorithm to systematically traverse to create test sequences and measure and reach structural coverage. Initially, the model parser constructs a directed graph structure of the model views (vertices) and transitions (edges), the application graph. An extended breadth-first search (BFS) is used to explore that graph from the designated start view. The start view is typically the ’ home page’ of the web application. In every vertex, all paths leading to it from the start view using either only the BFS spanning tree edges or paths ending with one ’cross edge’ are stored. This avoids having to repeatedly search the graph for path alternatives in the following steps. This study presents only an overview of a subsequent publication will contain a more detailed description of the algorithm. |
| System Under Testing (SUT) data are represented in the models and how they are represented | The model must reflect the application structure, that is web pages and navigational connections between them. |
| Benefits in using DSL | Xpand, and Xtext is based on the Eclipse Modeling Framework (EMF), a graphical editor could be implemented relatively easy as well using the Eclipse Graphical Modelling Project (GMP). |
| Drawbacks in using DSL | Modern web applications increasingly employ the AJAX technology, which, currently, is not fully supported by the proposed approach. |
| Trends in using DSL | Model-based testing is a promising technique for test case design that is used in an increasing number of application domains. |
| Others comments about study | For test case generation, the algorithm selects paths in the graph to be executed, which means that the assignments of the transitions on the path are evaluated to update the involved model variable values. For path selection, the generator algorithm employs a greedy algorithm to select paths with a maximum amount of views and transitions not covered yet. The algorithm must not pick a path for execution that would block the usage of other still unvisited transitions. Using the dependency graph, paths with transitions that would be blocked by an assignment executed on a transition of the picked path can be determined. This step mostly determines the runtime complexity of the whole algorithm, as it requires partial simulation and permutation of path alternatives.Deriving meaningful test oracles is one of the major challenges in model-based testing caused by the need for abstraction [8], [9]. In the proposed approach, test oracles are generated using text expressions at the model views that may contain model variables or test data. In this way, the test oracles can check for functional properties of the application that are observable in the HTML source text of the web application pages. Whenever a view is reached by a transition during test case generation, verification steps for all the text expressions of this view are created. |
| Data | Response |
|---|---|
| Title of Study | Lessons learnt from using DSLs for automated software testing. |
| Authors | Mark Micallef, Christian Colombo. |
| Year of publication | 2015. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | This article presents all the data. |
| DSL Name | Gherkin, Not Present because the other DSLs are most specifics. |
| Use or propose a DSL? | Use. |
| DSL Features | Gherkin - provides constructs for defining a number of scenarios which in turn consist of a number of steps, each being either a precondition to the scenario (Given) followed by a stimulus to the system (When) and finally the postcondition (Then). The study also presents other DSLs, one for the domain: eCommerce, Android, and Games Applications |
| Which DSL focus domain? | Web Applications (Gherkin), Others DSLs: eCommerce, Android and games applications. |
| Technologies and Notations used in DSL development | Others DSLs - Xtext, in DSL e-commerce and Android, are used: JUnit, SelenDroid and WebDriver. Game DSL: Sikuli API for interactions |
| DSL type | Textual (Gherkin). |
| Technique/approach/method to ensure coverage of the system with test cases | The Developers then write code which translates each step into interaction with the system (Gherkin). |
| System Under Testing (SUT) data are represented in the models and how they are represented | Use Given-When-Then notation (Gherkin). |
| Benefits in using DSL | The Gherkin language is very flexible in that it only requires steps to start with one of the given-when-then keywords. Reusability: All developed languages provided reusability support in the form of allowing users of the language to group sequences of language phrases into procedures. |
| Drawbacks in using DSL | The Gherkin presented problems when the number of test scenarios began to grow. Due to the fact that Cucumber provides a very loose grammar, the language grew organically with people adding their own versions of notions as they went along. This resulted in substantial duplication as people would express the same notions in different ways. DSL for gaming, because every game is different. DSL for e-commerce, some e-commerce has very specific features that have been seen as difficulties. |
| Trends in using DSL | More structured languages or improve one's own Gherkin Language, to make it more structured.. |
| Others comments about study | These partners in the study were interested in setting up automated testing frameworks, and the authors were mainly interested in observing the challenges encountered when using Gherkin to develop a domain specific language. For subsequently investigate the possibility of using more rigorously defined languages for specifying automated tests through three case studies: (1) Android applications, (2) eCommerce websites and (3) graphics-based games. |
| Data | Response |
|---|---|
| Title of Study | Pattern Based GUI Testing for Mobile Applications. |
| Authors | Pedro Costa, Miguel Nabuco, Ana C. R. Paiva. |
| Year of publication | 2014. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | This study uses the DSL only one mobile application for test. |
| DSL Name | PARADIGM-DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | A domain specific language (DSL) for building GUI test models based on UI test patterns (UITP). |
| Which DSL focus domain? | PGBT is architecture used for Web applications, this study presents a purpose of the DSL for test Mobile Applications. |
| Technologies and Notations used in DSL development | This study does not present this data. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | This study presents the test scripts generate but not how. |
| System Under Testing (SUT) data are represented in the models and how they are represented | The list of UITPs supported by PARADIGM-DSL: Login, Find, Sort, Master Detail, input, Call (contains types). |
| Benefits in using DSL | Code generate, generate a report and use into web applications (The focus this study is to verify your use into mobile applications). |
| Drawbacks in using DSL | DSL does not support some interactions in applications that are different. In addition to adapting to different layouts and different screen sizes. |
| Trends in using DSL | More support into mobile interactions with "longpress" action and other actions. |
| Others comments about study | This study presents the PGBT an architecture used to create web and mobile tests. |
| Data | Response |
|---|---|
| Title of Study | A Pattern-Based Approach for GUI Modeling and Testing. |
| Authors | Rodrigo M. L. M. Moreira, Ana C. R. Paiva, Atif Memon. |
| Year of publication | 2013. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | This study not validate the DSL in all platforms. |
| DSL Name | PARADIGM-DSL. |
| Use or propose a DSL? | Propose. |
| DSL Features | Create models of the systems, generate test scripts with coverage. |
| Which DSL focus domain? | Cross-platform (e.g., web, mobile and desktop). |
| Technologies and Notations used in DSL development | Eclipse Modeling Framework (EMF). |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | This study presents a method of coverage by means of calculations. Generally, the algorithm traverses the model elements considering the connectors along the way. Elements within the model can be optional. When an element is optional, it means that there will be test cases that skip the corresponding test strategy. Group elements have a particularity: the inner elements can be executed arbitrarily. Regarding test case generation, it means that there will be test cases for different permutations of those elements. |
| System Under Testing (SUT) data are represented in the models and how they are represented | Init (to mark the beginning of a model), End (to mark the termination of a model), Structural (to structure the models in different levels of abstraction), and Behavioral (UI Test Patterns describing the behavior to test). |
| Benefits in using DSL | This testing approach is based on a DSL language, called PARADIGM, which aims to simplify the modeling process by promoting reuse. |
| Drawbacks in using DSL | The study did register some complaints about the usability of the modeling environment. They complained that some configurations, in particular, the configuration of the Find UI Test Pattern, demanded too many steps, which became boring. They also had difficulty in modeling a situation in which a B element appears after an optional A element, but that should only be run when A is executed. |
| Trends in using DSL | Increase the expressiveness of the DSL language. |
| Others comments about study | This study proposes a PARADIGM DSL, but not evaluate. |
| Data | Response |
|---|---|
| Title of Study | PARADIGM-COV: A multimensional test coverage analysis tool. |
| Authors | Liliana Vilela, Ana C. R. Paiva. |
| Year of publication | 2014. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | Not presents. |
| DSL Name | PARADIGM-DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | This DSL use UITP, a UITP is a UI Test Patterns for example a form. Other feature are generate test scripts. |
| Which DSL focus domain? | This study presents the PARADIGM-DSL in Web Applications. |
| Technologies and Notations used in DSL development | This data not is presented. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | Does not show how, only DSL provides coverage using pre-defined settings. |
| System Under Testing (SUT) data are represented in the models and how they are represented | This DSL has structural elements (to structure the model in different levels of abstraction – Form element –, and to define a set of elements that can be exercised in any order – Group element); behavioral elements (UI Test Patterns); Init and End elements (to mark the start and end points of a model); and connectors, that establish relations among elements and define their sequencing. |
| Benefits in using DSL | Not present this data. |
| Drawbacks in using DSL | Not present this data. |
| Trends in using DSL | Not present this data. |
| Others comments about study | This study focus is PARADIGM-COV a coverage analysis tool for tests created by PARADIGM-DSL. |
| Data | Response |
|---|---|
| Title of Study | Pattern Based GUI Testing Modeling Environment. |
| Authors | Tiago Monteiro, Ana C.R. Paiva. |
| Year of publication | 2013. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | This study present all data. |
| DSL Name | PARADIGM-DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | This DSL use UITP, a UITP is a UI Test Patterns for example a form. Other feature are generate test scripts. |
| Which DSL focus domain? | Not present a specific domain, only UI Tests. |
| Technologies and Notations used in DSL development | Eclipse Graphical Modeling Framework (GMF). |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | This DSL not present in details, only use a recursive algorithm for define paths using the model. |
| System Under Testing (SUT) data are represented in the models and how they are represented | This DSL has structural elements (to structure the model in different levels of abstraction – Form element –, and to define a set of elements that can be exercised in any order – Group element); behavioral elements (UI Test Patterns); Init and End elements (to mark the start and end points of a model); and connectors, that establish relations among elements and define their sequencing. |
| Benefits in using DSL | Not present, because this focus is a environment not the DSL. |
| Drawbacks in using DSL | Not present, because this focus is a environment not the DSL. |
| Trends in using DSL | Includes new elements and new test case generation algorithms. |
| Others comments about study | The study focus is the environment not the DSL. |
| Data | Response |
|---|---|
| Title of Study | A GUI modeling DSL for pattern-based GUI testing PARADIGM. |
| Authors | Rodrigo M. L. M. Moreira and Ana C. R. Paiva. |
| Year of publication | 2014. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | PARADIGM-DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | UI Test Pattern configuration, test cases generation from models, test case execution and test coverage information. |
| Which DSL focus domain? | Not present explicitly, but in evaluation this DSL is used for web, C# and others applications not must specified. |
| Technologies and Notations used in DSL development | Eclipse Graphical Modeling Framework (GMF). |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | This data not is detailed in this study. |
| System Under Testing (SUT) data are represented in the models and how they are represented | This DSL has structural elements (to structure the model in different levels of abstraction – Form element –, and to define a set of elements that can be exercised in any order – Group element); behavioral elements (UI Test Patterns); Init and End elements (to mark the start and end points of a model); and connectors, that establish relations among elements and define their sequencing. |
| Benefits in using DSL | Diminished effort in modeling GUIs with PARADIGM. Furthermore, the authors also evaluated the DSL at the industrial level and the feedback was promising. |
| Drawbacks in using DSL | Not presented. |
| Trends in using DSL | Not presented. |
| Others comments about study | This DSL is based into VAN4GUIM a UML Profile proposed by authors. |
| Data | Response |
|---|---|
| Title of Study | Model-Based Test Case Generation for Web Applications. |
| Authors | Miguel Nabuco, Ana C. R. Paiva. |
| Year of publication | 2014. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | PARADIGM-DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | Generate Test Cases and configuration, Filter Paths, Path Tree. |
| Which DSL focus domain? | PARADIGM is used to test web and mobile applications, but it can be expanded to test desktop applications. |
| Technologies and Notations used in DSL development | Not present. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | Configure a Path Generator, Path Filters using: Cycles, Mandatory and Exclusion Elements, Random. Also contains path generation modes: How all invalids Strategy, Default Strategy and Random Strategy. |
| System Under Testing (SUT) data are represented in the models and how they are represented | Each UI pattern instance may have multiple configurations (for example, for a login test pattern this includes the set of usernames and passwords used). |
| Benefits in using DSL | PARADIGM proved to be effective, as the filtering can provide better coverage, finding more bugs and killing more mutants. |
| Drawbacks in using DSL | To build and configure a PARADIGM model, it takes more effort than compared to technologies already used as Selenium. |
| Trends in using DSL | The features planned for the near future include a broader set of filtering options, test strategies, and test coverage statistics. |
| Others comments about study | . |
A41 - Natural-language scenario descriptions for testing core language models of domain-specific languages (2014)
| Data | Response |
|---|---|
| Title of Study | Natural-language scenario descriptions for testing core language models of domain-specific languages. |
| Authors | Bernhard Hois, Stefan Sobernig, Mark Strembeck. |
| Year of publication | 2014. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | Natural-language Requirements - DSL. |
| Use or propose a DSL? | Propose. |
| DSL Features | Specify natural-language requirements via scenarios and generate test cases executable. |
| Which DSL focus domain? | This focus is a DSMLs or DSLs. |
| Technologies and Notations used in DSL development | Eclipse Xtext. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | If each requirement-level scenario is checked via one (or more) executable test scenario(s), critical test coverage of the most relevant requirements can be achieved. The scenario tests are reviewed by the domain expert and the DSML engineer. This ensures that the executable scenario descriptions reflect the requirements sufficiently. |
| System Under Testing (SUT) data are represented in the models and how they are represented | The domain expert is able to define scenario-based requirements via natural-language statements. The syntax rules allow for using synonyms for steps (e.g., And, But). This way, the domain expert can, on the one hand, phrase requirements in a natural and readable way and, on the other hand, concatenate multiple steps into composite statements (i.e., adding multiple steps to each Given , When , or Then section). |
| Benefits in using DSL | A benefit of this work is its design for reuse. Step definitions provide a mapping convention for translating natural-language requirements into executable test scenarios. These mapping conventions are separated from the transformation routines. In order to provide for further scenario-based DSML core language model tests, the transformation routines do not change (as they are only dependent on the requirements specification language). |
| Drawbacks in using DSL | Not cover the metamodel development. |
| Trends in using DSL | Cover (iterative) metamodel development which differs from the coupled metamodel evolution under DSML integration. |
| Others comments about study | This study presents a DSML for test a DSML. |
| Data | Response |
|---|---|
| Title of Study | Towards Domain-Specific Testing Languages for Software-as-a-Service. |
| Authors | Dionny Santiago, Adam Cando, Cody Mack, Gabriel Núñez, Troy Thomas, Tariq M. King. |
| Year of publication | 2013. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented in this study. |
| DSL Name | Legend. |
| Use or propose a DSL? | Propose. |
| DSL Features | The DSL allows us to provide test case editing, execution, and debugging tools tailored for domain experts, test engineers, and end users. Generate Test Scripts. |
| Which DSL focus domain? | The DSL focus on web-based applications which are deployed on cloud computing platforms and Software-as-a-Service. |
| Technologies and Notations used in DSL development | Legend has been developed in C# as a Visual Studio (VS) extension. The VS 2010 SDK provides components for extending the VS Editor with a custom language service. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | This DSL uses Echo Framework to generate test scripts. In order to support debugging at multiple levels of abstraction, the EchoGenerator has two modes of generation: Script – generates test scripts written in the Echo syntax, and Assembly – generates a Common Intermediate Language (CIL) representation of the Echo test. Our prototype maps the CIL to the domain-specific test steps. A test can, therefore, be executed and debugged at the level of the domain-specific test language, or the script language of the underlying Echo testing framework. |
| System Under Testing (SUT) data are represented in the models and how they are represented | At the core of the language is a set of abstract test commands. There are two types of commands: Action Commands and Assertion Commands (left of Figure 1). Action commands apply inputs that exercise the system under test (SUT). This includes stimulating UI controls such as text boxes, dropdowns, and buttons, as well as database-related actions. On the other hand, assertion commands perform UI and database checks to verify the behavior of the SUT. User Interface: This model is a generalization of the user interface of the SUT. Macros: Testers can define frequently used test setup, input, or assertion command sequences, and store them in a central location. These macros are then named using domain-specific terms, and integrated into our testing language. Similar to our abstract test commands, test macros can target user or database interactions. |
| Benefits in using DSL | A major benefit of Legend is the ease with which test cases can be specified and reviewed by nontechnical users. The tool can, therefore, be leveraged by domain experts and end users during acceptance testing. It also allows these stakeholders to assist in debugging issues using a language they understand, and without being concerned with the low-level implementation details of the test automation. Since tests are specified in an English-like syntax, using Legend could reduce or eliminate the need to maintain a separate inventory of test documentation. |
| Drawbacks in using DSL | A limitation of the current prototype is the lack of an externalized point of extension for the test commands and their syntax. Since domain experts, testers and developers from several teams will be defining new model elements as the application evolves, the authors need to provide a mechanism that allows a new page or control-specific commands to be easily added to the language. Web UI elements with dynamically generated identifiers are also not supported by the prototype, or the underlying Echo testing framework, but are planned for future releases. |
| Trends in using DSL | Not present this data, only require news implementations. |
| Others comments about study | All data are presented. |
| Data | Response |
|---|---|
| Title of Study | Using Built-In Domain-Specific Modeling Support to Guide Model-Based Test Generation. |
| Authors | Teemu Kanstrén, Olli-Pekka Puolitaival. |
| Year of publication | 2012. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented in this study. |
| DSL Name | OSMOTester MBT. |
| Use or propose a DSL? | Propose. |
| DSL Features | Generate test from the generic test model, constrain to generate varied test cases for a specific scenario(s) or manually create specific test cases from the model and, generate test data |
| Which DSL focus domain? | This approach works both for online (direct test execution in generation) and offline (generate scripts and execute later) MBT approaches. |
| Technologies and Notations used in DSL development | OSMOTester MBT tool framework to develop test models using a full (Java) programming language. The OSMOTester toolset provides a framework for creating these test models, and from these test models automatically creates a DSM language that the domain expert can use to guide test generation. (Annotations) |
| DSL type | Textual, in addition the OSMOTester also provides a graphical user interface (GUI) to create these scripts. |
| Technique/approach/method to ensure coverage of the system with test cases | Similarly, different algorithms for traversing the given models can be defined based on the different elements of the model objects. OSMOTester includes the following algorithms: Random: picks a random transition from the ones available, Balancing: randomly picks an available transition but favors less covered ones, Weighted: randomly picks an available transition but gives a higher probability to ones with higher weight. |
| System Under Testing (SUT) data are represented in the models and how they are represented | Together with the domain expert, the language expert defines the generic overall test model. From this model, OSMOTester automatically forms a higher-level DSM language that the domain expert can use to guide the test generation from this model. In DSM terminology, a transformation is applied by OSMOTester based on the constraints defined by the domain expert to produce constrained variants of the test model (TM1, TM2, TM3 in the figure). The OSMOTester test generator then generates test cases from these test model variants. |
| Benefits in using DSL | Reuse of skills, test libraries, IDE integration with debugging, refactoring. |
| Drawbacks in using DSL | As the models are not extensively checked by the tool, they may require some more analysis effort by the user (which is not necessarily a bad thing as it increases the understanding of the system). For example, one has to realize that requiring 10 event removals will never terminate if only 5 event creations are allowed. |
| Trends in using DSL | Specification mining. |
| Others comments about study | This study propose a MBT tool and DSML for design automation tests. |
| Data | Response |
|---|---|
| Title of Study | SeTT: Testing-tool for Measurement System DEWESoft. |
| Authors | TOMAŽ KOS, TOMAŽ KOSAR, MARJAN MERNIK, JURE KNEZ. |
| Year of publication | 2011. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | SeTT-DSL (Sequencer). |
| Use or propose a DSL? | Use. |
| DSL Features | The domain framework is in charge of the entire process during the test case and also controls and supervises the hardware, Sequencer enables domain experts to model their own measurement procedures. |
| Which DSL focus domain? | Measurement procedures. |
| Technologies and Notations used in DSL development | As previously mentioned, the SeTT is included within the product DEWESoft. The whole product is implemented in the Delphi. |
| DSL type | Visual, Textual, or XML. |
| Technique/approach/method to ensure coverage of the system with test cases | No contains. |
| System Under Testing (SUT) data are represented in the models and how they are represented | In models using Asserts. |
| Benefits in using DSL | Sequencer reduces the cost of failures and increases the quality of the measurement system, Accuracy without human influence, elimination of human errors, test repetition, and reusability, significantly faster than the manual tests performed by humans. |
| Drawbacks in using DSL | These authors present that it is necessary to invest in the development of a tool. |
| Trends in using DSL | No contains. |
| Others comments about study | This study presents test for hardware and software (The focus is in Hardware). |
A45 - From DCOM Interfaces to Domain-Specific Modeling Language: A Case Study on the Sequencer (2011)
| Data | Response |
|---|---|
| Title of Study | From DCOM Interfaces to Domain-Specific Modeling Language: A Case Study on the Sequencer. |
| Authors | Tomaž Kos, Tomaž Kosar, Jure Knez, and Marjan Mernik. |
| Year of publication | 2011. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | Sequencer. |
| Use or propose a DSL? | Use. |
| DSL Features | Only are all events related with measurement systems. |
| Which DSL focus domain? | Measurement systems. |
| Technologies and Notations used in DSL development | DLex. |
| DSL type | Graphical and Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | No contains. |
| System Under Testing (SUT) data are represented in the models and how they are represented | In scripts models, the study presents examples of the use. |
| Benefits in using DSL | The main goal of the Sequencer is to push the development of the application from using DCOM objects to a specialized tool that enables domain experts to develop measurement sequences efficiently in a simple manner, without the need of support from programming engineers. Sequences can be developed in a textual or visual mode, which is customized for application development. |
| Drawbacks in using DSL | No contains in this study. |
| Trends in using DSL | From a usability point of view, the Sequencer’s next feature is to record a sequence execution and save it in text format. In this manner, sequences can be analyzed in time to see more details. |
| Others comments about study | Use XML to conversions between types(Graphical and textual). |
| Data | Response |
|---|---|
| Title of Study | The ETSI Test Description Language TDL and its application. |
| Authors | Andreas Ulrich, Sylvia Jell, Anjelika Votintseva, Andres Kull. |
| Year of publication | 2014. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | ETSI-TDL. |
| Use or propose a DSL? | Propose. |
| DSL Features | It allows describing scenarios on a higher abstraction level than programming or scripting languages. Furthermore, TDL can be used as an intermediate representation of tests generated from other sources, e.g. simulators, test case generators, or logs from previous test runs. |
| Which DSL focus domain? | Only presents different application domains. |
| Technologies and Notations used in DSL development | The authors defined the metamodel and implemented using EMF and Xtext. |
| DSL type | Textual and Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | No contains. |
| System Under Testing (SUT) data are represented in the models and how they are represented | The Test Architecture package describes all elements needed to define a test configuration consisting of tester and SUT components, gates, and their interconnections. Connections link two or more gates of different components. A component can act either in the role of a tester or a SUT. |
| Benefits in using DSL | The authors assume that the following application areas will benefit from the proposed homogeneous, standardized approach of test design with TDL: Model-based design of test descriptions derived from the given test objectives, e.g. test purpose specifications (ETSI process), user stories (TDD) or other sources; Representation of test descriptions obtained from other sources, e.g. generated tests (output from test generation tools), system simulators, test execution traces from previous test runs. |
| Drawbacks in using DSL | Being a new notation, there is naturally little tool support that is ready to use. |
| Trends in using DSL | Focus into coverage. |
| Others comments about study | The Foundation package covers the fundamental concepts needed to express the structure and contents of a TDL specification and defines additional elements of a test description such as test objectives, annotations, and comments. The Test Architecture package describes all elements needed to define a test configuration consisting of tester and SUT components, gates, and their interconnections. Connections link two or more gates of different components. A component can act either in the role of a tester or a SUT. The Data package defines the elements needed to express data sets and data instances used in test descriptions. TDL does not feature a complete data type system. Instead, it relies on parameterizable data instances, which serve as surrogates for concrete data types and values outside of TDL. The Test Behaviour package defines all elements needed to describe the structure and behaviour of a test description. It is the most elaborate package to accommodate various behaviour kinds and provides the definitions for interactions and actions. Moreover, it defines a way to link a test description or any of the behavioural elements it contains to test objectives reflecting test purposes or other forms of system requirements to be validated. The Time package defines all elements to express time and operations over time. There are two different concepts in TDL to operate on time: A descriptive way to express time in terms of wait (for the tester) and quiescence (for the SUT) operations; An operational way in terms of timers and operations over timers start, stop, timeout. Additionally, time constraints can be specified between behavioural elements. |
| Data | Response |
|---|---|
| Title of Study | Model-Based API Testing of Apache ZooKeeper. |
| Authors | Cyrille Artho, Quentin Gros, Guillaume Rousset, Kazuaki Banzai, Lei Ma, Takashi Kitamura, Masami Hagiya, Yoshinori Tanabe, Mitsuharu Yamamoto. |
| Year of publication | 2017. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | Modbat. |
| Use or propose a DSL? | Use. |
| DSL Features | Describe system behavior, Enable preconditions, actions and, transitions. In addition results of actions on the SUT can be checked using assertions and support exception scenario, overriding the state with exception. Call API. |
| Which DSL focus domain? | Networked systems. |
| Technologies and Notations used in DSL development | Modbat is based in Scala language. |
| DSL type | data not presented. |
| Technique/approach/method to ensure coverage of the system with test cases | Test cases are derived by exploring available transitions, starting from the initial state (EFMS). A test case continues until a configurable limit is hit or a property is violated. Properties include unexpected exceptions and assertion failures. (SPIN) |
| System Under Testing (SUT) data are represented in the models and how they are represented | This data not is presented in this study. |
| Benefits in using DSL | Modbat was superior to Junit in this type of test.. |
| Drawbacks in using DSL | Not presented. |
| Trends in using DSL | Challenges in the modeling process involve writing a correct test oracle in the presence of concurrent requests on the same data. |
| Others comments about study | This study focus is test using the Modbat in Zookeper creating a news users in system. |
| Data | Response |
|---|---|
| Title of Study | Model-Based Testing of Stateful APIs with Modbat. |
| Authors | Cyrille Artho, Martina Seidl, Quentin Gros, Eun-Hye Choi, Takashi Kitamura, Akira Mori, Rudolf Ramler, Yoriyuki Yamagata. |
| Year of publication | 2015. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | Modbat. |
| Use or propose a DSL? | Use. |
| DSL Features | Probabilistic and non-deterministic transitions, component models with inheritance, and exceptions. |
| Which DSL focus domain? | Its runs on the Java VM, which makes it easy to test programs written in Java, Scala, C, and other language. |
| Technologies and Notations used in DSL development | This DSL is scala-based. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | Use the transition-based paradigm. Random and stochastic tests using random and search-based, methods to generate. |
| System Under Testing (SUT) data are represented in the models and how they are represented | In model script using how input and output. |
| Benefits in using DSL | This is based into flexible language (Scala). |
| Drawbacks in using DSL | Even with an elegant and expressive modeling platform, writing a model that includes an output oracle requires an in-depth understanding of the system. |
| Trends in using DSL | Provide a graphical support and focus into coverage. |
| Others comments about study | . |
| Data | Response |
|---|---|
| Title of Study | Domain-Specific Languages with Scala. |
| Authors | Cyrille Artho, Klaus Havelund, Rahul Kumar, Yoriyuki Yamagata. |
| Year of publication | 2015. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | Modbat. |
| Use or propose a DSL? | Use. |
| DSL Features | Generates test cases from extended finite-state machines. |
| Which DSL focus domain? | Not presented. |
| Technologies and Notations used in DSL development | Modbat has been used to verify a Java model library and a SAT solver. And use Scala |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | This data not is presented, only use a extended finite-state machines. |
| System Under Testing (SUT) data are represented in the models and how they are represented | A Modbat model contains definitions of transitions: source and target states, and transition actions (code to be executed when a transition is taken). The key Scala features that are used for deeply embedding the DSL are the definition of a custom operator: = in Transition.scala, together with an implicit conversion of a string pair "a" -> "b", to a transition (in Model.scala). |
| Benefits in using DSL | The Benefits presented not is of the DSL. |
| Drawbacks in using DSL | The Drawbacks presented not is of the DSL. |
| Trends in using DSL | The Trends presented not is of the DSL. |
| Others comments about study | This study contains a comparative between develop external and internal DSLs. |
| Data | Response |
|---|---|
| Title of Study | An environment for automatic tests generation from use case specifications. |
| Authors | Cunha, Carolina D; Song, Mark A J. |
| Year of publication | 2014. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | Not presented (UC-DSL). |
| Use or propose a DSL? | Propose. |
| DSL Features | Test specification and Test Generation. |
| Which DSL focus domain? | Web Applications. |
| Technologies and Notations used in DSL development | Xtext. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | This article use the activity diagram to generate tests by understood by users. |
| System Under Testing (SUT) data are represented in the models and how they are represented | In use case model. This study use the use case model "Fully Dressed".This template uses one text column and numbered steps with a convention of numbers and letters for alternate steps. This numbering structure supports the identification and generation of scenarios. |
| Benefits in using DSL | Generate test scripts, provides debug in script generated. |
| Drawbacks in using DSL | Non-treatment of asynchronous steps and lack of permission for variations of more than one level, inside the alternate flux, situations. |
| Trends in using DSL | Pairwise test, boundary-value analysis and equivalence class partitioning techniques, Screen modeling, Importing specifications created with other tools and export (XMI), capture and Replay, execute only UC modifieds. |
| Others comments about study | . |
| Data | Response |
|---|---|
| Title of Study | Capturing and validating personalization requirements in Web applications. |
| Authors | Esteban Robles Luna, Irene Garrigós, Gustavo Rossi. |
| Year of publication | 2010. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | WebSpec. |
| Use or propose a DSL? | Use. |
| DSL Features | Specify the requirements. |
| Which DSL focus domain? | Web Applications. |
| Technologies and Notations used in DSL development | EMF and GMF. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | After a requirement has been specified by means of WebSpec diagrams, we are able to automatically derive meaningful interaction tests to assess whether the requirement has been successfully implemented. An interaction test opens a Web browser and executes a set of actions in the same way a user would do it. Interaction tests allow making assertions on HTML elements based on XPath expressions so we can check the values of the different widgets. For each diagram, we create a test suite. Each path depicted in the diagram will be translated into a test case that will be named as the complete path’s trail. A test case will follow the actions specified in the path, and assertions will be generated from the invariants of every interaction. The actions specified on navigations will be translated into sentences in the test, for example typing text into a text field or clicking buttons. Reaching an interaction will require that we check its invariant (if any), by generating assertions on the test. As different interactions may alter the variables bound to an invariant, it may be necessary to repeat the updated assertions after navigating to the same interaction more than once. |
| System Under Testing (SUT) data are represented in the models and how they are represented | A WebSpec diagram has two key elements: interactions and navigations. An interaction (the counterpart of a Web page in the requirements stage) represents a point where the user can interact with the application by using its interface objects. Interactions may have widgets such us: labels, list boxes, buttons, radio buttons, checkboxes, and panels. Labels define the content (information) shown by an interaction. A diagram has a starting interaction which is represented with dashed lines. Some actions (clicking a button, adding some text in a text field, etc) might activate navigation from one interaction to another. |
| Benefits in using DSL | Specify web application and generate test scripts in Selenium WebDriver. |
| Drawbacks in using DSL | This data not is presented. |
| Trends in using DSL | Deal with the evolution of model. |
| Others comments about study | . |
A52 - User acceptance testing for Agile-developed web-based applications: Empowering customers through wikis and mind maps (2017)
| Data | Response |
|---|---|
| Title of Study | User acceptance testing for Agile-developed web-based applications: Empowering customers through wikis and mind maps. |
| Authors | I.Otaduy, O.Diaz. |
| Year of publication | 2017. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented in this article. |
| DSL Name | TestMind - DSL. |
| Use or propose a DSL? | Propose. |
| DSL Features | Collaborative test by Fitnesse and TestMind generate test script can be then saved as Fitnesse test pages, test specification, screenshots. |
| Which DSL focus domain? | Web-based applications. |
| Technologies and Notations used in DSL development | This data not is presented, only TestMind are a Freemind for test plugin. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | UAT Scripts. These scripts are the executable artifacts generated from UATCases. Each UATScript exercises a UATCase using different input data sets to check out the behaviour of the application in different scenarios. For each set of input data, the customer might have a different expectation (e.g., “if I introduce 16/02/02, the application should pass”, “if I introduce, 16/42/42, the application should throw an error”). The customer manually defines this expectation based on application screenshots that are taken during the test execution (ExpectationShots). Once checked, ExpectationStatements can be created in order to point out whether the test passed (i.e., the application worked as desired) or failed (i.e., the application did not behave as expected). |
| System Under Testing (SUT) data are represented in the models and how they are represented | UAT Cases: Test cases are defined as “sets of test inputs, execution conditions, and expected results” [41]. In our approach, test cases represent a sequence of steps denoted as “UATActions”, i.e. commands that mimic user interaction with the web interface (e.g. click, data input). To ease customer understanding, UATActions are grouped into the Page where these actions happen. As for the expected results, they are defined as test oracles, that is, methods for checking whether the WAUT has behaved correctly on a particular execution [42]. In scripting languages, these oracles are realized as assertions, i.e. checking actions to be performed in order to validate the UAT case result (e.g. check the content of an element, check that a specific page is loaded). Traditionally, assertions are boolean expressions set by developers at a specific point in a program which will be true unless there is a bug. |
| Benefits in using DSL | Final user involvement with test development. |
| Drawbacks in using DSL | No presents. |
| Trends in using DSL | Use of the mind maps into Regression test. |
| Others comments about study | "In acceptance testing, Cucumber is a popular tool [12]. As another example, Selenium provides a different notation (called Selenese) that permits to define test cases in terms of a sequence of user interactions with the application UI [36]. No matter the notation used, an important requirement is for the test cases to be executable since users other than the author might need to run the test". Contains because a stakeholders not are involvement. |
| Data | Response |
|---|---|
| Title of Study | Evaluation of an Integrated Tool Environment for Experimentation in DSL Engineering. |
| Authors | Florian Häser, Michael Felderer, Ruth Breu. |
| Year of publication | 2017. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented in this study. |
| DSL Name | Not present (Sample-DSL). |
| Use or propose a DSL? | Use. |
| DSL Features | Specify test scenarios. |
| Which DSL focus domain? | Web applications. |
| Technologies and Notations used in DSL development | MPS. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | This study not presents this data. |
| System Under Testing (SUT) data are represented in the models and how they are represented | In script test using keywords with click, type and more. |
| Benefits in using DSL | No contains. |
| Drawbacks in using DSL | No contains. |
| Trends in using DSL | This data not is presented. |
| Others comments about study | . |
| Data | Response |
|---|---|
| Title of Study | Towards Flexible and Automated Testing in Production Systems Engineering Projects. |
| Authors | Dietmar Winkler, Kristof Meixner, Stefan Biffl. |
| Year of publication | 2018. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | Gherkin. |
| Use or propose a DSL? | Use. |
| DSL Features | Define test cases by system behaviour. |
| Which DSL focus domain? | This data not is presented. |
| Technologies and Notations used in DSL development | This data not is presented. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | This data not is presented. |
| System Under Testing (SUT) data are represented in the models and how they are represented | Standardized input sequence [12], i.e., GIVEN (x), WHEN (y), THEN (z). While describes pre-conditions and the test environment, focuses on the action under test, and describes the expected outcomes of the test. |
| Benefits in using DSL | Define the test cases without knowing details on the implemented test cases. |
| Drawbacks in using DSL | This data not is presented. |
| Trends in using DSL | this data not is presented. |
| Others comments about study | . |
| Data | Response |
|---|---|
| Title of Study | Pattern‐based GUI testing: Bridging the gap between design and quality assurance. |
| Authors | Rodrigo M. L. M. Moreira, Ana Cristina Paiva, Miguel Nabuco, Atif Memon. |
| Year of publication | 2017. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented in this Study. |
| DSL Name | Paradigm-DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | Building GUI test models based on UI Test Patterns, generate test cases from PARADIGM models, generate PARADIGM models from Web pages without requiring access to the source code, and has a execution tool for run test cases and analyze coverage, generate reports. |
| Which DSL focus domain? | Web Applications. |
| Technologies and Notations used in DSL development | Eclipse Modeling Framework. |
| DSL type | Graphical (Similar with Activity Diagram). |
| Technique/approach/method to ensure coverage of the system with test cases | Generically, the first step is to flatten the model into 1 level by recursively expanding all structural (Forms and Groups) elements. After this process, the model does not have Forms nor Groups, so every node is a UI Test Pattern. Then, the tool generates (whenever possible) all possible paths that traverse the model from Init to End. In short, a test path is a sequence of UI Test Patterns. For instance, for the model in Figure 14, the test paths are [1,2] and [3], where 1, 2, and 3 are the IDs of the UI Test Patterns within the model in Figure 14. And contains random, default and invalids strategy for generate TC. |
| System Under Testing (SUT) data are represented in the models and how they are represented | Provide a generic test strategy for GUIs that feature input fields or drop-down menus, across their different implementations. The test strategy consists in the following: 1. Test Goals: “Valid data” (INP_VD) and “Invalid data” (INP_ID); 2. Set of variables V: {input}; 3. Sequence of actions A: [provide input]. 4. Set of checksC: {“message box X”, “label Y”, “error provider Z”} where X, Y, and Z correspond to the text to be displayed; During configuration, the user has to provide valid input data for INP_VD (and invalid input data for INP_ID), select the checks to perform, and define the precondition. |
| Benefits in using DSL | Higher level of abstraction requiring less modeling effort, the PBGT approach can also be useful during development of Web applications starting the construction of the PARADIGM models when eliciting requirements, Along with the development, the PARADIGM models can be updated (if needed) and configured. |
| Drawbacks in using DSL | Too much effort to create the models, usability issues in reporting.. |
| Trends in using DSL | Reverse engineering approached to extract automatically UI Test Patterns (and possible some configurations of those patterns and mapping) from Web applications to diminish further the modeling effort. |
| Others comments about study | "Finally, we intend to promote the PBGT approach in the testing community and direct the approach towards industry adoption. The idea is to promote sessions at software companies, so test teams could see the value of PBGT, its benefits and adopt it". |
| Data | Response |
|---|---|
| Title of Study | Test patterns for android mobile applications. |
| Authors | Inês Coimbra Morgado, Ana C. R. Paiva. |
| Year of publication | 2015. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented in this study. |
| DSL Name | PARADIGM-DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | Build test models based on UITPs, Modelling environment to build and configure GUI models, Reverse engineering process to automatically extract and generate the model of a web application, test case generator based on PARADIGM models. |
| Which DSL focus domain? | Web and Mobile Applications. |
| Technologies and Notations used in DSL development | Eclipse Modelling Framework. |
| DSL type | This data not is presented in this study. |
| Technique/approach/method to ensure coverage of the system with test cases | This data not is presented in this study. |
| System Under Testing (SUT) data are represented in the models and how they are represented | Um Padrão de Teste de UI é o conjunto das estratégias de teste associadas e consiste em: <Objetivo, V, A, C, P>, conforme definido em [Moreira et al. 2013] em que: Meta. o ID do teste; V. um conjunto de pares variável, valor que relaciona dados de entrada de teste com as variáveis envolvidas no teste; A. a seqüência de ações a serem executadas durante a execução do caso de teste; C. o conjunto de verificações a executar durante a execução do caso de teste, isto é, a indicação de se o padrão está corretamente implementado ou não; P. a pré-condição (expressão booleana) definindo as condições nas quais é possível executar o teste. |
| Benefits in using DSL | The PBGT project has proven the usefulness of defining test strategies (UI Test Patterns) for testing recurring behaviour (UI Patterns) on web applications, and contains three UI Test Patterns to test mobile applications. |
| Drawbacks in using DSL | Moreover, the experiment conducted on Android has proven that even though the same approach can be applied to mobile applications, it is necessary to specify UI Test Patterns that are mobile-specific. This happens because mobile applications have additional behaviour that is not present in web applications, such as changing the orientation of the screen. |
| Trends in using DSL | Develop more UI Test Patterns specific to the mobile world, integrate these three UITPs within the PARADIGM language (the DSL used in the context of the PBGT project) in order to enable the modelling of additional test goals specific for the mobile world and implement the test strategies for these UITPs in order to allow its automatic execution. |
| Others comments about study | . |
| Data | Response |
|---|---|
| Title of Study | AppFlow: Using Machine Learning to Synthesize Robust, Reusable UI Tests. |
| Authors | Gang Hu, Linjie Zhu and, Junfeng Yang. |
| Year of publication | 2018. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | Gherkin. |
| Use or propose a DSL? | Use. |
| DSL Features | Describe Flows in application, . |
| Which DSL focus domain? | Mobile and Web Applications. |
| Technologies and Notations used in DSL development | This data not is presented. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | This data not is presented in this study. |
| System Under Testing (SUT) data are represented in the models and how they are represented | The actions in a flow are specified using a verb followed by its arguments. The verbs are common operations and checks, such as “see”, “click”, and “text”. The arguments can be widgets or values. For widgets, either canonical ones or real ones can be used. Canonical ones are referenced with @. Simple methods such as “id(arg)”, “text(arg)” and “desc(arg)” find widgets by comparing their corresponding attributes with the argument “arg,” while method “marked(arg)” matches any of those attributes. Here “arg” may be a constant or a configuration variable indicated using @. This study extend the Gherkin language additioning visible and abstract properties and use machine learning. |
| Benefits in using DSL | This machine learning approach enables the AppFlow tests to refer to canonical screens and widgets instead of app-specific ones, enjoying a variety of benefits. First, apps’ UI can now evolve without breaking tests as long as the new designs can be recognized by AppFlow. Second, app UI can now respond to device factors such as screen size without breaking tests. Third, canonical screens and widgets abstract app-specific variations, making it easy to share tests across apps. Fourth, AppFlow’s ability to recognize screens enables developers to focus on testing the specific flows of a screen without writing much boilerplate code to first bring the app to the screen or later restore the app to a previous state. |
| Drawbacks in using DSL | AppFlow aims at greatly reducing manual effort in implementing automated UI testing. We did not design AppFlow to replace manual testing completely: it is well known that as of now automated UI testing cannot replace manual UI testing completely because the user experience is highly subjective. Support only Android Plataform |
| Trends in using DSL | Support all plataforms. |
| Others comments about study | . |
| Data | Response |
|---|---|
| Title of Study | Model refactoring within a Sequencer. |
| Authors | TOMAŽ KOS, TOMAŽ KOSAR, JURE KNEZ, MARJAN MERNIK. |
| Year of publication | 2012. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | Sequencer. |
| Use or propose a DSL? | Use. |
| DSL Features | Used for developing and verification measurement procedures. |
| Which DSL focus domain? | Measurement procedures. |
| Technologies and Notations used in DSL development | This data not presented in this study. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | No contains. |
| System Under Testing (SUT) data are represented in the models and how they are represented | Use of Assert block in Sequencer Script. |
| Benefits in using DSL | This data not is presented, because this study focus is model refactoring. |
| Drawbacks in using DSL | This data not is presented, because this study focus is model refactoring. |
| Trends in using DSL | This data not is presented, because this study focus is model refactoring. |
| Others comments about study | This DSL not only for functional tests. |
A59 - PBGT tool: an integrated modeling and testing environment for pattern-based GUI testing (2014)
| Data | Response |
|---|---|
| Title of Study | PBGT tool: an integrated modeling and testing environment for pattern-based GUI testing. |
| Authors | Rodrigo M.L.M. Moreira, Ana C.R. Paiva. |
| Year of publication | 2014. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | PARADIGM-DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | Using this DSL you can: Reverse engineering, building GUI test models based on UI Test Patterns, generate and execute test cases. |
| Which DSL focus domain? | Web and Mobile Applications. |
| Technologies and Notations used in DSL development | Eclipse Modeling Framework (EMF). |
| DSL type | this data not is presented in this study (Graphical). |
| Technique/approach/method to ensure coverage of the system with test cases | This generate is graph-based, using the init and end elements. In addition use the configuration previously by Tester. |
| System Under Testing (SUT) data are represented in the models and how they are represented | The PARADIGM language has seven Base UI Test Patterns: • Input – this pattern should be used to test the behavior of input fields for valid and invalid input data; • Login – this pattern should be used to verify user authentication. The goal is to check if it is possible to authenticate with a valid username/password and check if it is not possible to authenticate otherwise; • Master/Detail – this pattern should be used to test two related objects (master and detail) in order to verify if changing the master’s value, correctly updates the contents of the detail; • Find – the purpose of this pattern is to check if the result of a search is the correct set of values; • Sort – this pattern is used to check if the result of a sort action is ordered accordingly to the chosen sort criteria; • Call – this pattern should be used to check the behavior of the corresponding call, for instance, check if a link opens a new page; • Option Set – this pattern should be used to check the expected behavior for multiple selections. |
| Benefits in using DSL | Reusability concerns, Reduced efforts, Goal focus, Platform independent, No source code is required, Low maintenance and evolutionary and Simple to use. |
| Drawbacks in using DSL | No test desktop and iOS applications. |
| Trends in using DSL | Promote the tool, and offer support for desktop and iOS applications. |
| Others comments about study | . |
| Data | Response |
|---|---|
| Title of Study | Multidimensional test coverage analysis: PARADIGM-COV tool. |
| Authors | Ana C. R. Paiva, Liliana Vilela. |
| Year of publication | 2017. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented in this study. |
| DSL Name | PARADIGM-DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | Build part of the model by a reverse engineering process and complete it manually afterwards, the configuration phase where the tester selects the testing goals (TGs) and provides test input data, establishing the mapping between model elements and GUI controls to identify the controls in which to act during test case execution, generating test cases from PARADIGM models, execution of the test cases over the application under test (AUT), build reports with the test cases execution results. |
| Which DSL focus domain? | This sample is using a Web Applications. |
| Technologies and Notations used in DSL development | This data not is presented, but samples in study this DSL is the Eclipse Plugin. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | In order to generate test cases from a PARADIGM model, the tester must go through a configuration step in which he selects the TGs for each UI Test Pattern within the model and provides test configuration data (TGConf): 1. the test input data, 2. the checks to perform, and 3. the preconditions defining the states in which a TGConf may be executed. A PARADIGM model can be structured into different levels of abstraction, thus, to generate test cases, the algorithm starts by (1) flattening the model. It does this by replacing recursively all structural elements (Forms and Groups) within a hierarchical level of a model by their internal elements, present at the next level, until all Forms and Groups are discarded. After this flattening process, (2) it calculates the set of paths (SPaths) that go from the Init to the End elements within the model and guarantees full connector coverage. Considering that some UITPs can be optional (as is the case of ID[2][True] inside Fig. 2), some additional paths may be created ignoring such optional UITPs. Then, (3) it expands every path within SPaths into test cases according to the TGConf defined for the UITP within each path. At the end, the algorithm (4) discards all the configurations that cannot be executed because their precondition can never be True. More details about this generation process can be found in. |
| System Under Testing (SUT) data are represented in the models and how they are represented | Using UITP (User Interface Test Patterns) in System graph. |
| Benefits in using DSL | Test requirements coverage (test goal analysis), model coverage (restriction analysis to check if preconditions ever hold), test case coverage (passed/executed metrics through Execution Analysis), code coverage (which parts of the code were exercised) and, test script analysis (assess the quality of a test script related to the coverage of a PARADIGM model). |
| Drawbacks in using DSL | The coverage analysis is functional only PHP applications. |
| Trends in using DSL | Implement this analysis for more languages. |
| Others comments about study | This study presents PARADIGM-COV, this is a tool for analysis coverage test. "This approach relies on the fact that most GUIs end up having similar elements and behavior (the so-called User Interface Patterns), which can be reused across several applications." |
| Data | Response |
|---|---|
| Title of Study | Pattern-Based Usability Testing. |
| Authors | Fernando Dias, Ana C. R. Paiva.. |
| Year of publication | 2017. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | PARADIGM-DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | PARADIGM, which allows building models describing testing goals. The test cases generated from these models are then executed on a graphical user interface to find failures. |
| Which DSL focus domain? | Web Plataforms. |
| Technologies and Notations used in DSL development | This data not is presented in this study. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | The test pattern is executed by exploring the website under analysis and by building a tree where each vertex corresponds to a link (or URL) in the website and each edge corresponds to a possible transition between two links. Thus, based on the depth n, indicated by the user, which corresponds to the maximum number of clicks to reach the destination URL, a tree is constructed resulting from the exploration. Whenever a new URL is reached, it is checked whether it is the destination sought. In constructing the tree we do not consider return transitions and each node is only inserted if it is not already present in the tree. If the desired URL can be reached in n steps, the test passes. If the tree is fully scanned to the indicated depth and the destination is not found, the test fails. The tree is built using the Breadth-first search algorithm, where all the children of a node are explored before moving to the node below. Each node has an associated URL achievable through the website exploration, and the children of the current node 368 and their children are always collected until the depth n is reached. The depth corresponds to the maximum number of clicks, n, to reach the destination. |
| System Under Testing (SUT) data are represented in the models and how they are represented | A Test Pattern is the set of associated test strategies that may be defined as [2]: <Goal; V; A; C; P> • Goal is the ID of the test • V is a set of pairs variable, value relating test input data with the variables involved in the test • A is the sequence of actions to perform during test case execution • C is the set of checks to perform during the test case execution • P is the precondition (Boolean expression) defining the conditions in which it is possible to run the test. In addition this data are represented in System Graph. |
| Benefits in using DSL | Different levels of abstraction. |
| Drawbacks in using DSL | Not support dynamic behavior of web applications. |
| Trends in using DSL | Insert new usability patterns and support to dynamic behavior. |
| Others comments about study | "Goal: "Is it possible to achieve a feature in n steps?" • V is the set of input variables and their values, the target URL and the maximum number of steps (n) to perform in order to reach the target URL; • A is empty. The actions performed aim to explore the GUI under test, are automatic and do not need additional inputs • C will check if the target URL is achieved in n steps • P is true. It will always be possible to execute this test strategy". |
| Data | Response |
|---|---|
| Title of Study | GUICop: Approach and toolset for specification‐based GUI testing. |
| Authors | Dalal Hammoud Fadi A. Zaraket Wes Masri. |
| Year of publication | 2017. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | GUI specification language. |
| Use or propose a DSL? | Propose. |
| DSL Features | The GUICop Specification Language aims at capturing positional, arithmetic, logic, and relational GUI behaviours. |
| Which DSL focus domain? | Only presents your use in JAVA Cross Plataform. |
| Technologies and Notations used in DSL development | ANTLR for create abstract syntax tree. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | This study not ensures coverage, only presents how create testscript = "The Code Weaver uses the input with preexisting parametrized Java aspects to produce the aspect. It uses aspect-oriented programming to weave calls (using AspectJ) from the SUT to the Solver at appropriate code locations. The calls invoke the Solver to check whether a given specification (whose identifier is passed as a parameter) has been violated or not. Specifically, the guicop.startCapture() call will enable capturing the GUI events, and the guicop.Check(spec) will stop capturing the events and invokes the Solver to check whether the captured events satisfy the specification. In case the GUICop aspect did not include a start statement, GUICop captures all events. In addition to startCapture() and check(), GUICop allows for stopCapture() and resumeCapture() calls to allow testers more control over when the GUI events are recorded". |
| System Under Testing (SUT) data are represented in the models and how they are represented | Insert behaviour in script using variables, properties and constraints. It comprises the construct “variables” in which the user declares the variables to be used. These variables could be of primitive types, namely, Rectangle, Line, Ellipse, Polygon, Triangle, Text, and Textrect, or they could be of complex types, ie, of a type previously defined by the user and archived in the Specification Library. It comprises the construct “properties,” which is a list of name value pairs specifying information such as the location and size of the component under test. The properties include by default the elements of the rectangle that bounds the component, namely, X, Y, WIDTH, and HEIGHT. It comprises the construct constraints in which the user describes how primitive or complex objects should appear and how they should be positioned with respect to each other. A constraint is an expression involving the declared variables as operands and the supported operators. Noting that several constraints could be defined. |
| Benefits in using DSL | GUICop is useful and applicable to real‐life applications, The thoroughness of the GUICop oracles is valuable given that the null oracle cannot detect any of the defects in the case studies, GUICop can be more beneficial than the GUI tree‐based approaches given that in the case studies, (1) names of GUI components were not known, eg, the list items in Jajuk (case study 1), and (2) positional properties, such as leftto, cannot be checked directly or easily using the GUI tree relationships, GUICop can be more beneficial than the image‐based approaches, for example, (1) in case study 3, RTL script detection may require Optical Character Recognition for the image, which is expensive and may not be reliable for non‐Latin text, and (2) in Case Study 1, checking that Track2 is above Track10 requires OCR, image segmentation, and registration. Writing Sikuli scripts to specify such complex tasks requires image processing expertise. |
| Drawbacks in using DSL | Not support to generic specifications to be checked in global mannar. |
| Trends in using DSL | 1. Explore the possibility of integrating GUICop within the GUITAR framework [38]. 2. Extend the GUICop specifications to support the following: (1) temporal operators to check event timings; (2) more string operators; (3) regular expression matching; (4) operators such as intersects and occludes; and (5) properties such as font, style, line thickness, line type, and colour. 3. Extend and simplify the GUICop Specification Language and provide a GUI tool that would facilitate writing GUICop specifications. 4. Allow for better reuse of existing specifications by extending the GUICop Library to include most widely used GUI components. 5. Allow for generic specifications to be checked in a global manner. For example, instead of requiring the user to configure the Code Weaver to inject MainMenuLabel checks at specific locations, checks would be performed every time a menu label is displayed. |
| Others comments about study | "PushedRadiobutton = { variables { Ellipse e1, e2; } properties { X = e1.x; Y = e1.y; WIDTH = e1.width; HEIGHT = e1.height; } constraints { (e1 contains e2); } } ". Example of push radio button specification |
A63 - Towards a Test Specification Language for Information Systems: Focus on Data Entity and State Machine Tests (2018)
| Data | Response |
|---|---|
| Title of Study | Towards a Test Specification Language for Information Systems: Focus on Data Entity and State Machine Tests. |
| Authors | Alberto Rodrigues da Silva, Ana C. R. Paiva, and, Valter Emanuel R. da Silva. |
| Year of publication | 2018. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | TSL (Test Specification Language). |
| Use or propose a DSL? | Propose. |
| DSL Features | TSL support human-readable executable specifications closer to natural language than models usually used in model-based testing approaches. TSL includes several constructs logically arranged into views according to multiple testing engineering strategies, commonly found in the information systems domain, such as: data entity tests and state machine tests, all of them produced from equivalent requirement specification in RSL. In addition is generate a Gherkin script for automate the test case generation process, parser to RSL to TSL. |
| Which DSL focus domain? | Information Systems. |
| Technologies and Notations used in DSL development | Xtext for use in Eclipse IDE. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | Use State Machine, and use Switch-0 coverage (Means that you exercise every direct transition possible in the model). |
| System Under Testing (SUT) data are represented in the models and how they are represented | In script using specification of data entities, can be use of the equivalence partitioning. |
| Benefits in using DSL | The benefit of the TSL is that it builds a view with all the entities and attributes for which the tester should define test input data. In case of sequential attribute values (such has numbers), it is also possible to apply boundary value analysis to define test input data. For instance, if we have an attribute B that ranges from 5 to 7, the tester can define test input data on the boundaries, e.g., 5 and 7 for valid, and 4 and 8 for invalid values. |
| Drawbacks in using DSL | The coverage criteria can not applicable in real scenario yet. |
| Trends in using DSL | Extend for generate Users Stories and Use Cases, implements others algorithms for coverage how Switch-1 or Switch-2. |
| Others comments about study | . |
A64 - A Test Specification Language for Information Systems Based on Data Entities, Use Cases and State Machines (2019)
| Data | Response |
|---|---|
| Title of Study | A Test Specification Language for Information Systems Based on Data Entities, Use Cases and State Machines. |
| Authors | Alberto Rodrigues da Silva, Ana C. R. Paiva, Valter E. R. da Silva. |
| Year of publication | 2018. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | Test Specification Language (TSL). |
| Use or propose a DSL? | Propose. |
| DSL Features | Conversion of the RSL to TSL and TSL to Gherkin, TSL to TSL for refined test cases. Specification and generation (whenever relevant) of software tests defined. (i) extend the RSLingo approach with the support for testing activities; (ii) define a set of strategies that would allow generating test cases from the RSL constructs; and (iii) automate the test case generation process. |
| Which DSL focus domain? | Information Systems Based on Data Entities. |
| Technologies and Notations used in DSL development | This data not is presented, only the TSL allows systematizing the test developing process with both Xtext-based and Excel RSL formats. Xtext based format is handled with the integration of the Eclipse IDE. |
| DSL type | Textual, based in RSL (Input for Language). |
| Technique/approach/method to ensure coverage of the system with test cases | DataEntityTestCases can be defined by applying equivalence class partitioning and boundary value analysis [31] over RSL DataEntities; – UseCaseTestCase can be defined by exploring multiple sequences of steps defined in RSL use cases’ scenarios, and also by associating data values to the involved data entities; – StateMachineTestCases can be defined by applying different algorithms to traverse the state machine defined in RSL, so that it shall be possible to build different test cases that correspond to different paths through the state machine. |
| System Under Testing (SUT) data are represented in the models and how they are represented | In Data Entity Test Cases: Define of the valid and invalid values into TSL. In Use Case Test Cases the scenario is defined using actor and is derived for flows the RSL specification. In State Machine Test Cases: defined using state machine based in RSL and using algorithms for extract sequences, but has defined if sequence is valid or not. |
| Benefits in using DSL | Generate test cases based of the RSL Specification. The benefit of the TSL is that it builds a view with all the entities and attributes for which the tester should define test input data. In the case of sequential attribute values (such as numbers), it is also possible to apply boundary value analysis to define test input data. |
| Drawbacks in using DSL | The language does not contain the most variety of the methods for generating tests sequences. |
| Trends in using DSL | Using of the test data generation. Using others algorithms for generate test sequences based into State Machine (i.e. Switch-1 and Switch-2), Integrations with Cucumber and Specflow. |
| Others comments about study | TSL allows specifying various black-box test cases in a syntactic manner similar to that expressed by RSL. |
| Data | Response |
|---|---|
| Title of Study | Towards a pattern language for model-based GUI testing. |
| Authors | Rodrigo M. L. M. Moreira, Ana C. R. Paiva. |
| Year of publication | 2014. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented in this study. |
| DSL Name | PARADIGM - DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | Generate Test Scripts, Parameterized data. |
| Which DSL focus domain? | Web Applications and Recently Mobile Applications, only Android SO. |
| Technologies and Notations used in DSL development | This article not present this data. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | This data not is presented in this study. |
| System Under Testing (SUT) data are represented in the models and how they are represented | All data required for create a test in PARADIGM-DSL: 1. The goal is the ID of the test; 2. V is a set of pairs {[variable, inputData]} relating test input data with the variables involved in the test; 3. A is the sequence of actions to perform during test case execution; 4. C is the set of possible checks to perform during test case execution and; 5. P is a Boolean expression (precondition) defining the conditions over variables that determine when it is possible to execute the test. In addition is possible parameter data in this DSL. |
| Benefits in using DSL | Parameterized data. Reuse of GUI testing strategies. |
| Drawbacks in using DSL | Currently, UI Patterns have to be identified by the tester, representing a manual task. |
| Trends in using DSL | Automate this process, so could automatically identify a set of elements that represent UI Patterns and then be able to map them to the associated UI Test Pattern. |
| Others comments about study | This study focus is in use of the PARADIGM-DSL. |
A66 - Automated test case generation from high-level logic requirements using model transformation techniques (2017)
| Data | Response |
|---|---|
| Title of Study | Automated test case generation from high-level logic requirements using model transformation techniques. |
| Authors | Oyindamola Olajubu, Suraj Ajit, Mark Johnson, Scott Thomson, Mark Edwards, and Scott Turner. |
| Year of publication | 2017. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | A66 - DSL (Name not is specified). |
| Use or propose a DSL? | Propose. |
| DSL Features | Specify high-level requirements. |
| Which DSL focus domain? | Aviation Systems. |
| Technologies and Notations used in DSL development | Xtext. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | Specifications with more than one condition decision have two outcomes, one when the condition is true and another when the condition is false. The MC/DC criterion is applied when there are multiple conditions within the decision. To achieve this coverage for the requirements-based test cases, the following requirements for MC/DC are considered: –every decision in the program has taken all possible outcomes at least once –every condition in a decision in the program has taken all possible outcomes at least once –every condition in a decision has shown to independently affect that decision’s outcome To derive test cases to satisfy the coverage criterion for specifications with multiple conditions with a single operator (AND / OR), the total number of test cases required is n+1 where n is the number of conditions in the specification [6]. Depending on the operator type, a walking false (AND) or walking true (OR) pattern is applied. |
| System Under Testing (SUT) data are represented in the models and how they are represented | Directly into Test Script. The script is Gherkin-like and contains the domain vocabulary. example: The displayPanel shall be Off when brightnessMonitor = Off. |
| Benefits in using DSL | Benefit of natural language expressiveness while supporting model manipulation techniques. |
| Drawbacks in using DSL | No Contains. |
| Trends in using DSL | Application of model transformation techniques to other types of requirement specifications including pseudo requirements, boundary value analysis, and timing requirements. |
| Others comments about study | "The use of a textual DSL combines the benefit of natural language expressiveness while supporting model manipulation techniques. This implies that stakeholders can use a notation which incorporates domain jargon to define the functionality of software". |
| Data | Response |
|---|---|
| Title of Study | A Domain Specific Test Language for Systems Integration. |
| Authors | Robin Bussenot, Hervé Leblanc, Christian Percebois. |
| Year of publication | 2018. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | A67-DSL (Name not specified). |
| Use or propose a DSL? | Propose. |
| DSL Features | Generate structure of the Unit Tests and Integration tests. In addition is possible use of the automations part of the procedures, integration with OCL constraints. |
| Which DSL focus domain? | Avionic Systems and Networks System Integration. |
| Technologies and Notations used in DSL development | MPS. |
| DSL type | Textual, Gherkin-Like. |
| Technique/approach/method to ensure coverage of the system with test cases | To place the TestProcedure correctly in the workflow document, we have added additional concepts as enumeration types: LabTestRequest, Tool, TestBench and TestObjective. TestObjectives are shared by LabRestRequest and TestChapter in order to generate a traceability matrix that ensures the cover of TestObjectives by TestProcedures. |
| System Under Testing (SUT) data are represented in the models and how they are represented | In this DSL the SUT data are inserted using five kinds instructions: 1)Step: an action is performed on the SUT, or on the test bench, or on a tool. 2)Check: a verification is performed about the state of the SUT. There are two contexts of use: a precondition on the SUT to perform the test and a verdict on the test. 3)Log: an action to store some parameter values involved in the test. 4)Trace: action on a tool in order to record relevant test data. 5)Reminder: instruction addressed to a tester concerning the management of test executions. |
| Benefits in using DSL | The authors to provide a common specific language that improves communications for the team of test designers and between test designers and test performers. In the same manner as agile test frameworks, this language allows structuring the test procedures. Moreover, this language tends to facilitate the automatic execution of some parts of the procedures. The structure we provide for this specific language may be reusable for other specific domains of avionic systems |
| Drawbacks in using DSL | Not generate executable tests. |
| Trends in using DSL | Use of the Ontologies for avionic scenario, use of the state machines for generation and new instructions. |
| Others comments about study | This study contains: Benefits of the DSL use, Others DSLs Language Workbenches (MPS, XText e Spoofax) and comparison between thus. And follow a bottom-up approach. |
| Data | Response |
|---|---|
| Title of Study | From use case maps to executable test procedures: a scenario-based approach. |
| Authors | Nader Kesserwan, Rachida Dssouli, Jamal Bentahar, Bernard Stepien, Pierre Labrèche. |
| Year of publication | 2019. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | TDL. |
| Use or propose a DSL? | Use. |
| DSL Features | Describe Test Cases, convert to executable test procedures, Specify Data Sets can be used in Tests and Test configurations, objectives and Descriptions. |
| Which DSL focus domain? | Embedded Systems, the case study is using a avionic system. |
| Technologies and Notations used in DSL development | Xtext, Xtend. |
| DSL type | Textual, but can be represented using a Graphical representation (example in article). |
| Technique/approach/method to ensure coverage of the system with test cases | The authors of [5], have explored the automated generation of TDL Test Descriptions from requirements expressed as UCM scenario models using the jUCMNav tool. This transformation enables the exploration of model-based testing where the use of TDL models simplifies the generation of tests in various languages such as TTCN-3. The authors determined the basic differences between scenarios and test cases in the handling of alternative paths that result from UCM alternatives. They concluded that the use of scenarios for test case generation is feasible, but requires either a different traversal mechanism with a different scenario metamodel or post-processing of scenarios to merge those that constitute alternate test behaviors. |
| System Under Testing (SUT) data are represented in the models and how they are represented | 1) A Test Objective that states the reason for designing either a Test Description or a particular behavior of a Test Description. It can be written as a simple text in natural language. 2) A set of typed Data Sets used in the interactions between components in a Test Description. 3) A Test Configuration, which is a set of interacting components (Tester and SUTs). 4) A set of Test Descriptions to describe one or more test scenarios based on the interactions of data exchanged between the Tester and the SUTs. The control flow of a Test Description is expressed in terms of the composition of operations such as sequential, parallel, alternative, iterative. |
| Benefits in using DSL | Describe scenarios on a higher abstraction level than programming or scripting language. Used mainly for communication between stakeholders as the basis for implementing concrete tests. |
| Drawbacks in using DSL | This data not is presented. |
| Trends in using DSL | This data not is presented. |
| Others comments about study | This study propose a novel methodology for testing functional requirements. |
| Data | Response |
|---|---|
| Title of Study | Testing and Gherkin in agile projects. |
| Author | Magnus Härlin. |
| Year of publication | 2016. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | Gherkin. |
| Use or propose a DSL? | Use. |
| DSL Features | Describe Software Behaviour, integration with cucumber framework to provide implementation of the behaviour described. |
| Which DSL focus domain? | The case used in this study is in web-based system, but this data not is explicity. |
| Technologies and Notations used in DSL development | This data not is presented in this study. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | The Gherkin use not ensure coverage of the system, this DSL require Cucumber Framework for implementation of the scenarios when. |
| System Under Testing (SUT) data are represented in the models and how they are represented | Gherkin is written in files with the extension ”feature” where each feature have an independent file, one feature file can consist of several scenarios. Each scenario is described with a title and with the given, when, then format. A scenario in Gherkin consists of a list of steps and is an example that illustrates a business value. A step starts with one of the keywords: ”Given”, ”When” and ”Then”, ”And” and ”But” can be used to write multiple steps underneath each of the keywords. The scenarios should, however, be kept simple and unambiguous. |
| Benefits in using DSL | Communication with stakeholder not technician. Provide feedback and ideas before any actual code is written depending on the implementation of the tester. |
| Drawbacks in using DSL | Can be ambiguous, depending on the writing mode other levels may not understand. |
| Trends in using DSL | No contains. |
| Others comments about study | The study is focus of the Gherkin Specification in communication and implementation scenario using cucumber framework. |
A70 - A Behavior-Driven Approach for Specifying and Testing User Requirements in Interactive Systems (2018)
| Data | Response |
|---|---|
| Title of Study | A Behavior-Driven Approach for Specifying and Testing User Requirements in Interactive Systems. |
| Author | Thiago ROCHA SILVA. |
| Year of publication | 2018. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented in this study. |
| DSL Name | Gherkin. |
| Use or propose a DSL? | Use. |
| DSL Features | Specifying executable requirements. |
| Which DSL focus domain? | Web-Based Applications. |
| Technologies and Notations used in DSL development | This data not is presented. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | Not contains. |
| System Under Testing (SUT) data are represented in the models and how they are represented | Use the Given-when-then format. And insert others reservates words for use a strategy for running tests. example: When I select ‘’ , “Click on ”. |
| Benefits in using DSL | Documentation and automated tests. Documentation and making easier for clients and other stakeholders to set their final acceptance tests. |
| Drawbacks in using DSL | Vocabulary limited, no classifier errors, Unknown impact of maintaining and successively evolving the mentioned artifacts throughout a real software development process. |
| Trends in using DSL | Use of ontology, present task model assessment results, analyzing charts and results, and coverage of concepts in the current ontology. |
| Others comments about study | this study use of the ontology for running test scripts. |
| Data | Response |
|---|---|
| Title of Study | A Comparative Study of GUI Testing Aproaches. |
| Authors | Rui Carvalho. |
| Year of publication | 2016. |
| Study objective clearly stated? | yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented in this study. |
| DSL Name | PARADIGM. |
| Use or propose a DSL? | Use. |
| DSL Features | Generate Test Scripts, Execution Test Scripts, Parameterized data. |
| Which DSL focus domain? | Web-Based Applications. |
| Technologies and Notations used in DSL development | This data not is presented. |
| DSL type | This data not is presented. |
| Technique/approach/method to ensure coverage of the system with test cases | This data not is presented. |
| System Under Testing (SUT) data are represented in the models and how they are represented | This data not is presented. |
| Benefits in using DSL | Automated tests, generated scripts test. |
| Drawbacks in using DSL | Data not is Presented. |
| Trends in using DSL | Data not is Presented. |
| Others comments about study | Datas the DSL PARADIGM not is explained in the thesis. |
| Data | Response |
|---|---|
| Title of Study | Reverse Engineering of Interaction Patterns. |
| Author | C. Sacramento. |
| Year of publication | 2014. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | Yes. |
| DSL Name | PARADIGM - DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | Define GUI testing models based on UI Test Patterns, generate test scripts, generate coverage methods and reverse engineering. |
| Which DSL focus domain? | Web Applications. |
| Technologies and Notations used in DSL development | This data not is presented. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | This data not is presented. |
| System Under Testing (SUT) data are represented in the models and how they are represented | The PARADIGM language is comprised of elements and connectors. There are four types of elements: Init (to mark the beginning of a model), End (to mark the termination of a model), Structural (to structure the models in different levels of abstraction), and Behavioral (UI Test Patterns describing the testing goals). A UI Test Pattern defines a test strategy to test a specific UI pattern, which is formally defined by a set of test goals (for later configuration) with the form: < Goal;V;A;C;P > Goal is the ID of the test. V is a set of pairs [variable, inputData] relating test input data with the variables involved in the test. A is the sequence of actions to perform during test case execution. C is the set of possible checks to perform during test case execution, for example, “check if it remains in the same page”. P is a Boolean expression (precondition) defining the set of states in which it is possible to execute the test. The language also defines language constraints to guarantee the building of well-formed models, such as “A Connector cannot connect an element to itself ” and “A Connector cannot have Init a destination, or End as source”, to cite a few examples. |
| Benefits in using DSL | Can be Reverse Engineering, and generate test scripts. |
| Drawbacks in using DSL | The tool for reverse engineering does not handle dynamic pages very well. |
| Trends in using DSL | Use of the exploration algorithms for navigate into web page and extract UI elements. |
| Others comments about study | The study propose is a reverse engineering of interaction Patterns into Paradigm DSL scripts. |
| Data | Response |
|---|---|
| Title of Study | Pattern Based Usability Testing. |
| Authors | Francisco Carvalho Rodrigues. |
| Year of publication | 2018. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | PARADIGM-DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | Test execution, analyzing the test coverage, test generation, reverse engineering and describe your application using graphical model. |
| Which DSL focus domain? | web applications. |
| Technologies and Notations used in DSL development | This data not is presented, but is a Eclipse Plugin. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | This data not is presented. |
| System Under Testing (SUT) data are represented in the models and how they are represented | This data not is presented. |
| Benefits in using DSL | Execution and reverse engineering. |
| Drawbacks in using DSL | This data not is presented. |
| Trends in using DSL | Extends to support usability patterns. |
| Others comments about study | . |
| Data | Response |
|---|---|
| Title of Study | Test Coverage Analysis. |
| Authors | Liliana Borges Vilela. |
| Year of publication | 2013. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | PARADIGM-DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | Describe SUT, test generation and can use the reverse engineering. |
| Which DSL focus domain? | In case study are website as tested. |
| Technologies and Notations used in DSL development | Eclipse Plugin. |
| DSL type | Graphical DSL. |
| Technique/approach/method to ensure coverage of the system with test cases | In order to generate test paths, and since a PARADIGM model can be structured into different levels of abstraction, PARADIGM-ME starts by flattening the model, i.e., it replaces recursively all structural elements (Forms) within a hierarchical level of a model by their internal elements, present at the next level, until all Forms are discarded. After this flattening process, PARADIGMME calculates the set of paths (SPaths) that go from the Init to the End elements within the model. Following, it expands every path within SPaths into test cases according to the TGConf defined for the UITP within each path. To conclude, it analyses the preconditions to determine which steps can be performed from said test cases. |
| System Under Testing (SUT) data are represented in the models and how they are represented | This DSL has structural elements (to structure the model in different levels of abstraction), behavioural elements (UI Test Patterns), Init and End elements (to mark the start and end points of a model), and connectors to establish relations among the elements and define their sequencing. |
| Benefits in using DSL | Has a environment and analysis coverage. |
| Drawbacks in using DSL | This data not is presented. |
| Trends in using DSL | Extension of code coverage analysis to other programming languages besides PHP. |
| Others comments about study | This study proposes PARADIGM-COV. |
| Data | Response |
|---|---|
| Title of Study | Model Based Testing - From requirements to tests. |
| Authors | Valter Emanuel Ribeiro da Silva. |
| Year of publication | 2017. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | RSL/TSTL(Test Specification Language)-DSL. |
| Use or propose a DSL? | Propose. |
| DSL Features | Use the RSL, generate functional tests. |
| Which DSL focus domain? | All domains, because this solution not generated Scripts tests only sequences for manual tests. |
| Technologies and Notations used in DSL development | XText framework and Eclipse work environment. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | As mentioned before, the sequence of states are defined to exercise the SUT following a certain criteria. In this case it was explored a Switch-0 approach, covering sequences with the maximum valid transactions present in the model. This was achieved through the usage of a Depth-First Algorithm (DFS). It traverses a graph in a depth-ward motion and uses a stack to store vertexes to effectuate the search. When a dead end is ecountered, then it means that a full state transaction (test case) has been disclose. |
| System Under Testing (SUT) data are represented in the models and how they are represented | This linguistic pattern contains the attributes for its definition (more specifically: , , , , , , , , , , and ). This construct begins by defining the test set, including ID, name and the use case type. Then it encompasses the references keys [UseCase] indicating the Use Case in which the test is proceeding, background [UseCase] in the circumstances of prevailing event flow that take place before the current Use Case, [DataEntity] referring to a possible data entity that is managed throughout the action flow. Considering the flow diagram, for each test case, multiple scenarios can be retrieved. For each of these scenarios it is specified a name, the Scenario Type (Main, Alternative or Exception flow, respectively), and the set of steps needed to be performed. For each step it must be indicated the actor who performs it [Actor], a reference to the Use Case step [Step] and a step definition, describing the action executed. |
| Benefits in using DSL | Functional test cases can be derived from Use Cases. Use Cases describe the interactions between an actor and the system through a sequence of steps. Has a Eclipse Editor Environment. |
| Drawbacks in using DSL | Not generate scripts tests. |
| Trends in using DSL | Provides automation execution of generated tests. |
| Others comments about study | Functional test cases are mapped from the various RSL package-system views, containing several constructs that describe the system behavior, concretely: Actor view, DataEntity view, UseCase view, and StateMachine view. This lead to the creation of three main test constructs by applying of black-box test design techniques. |
| Data | Response |
|---|---|
| Title of Study | Model-based testing: From requirements to tests. |
| Authors | Daniel Ademar Magalhães Maciel. |
| Year of publication | 2019. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | RSL(Requirement Specification Language). |
| Use or propose a DSL? | Use. |
| DSL Features | Supports requirements and tests specification, Besides the use and extension of the RSL grammar, the approach also uses support tools such as the Robot framework and Web Scrapper. |
| Which DSL focus domain? | This data not explicitly, but use case are used into web application. |
| Technologies and Notations used in DSL development | Xtext Framework. |
| DSL type | Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | use the model transformation for convert RSL to Robot Language |
| System Under Testing (SUT) data are represented in the models and how they are represented | RSL constructs are logically classified according to two perspectives: abstraction level and specific concerns they address. The abstraction levels are: business, application, software and hardware levels. On the other hand, the concerns are: active structure (subjects), behaviour (actions), passive structure (objects), requirements, tests, other concerns, relations and sets. RSL specifications based on Use Cases can involve the definition of some views with the respective constructs and inherent relations: • DataEntity view: defines the structural entities that exist in an information system, commonly associated to data concepts captured and identified from the domain analysis. A Data Entity denotes an individual structural entity that might include the specification of attributes, foreign keys, and other check data constraints; • DataEntityCluster view: denotes a cluster of several structural entities that present logical arrangements among themselves; • Actor view: defines the participants of Use Cases or user stories. Represent end-users and external systems that interact directly with the system under study, and in some particular situations can represent timers that trigger the start of some Use Cases; • Use Case view: defines the use cases of a system under study. Traditionally a use case means a sequence of actions that one or more actors perform in a system to obtain a particular result. RSL supports the specification and generation of software tests, directly from requirements specifications. As showed in Figure 3.2, RSL provides a hierarchy of Test constructs and supports the specification of the following specializations of test cases [Sil18]: • DataEntityTest: apply equivalence class partitioning and boundary value analysis techniques over the domains defined for the DataEntities [BQ15] in RSL DataEntities; • UseCaseTest: explores multiple sequences of steps defined in RSL use cases’ scenarios, and associates data values to the involved data entities; • StateMachineTest: apply different algorithms to traverse the RSL state machines so that it is possible to define different test cases that correspond to valid or invalid paths through the respective state machine; • AcceptanceCriteriaTest: define acceptance criteria based on two distinct approaches: scenario-based (i.e., Given-When-Then pattern) or rule-based; this test case is applied generically to any type of RSL Requirement. |
| Benefits in using DSL | Can be used and adapted by different organizations with different processes or methodologies and supported by multiple types of software tools, traceability between requirements and test cases, Logs Creation. |
| Drawbacks in using DSL | Human intervention continues to be imperative. |
| Trends in using DSL | Extract informations using the browser extension automatically and generate tests scripts that may be executed through other test automation tools. |
| Others comments about study | RSL is a DSL that provides a comprehensive set of constructs that might be logically arranged into views according to specific concerns. These constructs are defined by linguistic patterns and represented textually according to concrete linguistic styles. But is most complexity |
| Data | Response |
|---|---|
| Title of Study | Web Application Model Generation through Reverse Engineering and UI Pattern Inferring. |
| Authors | Clara Sacramento, Ana C.R. Paiva.. |
| Year of publication | 2014. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | PARADIGM - DSL. |
| Use or propose a DSL? | Use. |
| DSL Features | Can be Reverse Engineering, define GUI testing models based on UI Test Patterns, modelling and testing environment, automatic test case generation, executes test cases and detailed execution reports. |
| Which DSL focus domain? | Web application. |
| Technologies and Notations used in DSL development | This data not is presented. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | This data not is presented. |
| System Under Testing (SUT) data are represented in the models and how they are represented | The PARADIGM language is comprised of elements and connectors. There are four types of elements: Init (to mark the beginning of a model), End (to mark the termination of a model), Structural (to structure the models in different levels of abstraction), and Behavioural (UI Test Patterns describing the testing goals). |
| Benefits in using DSL | Domain abstractions. |
| Drawbacks in using DSL | Dynamic Pages. |
| Trends in using DSL | Using others Algorithms to exploration algorithms and provides a better support for Dynamic Pages. |
| Others comments about study | This paper presents a dynamic reverse engineering approach that aims to extract part of the model of an existing web application through the identification of User Interface (UI) patterns. |
A78 - Orchestration of Domain Specific Test Languages with a Behavior Driven Development approach (2018)
| Data | Response |
|---|---|
| Title of Study | Orchestration of Domain Specific Test Languages with a Behavior Driven Development approach. |
| Authors | Robin Bussenot, Hervé Leblanc, and Christian Percebois. |
| Year of publication | 2018. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | ATA21-DSL / ATA42-DSL (are used in study). |
| Use or propose a DSL? | Use. |
| DSL Features | Specify behaviour for a specific avionic system aspect and generate Pivot Language code. In addition the Pivot Language generate execution code in Python. (Using model-to-model transformations) |
| Which DSL focus domain? | Avionic Domain. |
| Technologies and Notations used in DSL development | This data not is presented. |
| DSL type | ATA21/ATA42:Textual. |
| Technique/approach/method to ensure coverage of the system with test cases | Each language specify a aspect system by area expert. |
| System Under Testing (SUT) data are represented in the models and how they are represented | In ATA21: Each path from SUT to a leaf is represented by a DSTL statement. For example, the Increase statement can be instantiated in two manners, the first one to increase the temperature of an area (Increase the temperature of Expected_Temperature_Area1 up to 20.◦c) and the second one to increase the air flow rate of a set of ducts (Increase the air flow rate of Ducts_Area1 up to 0.2 kg/s). In ATA42: Procedures are composed by a set of test cases and test cases are completed by sentences divided into five kinds (Step, Check, Trace, Log and Reminder). The authors have used Natural Language Processing tools to provide statistics about verbs the most used for each kind of sentence. |
| Benefits in using DSL | ATA21: focuses on airflow control systems of an airplane. ATA42:focuses on Integrated Modular Avionic systems. |
| Drawbacks in using DSL | Not support all avionic system requirements. |
| Trends in using DSL | Design a new DSTL dedicated to another ATA. |
| Others comments about study | This study proposes a new approach for orchestration DSLs for avionic domain. |
| Data | Response |
|---|---|
| Title of Study | Pattern-Based Usability Testing (2017). |
| Authors | Fernando Dias, Ana C. R. Paiva. |
| Year of publication | 2017. |
| Study objective clearly stated? | Yes. |
| Are there enough data to assess the validity of the findings? If not, what are they missing? | All data are presented. |
| DSL Name | PARADIGM-DSL (This study propose a extension). |
| Use or propose a DSL? | Use. |
| DSL Features | Build models describing testing goals. The test cases generated from these models are then executed on a graphical user interface to find failures. |
| Which DSL focus domain? | Web Applications. |
| Technologies and Notations used in DSL development | This data not is presented in this study. |
| DSL type | Graphical. |
| Technique/approach/method to ensure coverage of the system with test cases | Using the navigation tree: The navigation tree is build using Breadth-first search algorithm. |
| System Under Testing (SUT) data are represented in the models and how they are represented | PARADIGM is a graphical DSL with elements and connectors. Besides structural elements to allow structuring models in different levels of abstraction, there are elements called Test Patterns that define generic test strategies aiming to test common recurrent behavior on different GUIs. |
| Benefits in using DSL | The test cases generated from these models are then executed on a graphical user interface to find failures. |
| Drawbacks in using DSL | Not support return transitions and probably has errors into SPAs. |
| Trends in using DSL | Include dynamic behavior of web applications. |
| Others comments about study | . |
University Federal of Pampa (UNIPAMPA), Campus Alegrete - RS
Developer: Yury Alencar
Research group: Laboratory of Empirical Studies in Software Engineering (LESSE)
Involved in Project: Yury Alencar, Elder Rodrigues, Rafael Oliveira, Maicon Bernardino and Fábio P. Basso.
SUMMARY
- Home REQUIRE UPDATE
- Roadmap
- Theoretical Reference
- Language Definition DEPRECATED
- How Modeling Your Application TODO
- Test Project Architecture TODO
Contacts:
- Yury Alencar: yuryalencar19@gmail.com
- Elder Rodrigues: eldermr@gmail.com
- Maicon Bernardino: bernardino@acm.org
- Fábio Basso: fabiopbasso@gmail.com