-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Alexander Kaiser
committed
Dec 10, 2018
1 parent
50e09ae
commit 3e74946
Showing
156 changed files
with
27,181 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# Minimal makefile for Sphinx documentation | ||
# | ||
|
||
# You can set these variables from the command line. | ||
SPHINXOPTS = | ||
SPHINXBUILD = sphinx-build | ||
SOURCEDIR = . | ||
BUILDDIR = _build | ||
|
||
# Put it first so that "make" without argument is like "make help". | ||
help: | ||
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) | ||
|
||
.PHONY: help Makefile | ||
|
||
# Catch-all target: route all unknown targets to Sphinx using the new | ||
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). | ||
%: Makefile | ||
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) |
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Sphinx build info version 1 | ||
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. | ||
config: c0a43dd133fcecbf9fdd5f5efd8b615c | ||
tags: 645f666f9bcd5a90fca523b33c5a78b7 |
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
########################## | ||
About Eclipse IoT-Testware | ||
########################## | ||
|
||
IoT-Testware Team | ||
================= | ||
`Who's involved <https://projects.eclipse.org/projects/technology.iottestware/who>`_ | ||
|
||
Conformance Test Methodology and Framework | ||
========================================== | ||
The IoT-Testware test suites will have a well-defined test suite structure (TSS) and a set of | ||
protocol implementation conformance statements (PICS) as well as | ||
protocol implementation extra information for testing (PIXIT). | ||
The work will follow the standardized approach as defined in ISO “Conformance Test Methodology and Framework” | ||
ISO 9646 and the best practices as described by ETSI White Paper No 3 “Achieving Technical Interoperability – the ETSI Approach”. | ||
|
||
.. image:: images/process.png | ||
:width: 500px | ||
:alt: Conformance Test Methodology | ||
:align: center | ||
|
||
Implementation | ||
============== | ||
The Eclipse IoT-Testware project provides standardized Abstract Test Suite (ATS) for popular IoT protocols. | ||
For the implementation of the ATS for CoAP and MQTT we apply `ETSI <www.etsi.org>`_ Test Methodology which | ||
is well-proven in standardizing and testing of telecommunication systems. | ||
|
||
Such an ATS contains of several parts which are required to implement the Conformance Test Methodology and Framework. | ||
But ATS, as the name says, are abstract, which means we need a system which executes the ATS. | ||
Just like Java code requires the JVM to be executed, an ATS requires in our case a `TTCN-3 <http://www.ttcn-3.org/>`_ runtime. | ||
As our TTCN-3 runtime we have chosen `Eclipse Titan <https://projects.eclipse.org/projects/tools.titan>`_ which can compile and run | ||
our ATS. | ||
|
||
.. image:: http://www.sioktatas.hu/doc/Titan_architecture.png | ||
:width: 500px | ||
:alt: Titan Architecture | ||
:align: center | ||
|
||
The Executable Test Suite (ETS) is, as the name states, is a test suite under execution, just like running Java code. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,111 @@ | ||
############### | ||
CoAP Test Suite | ||
############### | ||
|
||
.. ###> Abbreviations for this document | ||
.. |coap| replace:: :abbr:`CoAP (Constrained Application Protocol)` | ||
.. |sut| replace:: :abbr:`SUT (System Under Test)` | ||
.. |iut| replace:: :abbr:`IUT (Implementation Under Test)` | ||
.. |ts| replace:: :abbr:`TS (Test System)` | ||
.. |etsi| replace:: :abbr:`ETSI (European Telecommunications Standards Institute)` | ||
|
||
.. contents:: | ||
|
||
------------- | ||
CoAP Protocol | ||
------------- | ||
|
||
The |coap| protocol... | ||
|
||
*TODO:* brief introduction to coap + references | ||
|
||
------------------- | ||
Test Configurations | ||
------------------- | ||
The concrete CoAP test configurations are listed below: | ||
|
||
CoAP Test Configuration 01 | ||
========================== | ||
|
||
.. image:: images/coap_conf_01.png | ||
:width: 350px | ||
:alt: CoAP Test Configuration 01 | ||
:align: center | ||
|
||
CoAP Test Configuration 02 | ||
========================== | ||
|
||
.. image:: images/coap_conf_02.png | ||
:width: 350px | ||
:alt: CoAP Test Configuration 02 | ||
:align: center | ||
|
||
CoAP Test Configuration 03 | ||
========================== | ||
|
||
.. image:: images/coap_conf_03.png | ||
:width: 350px | ||
:alt: CoAP Test Configuration 03 | ||
:align: center | ||
|
||
CoAP Test Configuration 04 | ||
========================== | ||
|
||
.. image:: images/coap_conf_04.png | ||
:width: 350px | ||
:alt: CoAP Test Configuration 04 | ||
:align: center | ||
|
||
------------- | ||
Test Purposes | ||
------------- | ||
*TODO:* link to .tplan2 from GitHub and .pdf from |etsi| | ||
|
||
----------- | ||
Test System | ||
----------- | ||
*TODO: describe Test System* -> TTCN-3 code | ||
|
||
---------- | ||
Test Cases | ||
---------- | ||
You can find all CoAP Tests on GitHub. | ||
|
||
.. literalinclude:: code_blocks/TC_COAP_SERVER_001.txt | ||
|
||
------------------- | ||
Test Case Functions | ||
------------------- | ||
*TODO:* describe a CoAP Test Case Function | ||
|
||
.. literalinclude:: code_blocks/f_TC_COAP_SERVER_001.txt | ||
|
||
---------------- | ||
Translation Port | ||
---------------- | ||
*TODO:* Explain translation port | ||
|
||
----------------- | ||
Generic Functions | ||
----------------- | ||
*TODO:* describe MQTT_Functions module | ||
|
||
-------------- | ||
Test Templates | ||
-------------- | ||
*TODO:* describe templates | ||
|
||
------------------ | ||
Default Behaviours | ||
------------------ | ||
*TODO:* describe the default behaviours | ||
|
||
|
||
.. seealso:: | ||
|
||
`CoAP Plugtests 1: Guide <https://portal.etsi.org/cti/downloads/TestSpecifications/CoAP_IOT_TestSpecification_v016_20120321.pdf>`_ | ||
ETSI CTI Plugtests Guide (First Draft V0.0.16 2012-03) for achieving interoperability | ||
|
||
`CoAP Plugtests 1: Report <https://portal.etsi.org/Portals/0/TBpages/CTI/Docs/CoAP_Plugtest_1_TR_2012.pdf>`_ | ||
The 1st CoAP Plugtest was held from 24 to 25 March 2012 in Paris, France and was co-located with IETF#83. | ||
This event was jointly organized by ETSI, IPSO Alliance and the FP7 Probe-IT project1. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
############################ | ||
Protocol Conformance Testing | ||
############################ | ||
|
||
.. ###> Abbreviations for this document | ||
.. |mqtt| replace:: :abbr:`MQTT (MQ Telemetry Transport)` | ||
.. |iut| replace:: :abbr:`IUT (Implementation Under Test)` | ||
.. |tp| replace:: :abbr:`TP (Test Purpose)` | ||
.. |tp_term| replace:: :term:`Test Purpose` | ||
.. |tc| replace:: :abbr:`TC (Test Case)` | ||
.. |tss| replace:: :abbr:`TSS (Test Suite Structure)` | ||
.. |etsi| replace:: :abbr:`ETSI (European Telecommunications Standards Institute)` | ||
.. |iso| replace:: :abbr:`ISO (International Organization for Standardization)` | ||
|
||
.. contents:: | ||
|
||
General | ||
======= | ||
The |iso| standard for the methodology of conformance testing (ISO/IEC 9646-1 and ISO/IEC 9646-2) as well as | ||
the |etsi| rules for conformance testing (ETSI ETS 300 406) are used as a basis for the test methodology. | ||
|
||
To implement this methodology we require several intermediary artefacts. | ||
Those single artefacts break down the whole complexity of conformance testing | ||
into smaller pieces, each with a specific perspective on the problem. | ||
|
||
.. image:: images/process.png | ||
:width: 500px | ||
:alt: Conformance Test Methodology | ||
:align: center | ||
|
||
Test Suite Structure | ||
==================== | ||
In the first step we define a |tss| for a specific |iut|. | ||
|
||
*TODO: link to ETSI document* | ||
|
||
Test Configurations | ||
==================== | ||
*TODO: Why do we need Test configurations?* | ||
|
||
Test Purpose Catalogues | ||
======================= | ||
A |tp| (|tp_term|) is a formal description of a test case. A formal description in the form of a |tp| offers a possibility of describing | ||
the purpose of a test without having the later technical implementation in mind. Following the |tss| the tester is supported in | ||
systematically covering the complete |iut| specification. | ||
|
||
*TODO: link to ETSI document* | ||
|
||
The exemplary |tp| below shows a tabular representation for a simple |mqtt| test. | ||
|
||
============= =============================================== | ||
**TP-ID** TP_MQTT_BROKER_CONNECT_01 | ||
------------- ----------------------------------------------- | ||
**Selection** PICS_Broker | ||
------------- ----------------------------------------------- | ||
**Summary** The IUT MUST close the network connection if... | ||
------------- ----------------------------------------------- | ||
**Reference** [MQTT-2.2.2-1], [MQTT-2.2.2-2] | ||
------------- ----------------------------------------------- | ||
**Expected bahaviour** | ||
-------------------------------------------------------------- | ||
*initial condition* statement | ||
-------------------------------------------------------------- | ||
*ensure that* statement | ||
============================================================== | ||
|
||
IoT-Testware Test Suites | ||
======================== | ||
This steps focuses on a technical implementation of the TPs. We use `TTCN-3 <http://www.ttcn-3.org/>`_ and `Eclipse Titan <https://projects.eclipse.org/projects/tools.titan>`_ | ||
to implement each |tp| into a |tc| and orchestrate to executable test suites. | ||
|
||
.. toctree:: | ||
:maxdepth: 1 | ||
|
||
mqtt_test_suite.rst | ||
coap_test_suite.rst | ||
opc_test_suite.rst |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
####### | ||
Fuzzing | ||
####### | ||
|
||
*TODO:* What is fuzzing and how do we make use of fuzzing? | ||
|
||
.. image:: images/fuzzing_coap.png | ||
:width: 500px | ||
:alt: Fuzzing CoAP | ||
:align: center |
Oops, something went wrong.