mqtt_test_suite.rst

MQTT Test Suite

MQTT Protocol

A very brief summary of MQTT (MQ Telemetry Transport) from the FAQ

MQTT stands for MQ Telemetry Transport. It is a publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable networks.

Note

We provide an annotated version <./mqtt_spec> of the official MQTT specification which can be directly referenced (e.g. and )

Test Configurations

From a general and abstract perspective MQTT has two basic architectures for testing. This architecture directly reflects the choice of your SUT (System Under Test). We will call the first architecture Broker Testing. A MQTT Broker is the SUT (System Under Test) as shown in the figure blow:

The second major architecture we will call Client Testing as now, the Client is in focus as the SUT (System Under Test).

Now we can start to extract different configurations from the test architectures. The image below depicts the step of retrieving test configurations from the architecture:

The output of this exemplary step is a test configuration (CF01) where the Broker is the SUT (System Under Test) and the TS (Test System) takes the role of a MQTT Client.

The MQTT test suite uses four test configurations in order to cover the different test scenarios. In these configurations, the tester simulates one or several MQTT clients or brokers implementing the MQTT protocol.

ID:	MQTT_Conf_01
Description:	The MQTT Broker is the IUT (Implementation Under Test) and the TS (Test System) takes the role of a MQTT Client

ID:	MQTT_Conf_02
Description:	The MQTT Broker is the IUT (Implementation Under Test) and the TS (Test System) takes the role of multiple MQTT Clients.

ID:	MQTT_Conf_03
Description:	The MQTT Client is the IUT (Implementation Under Test) and the TS (Test System) takes the role of a MQTT Broker. For this configuration an optional UT (Upper Tester) might be required.

ID:	MQTT_Conf_04
Description:	As well the MQTT Broker as the MQTT Client, each is a IUT (Implementation Under Test) in this configuration. The part of the UT (Upper Tester) from the previous configuration is here replaced by a concrete application.

Test Purposes

TODO: link to .tplan2 from GitHub and .pdf from ETSI (European Telecommunications Standards Institute)

Test Purpose {
  TP Id TP_MQTT_Broker_CONNECT_001

  Test objective
  "The IUT MUST close the network connection if fixed header flags in CONNECT Control Packet are invalid"
  Reference
    "[MQTT-2.2.2-1], [MQTT-2.2.2-2], [MQTT-3.1.4-1], [MQTT-3.2.2-6]"
  PICS Selection PICS_BROKER_BASIC

  Expected behaviour
  ensure that {
    when {
        the IUT entity receives a CONNECT message containing
        header_flags indicating value '1111'B;
    }
    then {
        the IUT entity closes the TCP_CONNECTION
    }
  }
}

Test System

TODO: describe Test System -> TTCN-3 code

Test Cases

You can find all MQTT Tests on GitHub.

We will examine the procedure of a single TC (Test Case) in order to get the understanding of the code structure.

The code block below shows the TTCN-3 implementation of the TC (Test Case) TC_MQTT_BROKER_CONNECT_01 for the cohesive TP (Test Purpose) TP_MQTT_BROKER_CONNECT_01

code_blocks/TC_MQTT_BROKER_CONNECT_01.txt

Let's have a deeper look into the details of a TC (Test Case). The first block comment contains only two TTCN-3 documentation tags but these give us a direct connection between a TC (Test Case), a TP (Test Purpose), and the MQTT specification.

code_blocks/TC_MQTT_BROKER_CONNECT_01.txt

The signature of a TTCN-3 TC (Test Case) contains many information which help us to reflect the Test Architecture.

code_blocks/TC_MQTT_BROKER_CONNECT_01.txt

We have a distinct name for the TC (Test Case) which can be easy mapped to it's cohesive TP (Test Purpose). (see in the MQTT |tp| catalogue for TP_MQTT_BROKER_CONNECT_001) The signature tells us also, that this TC (Test Case) will be executed on a Client (runs on MQTT_Client).

The body of the TC (Test Case) is used to initialize the test configuration and start the TC (Test Case) behaviour which is wrapped into a single function.

code_blocks/TC_MQTT_BROKER_CONNECT_01.txt

Test Case Functions

How does a TC (Test Case) function look like?

code_blocks/f_TC_MQTT_BROKER_CONNECT_01.txt

While syntactically this function might appear confusing, though the semantic behind is quite powerful and simple. Let's go through the implemented TC (Test Case) and the according TP (Test Purpose).

Translation Port

TODO: Explain translation port

Generic Functions

TODO: describe MQTT_Functions module

Test Templates

TODO: describe templates

Default Behaviours

TODO: describe the default behaviours

MQTT Interop Test Day in Burlingame, CA - March 17, 2014

The goal was to have as many different MQTT client and server implementations participate in interoperability testing to validate the implementation of the upcoming OASIS MQTT standard.

MQTT Interop Test Day in Ottawa, Canada – April 8, 2014

MQTT Test Day Demonstrates Successful Interoperability for the Internet of Things

MQTT Interop Test Day in Burlingame, CA - March 9, 2015

The goal was to have as many different MQTT client and server implementations participate in interoperability testing to validate the implementation of the OASIS MQTT 3.1.1 specification.