diff --git a/TESTS/netsocket/README.md b/TESTS/netsocket/README.md index 7c45460677b..654a25b2e4c 100644 --- a/TESTS/netsocket/README.md +++ b/TESTS/netsocket/README.md @@ -18,14 +18,15 @@ The target for this plan is to test: - [UDPSocket](https://github.com/ARMmbed/mbed-os/blob/master/features/netsocket/UDPSocket.h). - [TCPSocket](https://github.com/ARMmbed/mbed-os/blob/master/features/netsocket/TCPSocket.h). - [TLSSocket](https://github.com/ARMmbed/mbed-os/blob/master/features/netsocket/TLSSocket.h). +- [DNS](https://github.com/ARMmbed/mbed-os/blob/master/features/netsocket/DNS.h). -Reference documentation: https://os.mbed.com/docs/latest/reference/network-socket.html +See [Network Socket Documentation](https://os.mbed.com/docs/mbed-os/latest/apis/network-socket.html) for reference. Tools to use ---------------- - Mbed OS. -- Standard Mbed OS development tools as described in https://os.mbed.com/docs/latest/tools/index.html. +- Standard Mbed OS development tools as described in [Arm Mbed tools overview](https://os.mbed.com/docs/latest/tools/index.html). - Test server. These test cases themselves do not require any special tooling, other than @@ -227,6 +228,20 @@ pass the test if the driver implements the feature in question. | 60 | TLSSOCKET_SIMULTANEOUS_TEST | SHOULD | | 61 | TLSSOCKET_ECHOTEST_BURST | SHOULD | | 62 | TLSSOCKET_ECHOTEST_BURST_NONBLOCK | SHOULD | +| 63 | ASYNCHRONOUS_DNS | MUST | +| 64 | ASYNCHRONOUS_DNS_CACHE | MUST | +| 65 | ASYNCHRONOUS_DNS_CANCEL | MUST | +| 66 | ASYNCHRONOUS_DNS_EXTERNAL_EVENT_QUEUE | MUST | +| 67 | ASYNCHRONOUS_DNS_INVALID_HOST | MUST | +| 68 | ASYNCHRONOUS_DNS_NON_ASYNC_AND_ASYNC | MUST | +| 69 | ASYNCHRONOUS_DNS_SIMULTANEOUS_CACHE | MUST | +| 70 | ASYNCHRONOUS_DNS_SIMULTANEOUS_REPEAT | MUST | +| 71 | ASYNCHRONOUS_DNS_SIMULTANEOUS | MUST | +| 72 | ASYNCHRONOUS_DNS_TIMEOUTS | MUST | +| 73 | SYNCHRONOUS_DNS | MUST | +| 74 | SYNCHRONOUS_DNS_CACHE | MUST | +| 75 | SYNCHRONOUS_DNS_INVALID_HOST | MUST | +| 76 | SYNCHRONOUS_DNS_MULTIPLE | MUST | @@ -1654,6 +1669,311 @@ through a dedicated socket Echo server returns data to both threads and received data matches to send data. The additional thread isn't stopped prematurely. +Test cases for DNS class +--------------------------- + +### ASYNCHRONOUS_DNS + +**Description:** + +Verify basic functionality of asynchronous DNS. Call `NetworkInterface::gethostbyname_async()` with a valid host name and verify result. + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Call `gethostbyname_async()` with a valid host name and a callback. +2. Verify that callback is called with correct parameters. + +**Expected result:** + +Callback is called with NSAPI_ERROR_OK and IP address. + +### ASYNCHRONOUS_DNS_SIMULTANEOUS + +**Description:** + +Verify that simultaneous asynchronous DNS queries work correctly. Call `NetworkInterface::gethostbyname_async()` in a row 6 times with a different host names. Wait for all requests to complete and verify result. Cache shall not contain host names used in asynchronous request. + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Call `gethostbyname_async()` in a row 6 times with a different host names. Host names shall not be found from cache. +2. Verify that last `gethostbyname_async()` operation is rejected since there is room only for 5 simultaneous operations. +3. Verify that callback is called with correct parameters 5 times. + +**Expected result:** + +Sixth `gethostbyname_async()` is rejected. Callback is called with NSAPI_ERROR_OK and IP address 5 times. + +### ASYNCHRONOUS_DNS_SIMULTANEOUS_CACHE + +**Description:** + +Verify that the caching of DNS results works correctly with simultaneous asynchronous DNS queries. Call `NetworkInterface::gethostbyname_async()` in a row 6 times with a different host names. Wait for all requests to complete and verify result. Cache shall contain at least one host name used in asynchronous request. This can be achieved e.g. by running test "Asynchronous DNS simultaneous" before this test and using same host names in this run. + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Call `gethostbyname_async()` in a row 6 times with a different host names. At least one host name shall be found from cache. +2. Verify that callback is called with correct parameters 6 times. + +**Expected result:** + +Callback is called with NSAPI_ERROR_OK and IP address 6 times. + +### ASYNCHRONOUS_DNS_CACHE + +**Description:** + +Verify that the caching of DNS results works correctly. Call `NetworkInterface::gethostbyname_async()` 5 times with the same host name and verify result after each request. For first request, cache shall not contain the host name. Verify that first request completes slower than the requests made after it (where the response is found from cache). + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Call `gethostbyname_async()` with a host name. For first request, host name shall not be found from cache. +2. Verify that callback is called with correct parameters. +3. Repeat the sequence 4 times using the same host name. +4. For each request, calculate how long time it takes for DNS query to complete. + +**Expected result:** + +Callback is called with NSAPI_ERROR_OK and IP address 5 times. First request shall complete slower than the requests made after it (where the response is found from cache). + +### ASYNCHRONOUS_DNS_NON_ASYNC_AND_ASYNC + +**Description:** + +Verify that non-asynchronous i.e. blocking DNS queries and asynchronous i.e. non-blocking queries work at the same time. Call `NetworkInterface::gethostbyname_async()`. Right after that make 6 non-asynchronous `NetworkInterface::gethostbyname()` calls with different host names. + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Call `gethostbyname_async()` with a host name. Host name shall not be found from cache. +2. Call `gethostbyname()` 6 times with a different host names (none of the names shall be same as in step 1). +3. Verify that each `gethostbyname()` returns success. +4. Verify that the asynchronous callback is called with correct parameters. + +**Expected result:** + +All operations shall return NSAPI_ERROR_OK and IP address. + +### ASYNCHRONOUS_DNS_CANCEL + +**Description:** + +Verify that asynchronous DNS query cancel works correctly. Call `NetworkInterface::gethostbyname_async()` in a row 6 times with a different host names. Cache shall contain 3 host names used in requests. This can be achieved e.g. by running test "Asynchronous DNS non-asynchronous and asynchronous" before this test and using same host names in this run. For each request that was given an unique id, call cancel. Verify that callback is not called for cancelled requests. + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Call `gethostbyname_async()` in a row 6 times with a different host names. Cache shall contain in maximum 3 host names used in requests. +2. Call `gethostbyname_async_cancel()` for each request that was given an unique id. +3. Verify that for cancelled requests, callback is not called. +4. Verify that for other request, callback is called. + +**Expected result:** + +Callback shall be called only for requests that were not cancelled. + +### ASYNCHRONOUS_DNS_EXTERNAL_EVENT_QUEUE + +**Description:** + +Verify that providing an external event queue works correctly. Define a thread and an event queue running on it. Define a DNS call in callback function that uses the event queue (call_in_callback_cb_t). Enable external event queue. Call `NetworkInterface::gethostbyname_async()` in a row 6 times with a different host names. + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Define a thread and an event queue running on it. +2. Define a DNS call in callback function that uses the event queue (call_in_callback_cb_t). +3. Start thread and event queue. +4. Set DNS callback function using the `nsapi_dns_call_in_set()` call. +5. Call `gethostbyname_async()` in a row 6 times with a different host names. Host names shall not be found from cache. +6. Verify that last `gethostbyname_async()` operation is rejected since there is room only for 5 simultaneous operations. +7. Verify that callback is called with correct parameters 5 times. +8. Restore default DNS callback function using the `nsapi_dns_call_in_set()` call. + +**Expected result:** + +Sixth `gethostbyname_async()` is rejected. Callback is called with NSAPI_ERROR_OK and IP address 5 times. + +### ASYNCHRONOUS_DNS_INVALID_HOST + +**Description:** + +Verify that DNS failure error is provided for invalid hosts. Call `NetworkInterface::gethostbyname_async()` in a row 6 times with a different host names. First, third and fifth host name shall be invalid. + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Call `gethostbyname_async()` in a row 6 times with a different host names. Host names shall not be found from cache. First, third and fifth host name shall be invalid. +2. Verify that last `gethostbyname_async()` operation is rejected since there is room only for 5 simultaneous operations. +3. Verify that callback is called with correct parameters 5 times. + +**Expected result:** + +Sixth `gethostbyname_async()` is rejected. Callback is called with NSAPI_ERROR_DNS_FAILURE for first, third and fifth host name. Callback is called with NSAPI_ERROR_OK and IP address for second and fourth host name. + +### ASYNCHRONOUS_DNS_TIMEOUTS + +**Description:** + +Test DNS timeouts using an external event queue that is modified to timeout the events faster that standard event queue. In this test event queue shall not delay events, instead it handles those immediately. Call `NetworkInterface::gethostbyname_async()` in a row 6 times with a different host names. All or some of the request shall timeout and timeout return value is returned. + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Define a thread and an event queue running on it. +2. Define a DNS call in callback function that uses the event queue (call_in_callback_cb_t). Callback function shall not delay callbacks; instead it shall handle those immediately. +3. Start thread and event queue. +4. Set DNS callback function using the `nsapi_dns_call_in_set()` call. +5. Call `gethostbyname_async()` in a row 6 times with a different host names. Host names shall not be found from cache. +6. Verify that last `gethostbyname_async()` operation is rejected since there is room only for 5 simultaneous operations. +7. Verify that callback is called with correct parameters 5 times. + +**Expected result:** + +Sixth `gethostbyname_async()` is rejected. At least for one operation, callback is called with NSAPI_ERROR_TIMEOUT value. + +### ASYNCHRONOUS_DNS_SIMULTANEOUS_REPEAT + +**Description:** + +Verify that simultaneous asynchronous DNS queries work correctly when repeated in sequence. Call `NetworkInterface::gethostbyname_async()` in a row 5 times with a different host names. Wait for all requests to complete and verify result. Repeat the procedure 100 times. + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Call `gethostbyname_async()` in a row 5 times with a different host names. +2. Verify that callback is called with correct parameters 5 times for first operation. +3. Repeat 100 times steps 1 to 2. + +**Expected result:** + +Callback is called with NSAPI_ERROR_OK and IP address 5 times for every repeat. + +### SYNCHRONOUS_DNS + +**Description:** + +Verify basic functionality of synchronous DNS. Call `NetworkInterface::gethostbyname()` with a valid host name and verify result. + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Call `gethostbyname()` with a valid host name. +2. Verify that address was resolved and return value was valid. + +**Expected result:** + +Return value is NSAPI_ERROR_OK and IP address is obtained from the function call. + +### SYNCHRONOUS_DNS_MULTIPLE + +**Description:** + +Verify basic functionality of synchronous DNS. Call `NetworkInterface::gethostbyname()` with a list of 6 host names and verify result. + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Call `gethostbyname()` with a list of 6 host names +2. Verify that each of the addresses was resolved and return value was valid. + +**Expected result:** + +Return value is NSAPI_ERROR_OK and IP addresses are obtained from the function call. + +### SYNCHRONOUS_DNS_CACHE + +**Description:** + +Verify that the caching of DNS results works correctly. Call `NetworkInterface::gethostbyname()` 5 times with the same host name and verify result after each request. For first request, cache shall not contain the host name. Verify that first request completes slower than the requests made after it (where the response is found from cache). + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Call `gethostbyname()` with a host name. For the first request, host name shall not be found from cache. +2. Verify that address was resolved and return value was valid. +3. Repeat the sequence 4 times using the same host name. +4. For each request, calculate how long time it takes for DNS query to complete. + +**Expected result:** + +Return value is NSAPI_ERROR_OK and IP address is obtained from the function call 5 times. First request shall complete slower than the requests made after it (where the response is found from cache). + +### SYNCHRONOUS_DNS_INVALID_HOST + +**Description:** + +Verify that DNS failure error is provided for invalid hosts. Call `NetworkInterface::gethostbyname()` in a row 6 times with a different host names. First, third and fifth host name shall be invalid. + +**Preconditions:** + +1. Network interface is initialised. +2. Network connection is up. + +**Test steps:** + +1. Call `gethostbyname()` in a row 6 times with a different host names. Host names shall not be found from cache. First, third and fifth host name shall be invalid. +2. Verify that return value was valid and for valid hostnames the address was resolved 6 times. + +**Expected result:** + +Return value is NSAPI_ERROR_DNS_FAILURE for first, third and fifth host name. Return value is NSAPI_ERROR_OK and IP address is obtained for second and fourth host name. + Subset for driver test ---------------------- @@ -1677,4 +1997,4 @@ Subset for driver test ### For socket layer driver (AT-driven, external IP stack): -All Socket, UDPSocket, TCPSocket and TLSSocket testcases. +All Socket, UDPSocket, TCPSocket and TLSSocket testcases. \ No newline at end of file diff --git a/TESTS/network/emac/README.md b/TESTS/network/emac/README.md index 0a63709bb8b..3bcab0f042d 100644 --- a/TESTS/network/emac/README.md +++ b/TESTS/network/emac/README.md @@ -8,7 +8,7 @@ To configure a device to be a CTP echo server, you need to enable the `echo-serv ## Other configuration options -Targets with connectivity set the target.network-default-interface-type configuration variable appropriately, either to their only interface or their most-commonly-used one. For targets that provide more than one type of connectivity, you may choose the default by overriding the target.network-default-interface-type configuration variable. +Targets with connectivity set the `target.network-default-interface-type` configuration variable appropriately, either to their only interface or their most commonly used one. For targets that provide more than one type of connectivity, you may choose the default by overriding the `target.network-default-interface-type` configuration variable. For Ethernet, if you want to overrride the default, set the `json` configuration to: @@ -21,7 +21,7 @@ For Ethernet, if you want to overrride the default, set the `json` configuration } ``` -For Wi-Fi set the `json` configuration to: +For Wi-Fi, set the `json` configuration to: ``` "target_overrides": { @@ -35,7 +35,24 @@ For Wi-Fi set the `json` configuration to: } ``` -For Wi-Fi you also need to configure Wi-Fi SSID and security options to the configuration file. +For Wi-Fi, you also need to configure Wi-Fi SSID and security options to the configuration file. + + +Test case priorities +-------------------- + +Please refer to the following table for priorities of test cases. Priorities are labeled as MUST and SHOULD. MUST means this is a requirement and therefore mandatory to pass the test. SHOULD means it is recommended to pass the test if the driver implements the feature in question. + +| | Test case | Priority | +|-----|-----------------------------------------|----------| +| 1 | EMAC initialize | MUST | +| 2 | EMAC broadcast | MUST | +| 3 | EMAC unicast | MUST | +| 4 | EMAC unicast frame length | MUST | +| 5 | EMAC unicast burst | MUST | +| 6 | EMAC unicast long | MUST | +| 7 | EMAC multicast filter | MUST | +| 8 | EMAC memory | MUST | ## Example commands @@ -107,90 +124,174 @@ To verify whether the echo server is receiving CTP Ethernet frames, enable `echo ### EMAC initialize -The test case initializes the EMAC driver and the test network stack. +**Description:** -The EMAC test environment uses the test network stack as the default stack. To enable the stack, set the `nsapi.default-stack` option in the `json` file of the test environment application to value `TEST`. +The test case initializes and connects the EMAC driver and the test network stack. -The test network stack is a bare minimum implementation and has the functionality needed to set up the network interface. The test network stack is defined in the `emac_TestNetworkStack.h` and `emac_TestNetworkStack.cpp` files. The stack uses the test memory manager for the EMAC. The test memory manager is defined in the `emac_TestMemoryManager.h` and `emac_TestMemoryManager.cpp` files. Message buffers sent to the EMAC in `link_out()` are allocated from the buffer pool of the test memory manager. The test memory manager pool allocation unit (buffer size) is 610 bytes. +**Precondition:** -The initialization test constructs the network interface and connects to it. The test network stack and the EMAC are bound to the network interface using `get_default_instance()` calls to the stack and to the EMAC. +The device is ready to be connected. (The ethernet cable is plugged in, or a Wi-Fi base station is available). -After the construction, the network interface is connected. A connect call triggers a set up call to the test network stack. The set up call triggers a call to `emac_if_init()` function in the EMAC initialization test case. +**Test steps:** -The `emac_if_init()` function of the test case configures and powers up the EMAC. +1. Constructs the network interface. +2. Connects the network interface. -The configuration steps are: +**Expected result:** -* Setting the test memory manager for the EMAC. -* Setting the EMAC link input and state callbacks to call the test environment input and state callback handlers. -* Reading and setting the Ethernet MAC address. +The network interface is connected. ### EMAC broadcast -1. Sends three CTP broadcast messages (100 bytes each) -2. Waits for three seconds -3. Sends three CTP broadcast messages (60 bytes each). -4. Listens for the CTP echo server responses. -5. Stores the addresses of the echo servers if replies are received. +**Description:** + +The test case tests basic broadcast functionality and resolves CTP echo server MAC address. + +**Precondition:** + +The network interface is connected. -The test case passes if there are no responses from the echo server, but further test cases are skipped. +**Test steps:** + +1. Sends three CTP broadcast messages (100 bytes each). +2. Waits for three seconds. +3. Sends three CTP broadcast messages (60 bytes each). +4. Listens for the CTP echo server responses. +5. Stores the addresses of the echo servers if replies are received. + +**Expected result:** + +The echo server replies to broadcast messages. For each sent broadcast message, a reply waits for three seconds. If the reply is not received, the reply is sent six more times before the test fails. ### EMAC unicast -1. Sends three CTP unicast messages (100 bytes each) to the CTP echo server. -2. Verifies that all are replied. +**Description:** + +The test case tests basic unicast functionality, and the CTP echo server replies to unicast messages. + +**Precondition:** + +The network interface is connected. + +**Test steps:** + +1. Sends three CTP unicast messages (100 bytes each) to the CTP echo server. +2. Verifies all replies. + +**Expected result:** + +The echo server replies to unicast messages. For each sent unicast message, a reply waits for three seconds. If the reply is not received, it is sent five more times before the test fails. ### EMAC unicast frame length - -1. Sends CTP unicast messages with Ethernet message length from 100 bytes to the maximum defined by the MTU of the EMAC with 50 bytes increments. -2. Verifies that all are replied. + +**Description:** + +The test case tests Ethernet frame lengths. + +**Preconditions:** + +The network interface is connected. + +**Test steps:** + +1. Sends CTP unicast messages with Ethernet message length from 100 bytes to the maximum defined by the MTU of the EMAC with 50-byte increments. +2. Verifies all replies. + +**Expected result:** + +The echo server replies to messages that are sent to it. For each sent unicast message, a reply waits for 500ms. If the reply is not received, it is sendt again five more times before the test fails. ### EMAC unicast burst - -1. Sends CTP unicast messages with Ethernet message length from 100 bytes to the maximum defined by the MTU of the EMAC with 50 bytes increments. -2. Repeats the sending 10 times. -3. Verifies that all are replied. + +**Description:** + +The test case tests Ethernet echoing at full speed. + +**Precondition:** + +The network interface is connected. + +**Test steps:** + +1. Sends CTP unicast messages with Ethernet message length from 100 bytes to the maximum defined by the MTU of the EMAC with 50-byte increments. +2. Repeats the sending 10 times. +3. Verifies all replies. + +**Expected result:** + +The echo server replies to messages that are sent to it. For each sent unicast message, a reply waits for five seconds. If the reply is not received, it is sent again five more times before the test fails. ### EMAC unicast long - -1. Sends CTP unicast messages with random Ethernet message length. -2. Repeats the sending 50000 times. -3. Verifies that all are replied. + +**Description:** + +Test case tests Ethernet echoing at full speed for an extended time. + +**Preconditions:** + +The network interface is connected. + +**Test steps:** + +1. Sends CTP unicast messages with random Ethernet message length. +2. Repeats the sending 50,000 times. +3. Verifies all replies. + +**Expected result:** + +The echo server replies to messages that are sent to it. For each sent unicast message, a reply waits for 350ms. If the reply is not received, it is sent five more times before the test fails. ### EMAC multicast filter - -Tests multicast filtering. Multicast filtering is an optional feature for the EMAC. The test does not fail if filtering is not implemented. -The multicast testing requests the CTP echo server to forward the CTP messages to a specified multicast address as the destination address. +**Description:** + +The test case tests multicast filtering. Multicast filtering is an optional feature for the EMAC. The test does not fail if filtering is not implemented. + +**Precondition:** + +The network interface is connected. -Test steps: +**Test steps:** -1. Using unicast, verify that the echo server responses are received. -2. Set the IPv6 multicast filter address and the echo server reply (forward) address to different values. Check if the echo response is filtered. -3. Set the IPv6 multicast filter address and the echo server reply address to same value. Check that the response is not filtered. -4. Set the IPv4 multicast filter address and the echo server reply address to different values. Check if the response is filtered. -5. Set the IPv4 multicast filter address and the echo server reply address to same value. Check that the response is not filtered. -6. Enable the receiving of all multicasts. Check that the response is not filtered. +1. Using unicast, verify that the echo server responses are received. +2. Set the IPv6 multicast filter address and the echo server reply (forward) address to different values. Check if the echo response is filtered. +3. Set the IPv6 multicast filter address and the echo server reply address to same value. Check that the response is not filtered. +4. Set the IPv4 multicast filter address and the echo server reply address to different values. Check if the response is filtered. +5. Set the IPv4 multicast filter address and the echo server reply address to same value. Check that the response is not filtered. +6. Enable the receiving of all multicasts. Check that the response is not filtered. + +**Expected result:** + +The echo server replies are received as expected by the test step. Supporting of the filtering will affect what messages are received. ### EMAC memory -Tests memory manager out-of-memory situations. The test case configures the test memory manager to reject memory buffer allocations made by the EMAC. Memory buffer allocations are divided into output and input memory allocations: +**Description:** + +The test case tests memory manager out-of-memory situations. The test case configures the test memory manager to reject memory buffer allocations made by the EMAC. Memory buffer allocations are divided into output and input memory allocations: -* The output memory allocations are the ones made by the EMAC in the `link_out()` function called by the network stack (test case). -* The input memory allocations are other memory allocations made by the EMAC. +- The output memory allocations are the ones made by the EMAC in the ``\`link\_out()\`` function called by the network stack (test case). +- The input memory allocations are other memory allocations made by the EMAC. Depending on the EMAC implementation, it may or may not allocate memory manager buffers in the link output function. If the memory manager buffers are not allocated, disabling the link output memory allocations in the test does not affect the functionality. -In each test step, the test case sends CTP unicast messages with Ethernet message length from 100 bytes to the maximum defined by the MTU of the EMAC with 50 bytes increments. Memory buffers sent to the EMAC in the `link_out()` function are forced to be non-aligned in this test case. +In each test step, the test case sends CTP unicast messages with Ethernet message length from 100 bytes to the maximum defined by the MTU of the EMAC with 50-byte increments. Memory buffers sent to the EMAC in the ``\`link\_out()\`` function are forced to be nonaligned in this test case. + +**Precondition:** + +The network interface is connected. + +**Test steps:** -Test steps: +1. Memory buffer allocations are allowed. Verify that echo server responses are received. +2. Disable input memory buffer allocations. The echo server responses should not be received. +3. Allow memory buffer allocations. Verify that the echo server responses are received. +4. Disable output memory buffer allocations. The echo server responses may or may not be received depending on the EMAC link out implementation. +5. Allow memory buffer allocations. Verify that the echo server responses are received. +6. Disable input and output memory buffer allocations. The echo server responses should not be received. +7. Allow memory buffer allocations. Verify that the echo server responses are received. +8. Allocate memory buffers that are sent to the EMAC in link out from the heap (contiguous memory). Verify that the echo server responses are received. -1. Memory buffer allocations are allowed. Verify that echo server responses are received. -2. Disable input memory buffer allocations. The echo server responses should not be received. -3. Allow memory buffer allocations. Verify that the echo server responses are received. -4. Disable output memory buffer allocations. The echo server responses may or may not be received depending on the EMAC link out implementation. -5. Allow memory buffer allocations. Verify that the echo server responses are received. -6. Disable input and output memory buffer allocations. The echo server responses should not be received. -7. Allow memory buffer allocations. Verify that the echo server responses are received. -8. Allocate memory buffers that are sent to the EMAC in link out from the heap (contiguous memory). Verify that the echo server responses are received. +**Expected result:** +The echo server replies to messages that are sent to it based on whether the driver can allocate memory for the frame or not. For each sent unicast message, a reply waits for 500ms. If the reply is not received, it is sent another three times. If the test expects that memory should be available and sending fails three times, the test fails. diff --git a/TESTS/network/interface/README.md b/TESTS/network/interface/README.md new file mode 100644 index 00000000000..90e912dfda4 --- /dev/null +++ b/TESTS/network/interface/README.md @@ -0,0 +1,164 @@ +Network interface test plan +======================== + +Target API +---------- + +The goal of this plan is to test [NetworkInterface.h](https://github.com/ARMmbed/mbed-os/blob/master/features/netsocket/NetworkInterface.h) + +Tools needed +---------------- + +- Mbed OS. +- Standard Mbed OS development tools as described in the [Arm Mbed tools overview](https://os.mbed.com/docs/latest/tools/index.html). + +Test environment +---------------- + +[Greentea](https://os.mbed.com/docs/mbed-os/latest/tools/greentea-testing-applications.html). + +Test case priorities +-------------------- + +Please refer to the following table for priorities of test cases. Priorities +are labeled as MUST and SHOULD. MUST means this is a requirement and +therefore mandatory to pass the test. SHOULD means it is recommended to +pass the test if the driver implements the feature in question. + +| | Test case | Priority | +|-----|-----------------------------------------|----------| +| 1 | NETWORKINTERFACE_CONN_DISC_REPEAT | MUST | +| 2 | NETWORKINTERFACE_STATUS | MUST | +| 3 | NETWORKINTERFACE_STATUS_NONBLOCK | MUST | +| 4 | NETWORKINTERFACE_STATUS_GET | MUST | + +Building test binaries +---------------------- + +For testing the board and driver, test against the Mbed OS +master branch to get the most recent, up-to-date test cases and drivers. + +To create a build environment: + +```.sh +mbed new network_interface_test +cd network_interface_test +cd mbed-os +git checkout master +cd .. +``` + +Now build test binaries: + +```.sh +mbed test --compile -t -m -n mbed-os-tests-network-interface +``` + +Running tests +------------- + +When device is connected to network, or if a wireless device is near +the access point: + +```.sh +mbed test -n mbed-os-tests-network-interface +``` + +Test cases for NetworkInterface class +------------------------------------- + +### NETWORKINTERFACE_CONN_DISC_REPEAT + +**Description:** + +Test `NetworkInterface::connect()` and `NetworkInterface::disconnect()` + +**Precondition:** + +Network interface is initialized. + +**Test steps:** + +1. Connect interface. +2. Disconnect interface. +3. Repeat connect and disconnect steps 1 to 2 four times. + +**Expected result:** + +`Connect()` and `disconnect()` calls return `NSAPI_ERROR_OK`. + +### NETWORKINTERFACE_STATUS + +**Description:** + +Test `NetworkInterface::attach()` and status indications. + +**Preconditions:** + +1. Network interface is initialized. +2. Status callback is attached to the network interface. +3. Network interface is on blocking mode. + +**Test steps:** + +1. Connect interface. +2. Check that interface indicates status `NSAPI_STATUS_CONNECTING`. +3. Optional: Check that interface indicates status `NSAPI_STATUS_LOCAL_UP` (whether this is indicated depends on network interface type). +4. Check that interface indicates status `NSAPI_STATUS_GLOBAL_UP`. +5. Disconnect interface. +6. Check that interface indicates status `NSAPI_STATUS_DISCONNECTED`. +7. Repeat connect and disconnect steps 1 to 6 four times. + +**Expected result:** + +Callback statuses are correct. `Connect()` and `disconnect()` calls return `NSAPI_ERROR_OK`. Note: The network interface may indicate the same status several times in a row, depending on the network interface type. This will not fail the test. + +### NETWORKINTERFACE_STATUS_NONBLOCK + +**Description:** + +Test `NetworkInterface::attach()` and status indications on nonblocking mode. + +**Preconditions:** + +1. Network interface is initialized. +2. Status callback is attached to the network interface. +3. Network interface is on nonblocking mode. + +**Test steps:** + +1. Connect interface. +2. Check that interface indicates status `NSAPI_STATUS_CONNECTING`. +3. Optional: Check that interface indicates status `NSAPI_STATUS_LOCAL_UP` (whether this is indicated depends on network interface type). +4. Check that interface indicates status `NSAPI_STATUS_GLOBAL_UP`. +5. Disconnect interface. +6. Check that interface indicates status `NSAPI_STATUS_DISCONNECTED`. +7. Repeat connect and disconnect steps 1 to 6 four times. + +**Expected result:** + +Callback statuses are correct. `Connect()` and `disconnect()` calls return `NSAPI_ERROR_OK`. Note: The network interface may indicate the same status several times in a row, depending on the network interface type. This will not fail the test. + +### NETWORKINTERFACE_STATUS_GET + +**Description:** + +Test `NetworkInterface::get_connection_status()`. + +**Preconditions:** + +1. Network interface is initialized. +2. Network interface is on blocking mode. + +**Test steps:** + +1. Check that `get_connection_status()` returns status `NSAPI_STATUS_DISCONNECTED`. +2. Connect interface. +3. Poll the `get_connection_status()` until it returns status `NSAPI_STATUS_GLOBAL_UP`. +4. Disconnect interface. +5. Check that `get_connection_status()` returns status `NSAPI_STATUS_DISCONNECTED`. +6. Repeat connect and disconnect steps 2 to 5 four times. + +**Expected result:** + +`Connect()` and `disconnect()` calls return `NSAPI_ERROR_OK`. The right status is returned by `get_connection_status()`. diff --git a/TESTS/network/wifi/README.md b/TESTS/network/wifi/README.md new file mode 100644 index 00000000000..31aeb0525b7 --- /dev/null +++ b/TESTS/network/wifi/README.md @@ -0,0 +1,501 @@ +Wi-Fi test plan +======================== + +This is a test plan for Mbed OS Wi-Fi API. + +Target API +---------- + +The goal of this plan is to test the [WiFiInterface](https://github.com/ARMmbed/mbed-os/blob/master/features/netsocket/WiFiInterface.h) class. + +Tools needed +---------------- + +- Mbed OS. +- Secure Wi-Fi access point. +- Unsecure Wi-Fi access point. + +This test plan is written in a tool-agnostic way and does not specify how to write, build or run test cases. + +Test environment +---------------- + +The general test environment consists of DUTs, base stations, a network connection and a test server: + +![Wi-Fi](../../netsocket/wifi_environment.png) + +- All DUTS should be in RAAS. +- Two Wi-Fi APs required: + - Nonsecured one: The SSID is later referred to as ``. + - Secured one: The SSID is later referred to as ``, password referred as ``. +- IPv4 and IPv6 must be routable between the Wi-Fi network IP address space and echo server. +- Firewall must not block test traffic to and from the test server **echo.mbedcloudtesting.com**. + - Echo Protocol, [RFC 862](https://tools.ietf.org/html/rfc862) is enabled in both TCP and UDP. Port 7. + - Discard Protocol, [RFC 863](https://tools.ietf.org/html/rfc863) is enabled in both TCP and UDP. Port 9. + - Character generator protocol, [RFC 864](https://tools.ietf.org/html/rfc864) is enabled in both TCP and UDP. Port 19. + - Daytime protocol, [RFC 867](https://tools.ietf.org/html/rfc867) in both TCP and UDP. Port 13. + - Time protocol, [RFC 868](https://tools.ietf.org/html/rfc868) in both TCP and UDP. Port 37. +- Channels to be used must be different for both APs. For secure on channel number is later referred as `` and for unsecure on ``. +- MAC addresses of Wi-Fi APs must be known. These are later referred to as `` and ``. + +**NOTE:** This document refers to an echo server because it is a requirement for running Socket API tests. The test cases defined in this document do not directly use it. + +Test case priorities +-------------------- + +Please refer to the following table for priorities of test cases. Priorities are labeled as MUST and SHOULD. MUST means this is a requirement and therefore mandatory to pass the test. SHOULD means it is recommended to pass the test if the driver implements the feature in question. + +| | Test case | | Priority | +|-----|-----------------------------------------|----------------------------|----------| +| 1 | WIFI_CONSTRUCTOR | | MUST | +| 2 | WIFI_CONNECT_NOCREDENTIALS | | MUST | +| 3 | WIFI_SET_CREDENTIAL | | MUST | +| 4 | WIFI_SET_CHANNEL | | SHOULD | +| 5 | WIFI_GET_RSSI | | SHOULD | +| 6 | WIFI_CONNECT_PARAMS_NULL | | MUST | +| 7 | WIFI_CONNECT_PARAMS_VALID_UNSECURE | | MUST | +| 8 | WIFI_CONNECT_PARAMS_VALID_SECURE | With security type: | | +| | | NSAPI_SECURITY_WEP | SHOULD | +| | | NSAPI_SECURITY_WPA | SHOULD | +| | | NSAPI_SECURITY_WPA2 | SHOULD | +| | | NSAPI_SECURITY_WPA_WPA2 | MUST | +| 9 | WIFI_CONNECT_PARAMS_CHANNEL | | SHOULD | +| 10 | WIFI_CONNECT_PARAMS_CHANNEL_FAIL | | SHOULD | +| 11 | WIFI_CONNECT | | MUST | +| 12 | WIFI_CONNECT_SECURE | With security type: | | +| | | NSAPI_SECURITY_WEP | SHOULD | +| | | NSAPI_SECURITY_WPA | SHOULD | +| | | NSAPI_SECURITY_WPA2 | SHOULD | +| | | NSAPI_SECURITY_WPA_WPA2 | MUST | +| 13 | WIFI_CONNECT_SECURE_FAIL | | MUST | +| 14 | WIFI_CONNECT_DISCONNECT_REPEAT | | MUST | +| 15 | WIFI_SCAN_NULL | | SHOULD | +| 16 | WIFI_SCAN | | SHOULD | + +Building test binaries +---------------------- + +For testing the board and driver, test against the Mbed OS master branch for the most recent, up-to-date test cases and drivers. + +To create a build environment: + +```.sh +mbed new wifi_test +cd wifi_test +cd mbed-os +git checkout master +cd .. +``` + +Now build test binaries: + +```.sh +mbed test --compile -t -m -n mbed-os-tests-network-wifi +``` + +Running tests +------------- + +The RAAS user and password have to be set in the shell to access it: + +```.sh +mbedgt -g :raas_client:ruka.mbedcloudtesting.com:8000 -n mbed-os-tests-network-wifi -V -v +``` + +Test cases for WifiInterface class +------------------------------------- + +### WIFI_CONSTRUCTOR + +**Description:** + + Test that the constructor of the driver works. + +**Preconditions:** + +None. + +**Test step:** + +Construct the driver by calling the constructor. + +**Expected result:** + +Constructor returns. + +### WIFI_CONNECT_NOCREDENTIALS + +**Description:** + +Test `WiFiInterface::connect()` without parameters. Don't set parameters with `set_credentials()`. + +This must be a first test to run after constructing the driver. No credentials should be given for the driver before this test. + +**Precondition:** + +Test the enviroment is set up as specified in the "Test Environment" chapter. + +**Test steps:** + +1. Initialize the driver. +2. Call `WiFiInterface::connect()`. +3. `disconnect()`. + +**Expected result:** + +`connect()` call returns `NSAPI_ERROR_PARAMETER or NSAPI_ERROR_NO_SSID`. + +### WIFI_SET_CREDENTIAL + +**Description:** + +This test case is to test whether the driver accepts valid credentials and rejects ones that are not valid. + +**Precondition:** + +Driver class is initialized. + +**Test step:** + +Call `set_credentials()` with various parameters described in "Expected results". + +**Expected result:** + +| Parameters | Password | Security | Expected return code | +|------------|---------------------------|----------------------------------|-----------------------------------------------------| +| ssid=NULL | NULL | NSAPI_SECURITY_NONE | NSAPI_ERROR_PARAMETER | +| ssid="OK" | NULL | NSAPI_SECURITY_WPA_WPA2 | NSAPI_ERROR_PARAMETER | +| ssid="OK" | NULL | NSAPI_SECURITY_WEP | NSAPI_ERROR_PARAMETER because password is missing. | +| ssid="OK" | NULL | NSAPI_SECURITY_NONE | NSAPI_ERROR_OK | +| ssid="OK" | [any 64 character string] | NSAPI_SECURITY_WPA2 | NSAPI_ERROR_PARAMETER because password is too long | +| ssid="OK" | [any 63 character string] | NSAPI_SECURITY_WPA2 | NSAPI_ERROR_OK | +| ssid="OK" | "" | NSAPI_SECURITY_WPA_WPA2 | NSAPI_ERROR_PARAMETER | +| ssid="OK" | "" | NSAPI_SECURITY_WEP | NSAPI_ERROR_PARAMETER because password is missing. | +| ssid="OK" | "" | NSAPI_SECURITY_NONE | NSAPI_ERROR_OK | +| ssid="OK" | "12345678" | NSAPI_SECURITY_WPA_WPA2 | NSAPI_ERROR_OK | +| ssid="OK" | "12345678" | NSAPI_SECURITY_WPA2 | NSAPI_ERROR_OK | +| ssid="OK" | "12345678" | NSAPI_SECURITY_WPA | NSAPI_ERROR_OK | +| ssid="OK" | "12345678" | NSAPI_SECURITY_WEP | NSAPI_ERROR_OK or NSAPI_ERROR_UNSUPPORTED | +| ssid="" | "" | NSAPI_SECURITY_NONE | NSAPI_ERROR_PARAMETER | + +### WIFI_SET_CHANNEL + +**Description:** + +Test validity of `WiFiInterface::set_channel()`. + +When proper channel number is feed, it should return `NSAPI_ERROR_OK`. API documentation is unclear what to report on error case, but we assume `NSAPI_ERROR_PARAMETER`. + +If the driver does not support setting channels for scan, it should report `NSAPI_ERROR_UNSUPPORTED` for all of the channels. + +**Precondition:** + +Driver class is initialized. + +**Test step:** + +Call `set_channel()` with various parameters described in "Expected results". + +**Expected result:** + +For 2.4 Ghz chips: + +| Channel | Expected return code | +|---------|-----------------------------------------------------------------------------------| +| 0 | NSAPI_ERROR_OK (0 = any channel) | +| 1 | NSAPI_ERROR_OK | +| 13 | NSAPI_ERROR_OK if supporting European frequencies | +| | NSAPI_ERROR_PARAMETER if configured for North America | +| 15 | NSAPI_ERROR_PARAMETER | +| | because not a valid 2.4 Ghz channel | + +For 5 Ghz chips: + +| Channel | Expected return code | +|---------|-----------------------------------------------------------------------------------| +| 3 | NSAPI_ERROR_PARAMETER | +| | because not a valid 5 Ghz channel number | +| 36 | NSAPI_ERROR_OK | +| 169 | NSAPI_ERROR_PARAMETERs | +| | should not be allowed in any country | + +Drivers not supporting + +| channel | Expected return code | +|---------|-----------------------------------------------------------------------------------| +| | NSAPI_ERROR_UNSUPPORTED should be returned for ALL channel numbers > 0. | + +### WIFI_GET_RSSI + +**Description:** + +Test `WiFiInterface::get_rssi()` API. When connected, it should return a valid RSSI value. When unconnected, it should return 0. + +API is a bit unclear whether 0 can also mean that driver does not support reporting of RSSI. + +**Precondition:** + +Test enviroment is set up as specified in "Test Environment" chapter. + +**Test steps:** + +1. Initialize the driver. +2. Call `get_rssi()`, store the result. +3. Call `connect()` with valid SSID parameters. Wait for connection. +4. Call `get_rssi()`. + +**Expected result:** + +The first call should return 0, and the second call should return a negative number between -10 and -100. + +### WIFI_CONNECT_PARAMS_NULL + +**Description:** + +Test `WiFiInterface::connect(ssid, pass, security, channel)` with NULL parameters. + +**Precondition:** + +Test enviroment is set up as specified in "Test Environment" chapter. + +**Test steps:** + +1. Initialize the driver. +2. Call `WiFiInterface::connect(NULL, NULL)`. +3. Call `WiFiInterface::connect("", "")`. + +**Expected result:** + +Both `connect()` calls return NSAPI_ERROR_PARAMETER. + +### WIFI_CONNECT_PARAMS_VALID_UNSECURE + +**Description:** + +Test `WiFiInterface::connect(ssid, pass, security)` with valid parameters for the unsecure network. + +**Precondition:** + +Test enviroment is set up as specified in the "Test Environment" chapter. + +**Test steps:** + +1. Initialize the driver. +2. Call `WiFiInterface::connect( , NULL)`. +3. `disconnect()`. +4. Call `WiFiInterface::connect( , "")`. +5. `disconnect()`. + +**Expected result:** + +`connect()` calls return NSAPI_ERROR_OK. + +### WIFI_CONNECT_PARAMS_VALID_SECURE + +**Description:** + +Test `WiFiInterface::connect(ssid, pass, security)` with valid parameters for secure network. + +**Preconditions:** + +Test enviroment is set up as specified in the "Test Environment" chapter. + +**Test steps:** + +1. Initialize the driver. +2. Call `WiFiInterface::connect( , , NSAPI_SECURITY_WPA2)`. +3. `disconnect()`. + +**Expected result:** + +`connect()` call returns NSAPI_ERROR_OK. + +### WIFI_CONNECT_PARAMS_CHANNEL + +**Description:** + +Test `WiFiInterface::connect(ssid, pass, security, channel)` with valid parameters for the secure network using the channel specified. + +This test only applies to devices that support selecting the channel (passes `WIFI-SET-CHANNEL`). + +**Precondition:** + +Test enviroment is set up as specified in the "Test Environment" chapter. + +**Test steps:** + +1. Initialize the driver. +2. Call `WiFiInterface::connect( , , NSAPI_SECURITY_WPA2, )`. +3. `disconnect()`. + +**Expected result:** + +`connect()` call returns NSAPI_ERROR_OK. + +### WIFI_CONNECT_PARAMS_CHANNEL_FAIL + +**Description:** + +Test `WiFiInterface::connect(ssid, pass, security, channel)` with valid parameters for the secure network using the channel specified. The channel specified must be wrong, so the connect call can fail. + +This test only applies to devices that support selecting the channel (passes `WIFI-SET-CHANNEL`). + +**Precondition:** + +Test enviroment is set up as specified in the "Test Environment" chapter. + +**Test steps:** + +1. Initialize the driver. +2. Call `WiFiInterface::connect( , , NSAPI_SECURITY_WPA2, )`. +3. `disconnect()`. + +**Expected result:** + +The `connect()` call returns `NSAPI_ERROR_CONNECTION_TIMEOUT` or `NSAPI_ERROR_NO_CONNECTION`. + +### WIFI_CONNECT + +**Description:** + +Test `WiFiInterface::connect()` without parameters. Use `set_credentials()` for setting parameters. This checks that driver does not try to retrieve SSID from a location it's not controlling. + +**Preconditions:** + +1. Test enviroment is set up as specified in the "Test Environment" chapter. + +**Test steps:** + +1. Initialize the driver. +2. `Call WiFiInterface::set_credentials( , NULL)`. +3. `Call WiFiInterface::connect()`. +4. `disconnect()`. +5. `Call WiFiInterface::set_credentials( , "")`. +6. `Call WiFiInterface::connect()`. +7. `disconnect()`. +8. Trash the memory storing SSID. +9. `Call WiFiInterface::set_credentials( , "")`. +10. `Call WiFiInterface::connect()`. +11. `disconnect()`. + +**Expected result:** + +`connect()` calls return `NSAPI_ERROR_OK`. + +### WIFI_CONNECT_SECURE + +**Description:** + +Test `WiFiInterface::connect()` without parameters. Use secure settings for `set_credentials`. + +**Precondition:** + +The test enviroment is set up as specified in the "Test Environment" chapter. + +**Test steps:** + +1. Initialize the driver. +2. Call `WiFiInterface::set_credentials( , , NSAPI_SECURITY_WPA2)`. +3. Call `WiFiInterface::connect()`. +4. `disconnect()`. + +**Expected result:** + +The `connect()` call returns `NSAPI_ERROR_OK`. + +### WIFI_CONNECT_SECURE_FAIL + +**Description:** + + Test `WiFiInterface::connect()` failing with the wrong password. + +**Precondition:** + +The test enviroment is set up as specified in the "Test Environment" chapter. + +**Test steps:** + +1. Initialize the driver. +2. Call `WiFiInterface::set_credentials( , , NSAPI_SECURITY_WPA2)`. +3. Call `WiFiInterface::connect()`. +4. `disconnect()`. + +**Expected result:** + +`connect()` call returns `NSAPI_ERROR_AUTH_FAILUR`, `NSAPI_ERROR_CONNECTION_TIMEOUT` or `NSAPI_ERROR_NO_CONNECTION`. + +### WIFI_CONNECT_DISCONNECT_REPEAT + +**Description:** + +Test `WiFiInterface::connect()` - `disconnect()` repetition works. + +**Precondition:** + +The test enviroment is set up as specified in the "Test Environment" chapter. + +**Test steps:** + +1. Initialize the driver. +2. Call `WiFiInterface::set_credentials( , NULL)`. +3. Repeat 10 times: + 1. Call `WiFiInterface::connect()`. + 2. `disconnect()`. + +**Expected result:** + +Each `connect()` call returns `NSAPI_ERROR_OK`. + +Each `disconnect()` returns `NSAPI_ERROR_OK`. + +### WIFI_SCAN_NULL + +**Description:** + +Call `WiFiInterface::scan()` with null parameters to get the number of networks available. + +**Precondition:** + +The test enviroment is set up as specified in the "Test Environment" chapter. + +**Test steps:** + +1. Initialize the driver. +2. Call `WiFiInterface::scan(NULL, 0)`;. + +**Expected result:** + +`scan()` returns a positive number that is higher or equal to two. (>=2). + +### WIFI_SCAN + +**Description:** + + Call `WiFiInterface::scan()` with a valid accesspoint list allocated. + +**Preconditions:** + +1. The test enviroment is set up as specified in the "Test Environment" chapter. +2. The test environment must contain less than 10 Wi-Fi SSID within the listening range of DUT; otherwise, adjust the value used in step 2. + +**Test steps:** + +1. Initialize the driver. +2. Allocate array of 10 WiFiAccessPoint objects. +3. Call `WiFiInterface::scan(, 10)`; + +**Expected result:** + +`scan()` returns a positive number that is higher or equal to two. (>=2). + +Scan elements from array indices between 0 and the number returned: + +- Call `get_ssid()` for each element. `` and `` must be found. +- Call `get_rssid()` for each element. The value must be between -10 and -100. + +For those known networks, test that the following return values are found: + +| get_ssid() | get_bssid() | get_security() | get_channel() | +|-------------------------|------------------------|-----------------------|-----------------------| +| `` | `` | NSAPI_SECURITY_NONE | `` | +| `` | `` | NSAPI_SECURITY_WPA2 | `` |