Add CLI input mode to test runner #2955

aggarw13 · 2021-01-26T23:49:18Z

This PR adds a another solution ( separate from the configurable delay config added in PR #2950 ) for the problem of race condition between AWS IoT Device Tester (that runs on host machine) getting serial console access to device and the test runner executing tests on the device.

This PR adds support to the test runner to wait for accepting a "start" command from the serial console (based on FreeRTOS+CLI library) before beginning execution of tests. This can allow the AWS IoT Device Tester to ensure that the device is serially accessible before tests begin executing on the device.
This feature can be enabled with the AWS_TEST_RUNNER_ENABLE_CLI_INPUT config in the aws_test_runner_config.h file.

Here is the log that shows interacting with the test runner when the feature is enable. Note that after the start command is provided on the serial prompt, the tests starts executing.

Welcome to the Test Runner console.
[Type the "start" command to beging executing tests]
>help

help:
 Lists all the registered commands

start : Instructs test runner to start executing tests on the board.
[Press Enter to run the previous command]
>start
---------STARTING TESTS---------
TEST(Full_PKCS11_StartFinish, AFQP_StartFinish_FirstTest) PASS
TEST(Full_PKCS11_StartFinish, AFQP_GetFunctionList) PASS

Note: This feature uses FreeRTOS+CLI library along with Common I/O interface for UART communication. Thus, this feature is only supported by boards that provide ports for Common I/O interface.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…input-for-test-runner

ravibhagavandas · 2021-01-27T02:00:42Z

libraries/freertos_plus/standard/freertos_plus_cli/source/FreeRTOS_CLI_Console.c

-                                size_t commandBufferLength,
-                                char * pOutputBuffer,
-                                size_t outpuBufferLength )
+void FreeRTOS_CLI_ProcessInputBuffer( xConsoleIO_t consoleIO,


This duplicates some code in FreeRTOS_CLI_ProcessInputBuffer to handle the exiting the CLI loop. Can we keep code same and instead add a new command such as exit for the loop to exit gracefully.

We discussed offline and decided that we can add a exit command support to the FreeRTOS_CLIEnterConsoleLoop function to allow the console loop to be terminated in a separate PR.

ravibhagavandas · 2021-01-27T02:15:41Z

tests/common/aws_test_runner.c

-    UNITY_BEGIN();
+        UNITY_BEGIN();
+
+        RunTests();


It seems the memory leak test wont start before the test begin
testrunnerFULL_MEMORYLEAK_ENABLED

Fixed by moving all the test setup, execution of tests and teardown logic in common function.

ravibhagavandas · 2021-01-27T02:17:33Z

tests/common/aws_test_runner.c

+
+        UNITY_BEGIN();
+
+        RunTests();


Needs to handle UNITY_END() ?

Have moved all the test logic (of setup, executing tests and tear down) to common function, which is called from the handler now.

ravibhagavandas · 2021-01-27T02:20:08Z

tests/common/aws_test_runner.c

+
+        /* Set flag to indicate reception of command from serial console to start */
+        /* executing tests. */
+        xReceivedCommand = pdTRUE;


An "OK" response back to the peer here will help in debugging a broken CLI connection to the actual test issues.

Added the response to acknowledge processing of command.

Per offline discussion, removed the response for now as the response would be visible only at the end of the test execution. We can consider adding the response string later if there are architectural changes of running the CLI console in a different task than the test runner.

… end of tests

This reverts commit 441e02e.

* Add cypress internal tests and 062 boards from cypress-202007.00 * Add 062 projects * Sync aws demo/test from 064 board to 062 boards. 064 got its changes from Amazon in #2544. * Sync aws demo/test from 064 board to 062 boards. 064 got its changes from Amazon in #2544. * Update 062 boards for PKCS11 changes from Amazon master. Update demo defaults for 062 boards from Amazon master. Regenerate GeneratedSource. * Regenerate GeneratedSource, remove iot_pkcs11_pal.c to align with Amazon PKCS11 implementation from master * Update Cy8CKIT-062-WIFI-BT with regenerated GeneratedSource and removed iot_pkcs11_pal.c to align with Amazon master updated to PKCS11. * Align aws_test config files to new Amazon master templates. Regenerate 062-WIFI-BT GeneratedSource with tools_2.2 * Regeneate 062S2-43012 aws_tests GeneratedSource with tools_2.2 * Regenerate 062S2S0-4343W aws_test GeneratedSource with tools_2.2 * Regenerate CY8CPROTO-062-4343W aws_test GeneratedSource with tools_2.2 * Updated IDT to 4.0.1 * Copy .gitlab-ci.yml from cypress-202007.00 branch * Align iot_config.h to set BLE/WIFI_SUPPORTED as on master. Add definitions for configCOMMAND_INT_MAX_OUTPUT_SIZE and configINCLUDE_DEMO_DEBUG_STATS to FreeRTOSConfig.h as on master * Add script for cleaning up old IDT OTA buckets * Align use of BLE_SUPPORT to BLE_SUPPORTED for consistency across CMake vs Make * Align test scripts to use BLE_SUPPORTED * Add Celluar definition to device.json for IDT tests as required * Move definition of BLE_SUPPORTED DEFINE from project Makefiles to the common mtb_global_settings.mk. Aligned aws_tests main.c to use #if BLE_SUPPORTED as seen in the aws_demos main.c * Add keys from cypress-2020007.00 branch back, as they were removed by Amazon on master. * Update userdata.json for IDT to 4.0 standard * Minor fixes to aws-tests to compile successfully * Update yml file to version IDT version 4.0.1 * Allow IAR build/compile/funcation CI to fail due to SWINTEGRATION-341 until a fix/workaround is available. * Remove tinycrypt from CMake configuration and re-enable IAR-based tests. Fixes SWINTEGRATION-341. * CY8CKIT-064S0S2-4343W: Fix aws_demo aws_demo is no longer connecting to AP. It looks like a merge error resulted in the removal of the needed function call. Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com> * Sync 062 main.c from 064 main.c * Update policy for swap+copy Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com> * Sync all boards to 064 Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com> * Revert "Add CLI input mode to test runner (#2955)" This reverts commit 441e02e. * Update cysecuretools to use 3.1.0 Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com> * Update TFM policies Policies are out of date and also needs adjustments for swap support. Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com> * Fix OTA_SUPPORT build for aws_tests Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com> * SWAP_WITH_STATUS bootloader support SWINTEGRATION-270 * Sync boards to 064 * fix to remove configASSERT() in Release build * Add PSOC6 Component to fix WDT enable problem * Add watchdog timer to 064S0 Create a thread to start and kick WDT before prvWifiConnect() Return WiFI status from prvWifiConnect() Stop after prvWifiConnect() returns. If we fail to connect in prvWifiConnect(), reboot * Update SoftAP creation sequence for aws demos * MIDDLEWARE-3942 Update AFR OTA docs * Update OTA block size and OTA thread priority - 4k block size speeds up HTTP - OTA agent can run at lowest priority Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com> * Sync OTA changes from 064 to other boards Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com> * Revert "fix to remove configASSERT() in Release build" This reverts commit 5c7c72a. This changed caused FullWiFi and FullSecureSockets IDT tests to fail. Signed-off-by: Raymond Ngun <raymond.ngun@infineon.com> * fix for SWINTEGRATION-386, use -j4 in make * Sync 064 main.c to 062 boards. * Correct version control conflict marker * Add subscrition container to afr source. * Add missing MQTT source files to Make build system * More missing MQTT files from Make build system * Remove keys for upstreaming to AWS github. * Remove 062 boards for upstream to AWS GitHub * Remove internal Cypress testing files. * Checkout the files from main branch which should not be changed by this PR. Co-authored-by: Fyall Ian (CYSC CSS ICW SW T 1) <Ian.Fyall@infineon.com> Co-authored-by: midd <midd@cypress.com> Co-authored-by: Raymond Ngun <raymond.ngun@cypress.com> Co-authored-by: RaymondNgun <raymond.ngun@infineon.com> Co-authored-by: stde <SteveO.DeFrisco> Co-authored-by: Chandran Praveen Babu (CYSC CSS ICW SW PSW) <PraveenBabu.Chandran@infineon.com> Co-authored-by: Bill Waters <bill.waters@infineon.com> Co-authored-by: Ming Yue <mingyue86010@gmail.com>

aggarw13 added 8 commits January 22, 2021 16:38

Changes for testing FreeRTOS+CLI based prompt support in test runner

22c1771

Fix bugs in supporting FreeRTOS+CLI functionality

83be257

Push CMake changes for test target

4b74a3b

Improve code style and add config to enable serial console mode

0fabb5f

Merge remote-tracking branch 'origin/master' into poc/support-prompt-…

39e3037

…input-for-test-runner

Formatting and documentation in FreeRTOS_CLI_Console

c58fe45

Add config for serial console to espressif test runner

af8b5bb

Rename config and update documentation for config

323bd90

aggarw13 requested a review from a team as a code owner January 26, 2021 23:49

aggarw13 added 2 commits January 27, 2021 00:30

Add missing RunTests in non serial console mode

87c592f

Attempt to fix Cmake build for boards that do not support common IO

954584b

ravibhagavandas reviewed Jan 27, 2021

View reviewed changes

aggarw13 added 5 commits January 27, 2021 04:03

Add response for start command

d9ad3cb

style: move all test setup and teardown code in common function

1960949

Remove serial input mode config accidentally enabled in aws_test_runner

c8fa02c

Make dependency on AFR::freertos_cli_plus_uart conditional

c42eeaf

Remove output response for "start" command as it is not visible until…

c587153

… end of tests

ravibhagavandas approved these changes Jan 27, 2021

View reviewed changes

cobusve approved these changes Jan 27, 2021

View reviewed changes

aggarw13 merged commit 441e02e into aws:master Jan 27, 2021

aggarw13 deleted the poc/support-prompt-input-for-test-runner branch January 27, 2021 23:45

aggarw13 mentioned this pull request Feb 4, 2021

Add updates about test runner and ESP-IDF in CHANGELOG #2970

Merged

ifyall pushed a commit to ifyall/amazon-freertos that referenced this pull request May 19, 2021

Revert "Add CLI input mode to test runner (aws#2955)"

5c44ccc

This reverts commit 441e02e.

mingyue86010 pushed a commit that referenced this pull request May 26, 2021

Revert "Add CLI input mode to test runner (#2955)"

9caba63

This reverts commit 441e02e.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CLI input mode to test runner #2955

Add CLI input mode to test runner #2955

aggarw13 commented Jan 26, 2021 •

edited

ravibhagavandas Jan 27, 2021

aggarw13 Jan 27, 2021

ravibhagavandas Jan 27, 2021

aggarw13 Jan 27, 2021

ravibhagavandas Jan 27, 2021

aggarw13 Jan 27, 2021

ravibhagavandas Jan 27, 2021

aggarw13 Jan 27, 2021

aggarw13 Jan 27, 2021

Add CLI input mode to test runner #2955

Add CLI input mode to test runner #2955

Conversation

aggarw13 commented Jan 26, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aggarw13 commented Jan 26, 2021 •

edited