Amazon Frustration Free Setup Certification Tool

Python-based automation test scripts which can help Non-Amazon device partners to launch their devices with FFS with shorten self-certification time. The document will describe how to run those scripts to complete the certification tests, especially the provisioning performance test.

Frustration Free Setup Overview

See Understanding Frustration-Free Setup for more information.

Frustration Free Setup Certification Process

See Provisionee Certification Guide for more information.

Requirements and Getting started

• 1 Android phone in USB debugging mode
  • Alexa App installed and login with your Amazon account
  • The phone is connected with your test machine (Windows, MacOS or Linux) with USB connection.
• 1 Provisioner/commissioner device, which is registered to your Amazon account, see recommended devices from Understanding Frustration-Free Setup - Testing Your Device
• 1 Provisionee/commissionee device, your device under test, called DUT in this document
• 2 Control devices, Amazon Smart Plugs are recommended
  • Please register both to your Amazon account.
  • Please plug DUT on top of the first smart plug, the plug is OFF and DUT is in factory reset mode.
  • Please plug the provisioner/commissioner device on top of the second smart plug, the plug is ON.
• Please register your DUT to your Amazon account, refer to Submit Test Devices Guideline

Test Machine Environment setup

• Please download and install **Python3 (>=3.7), Android SDK, and Appium (2.3.0) with Appium driver uiautomator2 **
• Please configure $PATH and $ANDROID_HOME to enable running python, adb and appium commands under any folder
• Please run the commands below to verify your environment (the workable environment are listed here for your reference)

$ python --version
Python 3.9.6

$ adb --version
Android Debug Bridge version 1.0.41
Version 34.0.5-10900879
Installed as ***/Android/sdk/platform-tools/adb
Running on Darwin 22.6.0 (x86_64)

$ appium --version
2.3.0

$ appium driver list --installed
✔ Listing installed drivers
- uiautomator2@2.35.0 [installed (npm)]

Run Test Scripts

Open a terminal on Mac/Linux or PowerShell (Admin) on Windows

Step 1: Setup Python virtual environment

$ python -m venv <your_venv_name>

Step 2: Activate virtual environment

[MacOS/Linux]

$ source venv/bin/activate

[Windows]

> .\venv\Scripts\Activate.ps1

Step 3: Install python dependencies in the virtual environment

(<your_venv_name>) pip install Appium-Python-Client pytest-repeat
(<your_venv_name>) pip install -e .

Step 4: Execute the test

(<your_venv_name>) pytest [Options]

Options:

--ffs_type

• FFS protocol type, one of ["BSS", "ZSS", "MSS"], "MSS" as default

--name_of_plug_to_control_dut

• Name of the smart plug (displayed on Alexa App) to power on and off DUT, "First plug" as default
  

--name_of_plug_to_control_provisioner

• Name of the smart plug (displayed on Alexa App) to power on and off provisioner/commissioner, "Second plug" as default
  

--name_of_dut

• Name of the device under test displayed on Alexa App after the setup, required option

--appium_server_port

• Port used to start appium server on localhost, 4723 as default

--count

• Num of iterations

-x

• Use it to stop the execution if any iteration failed (it's highly recommended)

Examples:

Run 10 iterations against Matter device named "First switch" and stop the execution if any iteration failed

(<your_venv_name>) pytest --name_of_dut="First switch" --count=10 -x

Run 5 iterations against ZSS device named "First light" and continue the execution if some iterations failed

(<your_venv_name>) pytest --ffs_type=ZSS --name_of_dut="First light" --count=5

Test Report and Logging

• pytest.ini file includes pytest configuration:
  • log-cli settings enable the live log and test summary from console output
  • log-file settings save the logging to pytest_log.txt in logs folder
• appium_server_log.txt from logs folder includes all appium_server logs during the test

Notes

• So far the tool only supports multiple rounds of BSS,ZSS and MSS test as removing both types of devices from Alexa App will factory reset them
• It could support WSS over Wifi if removing a WSS over Wifi device from the third party app can factory reset it and you need to override methods in wss_device.py (template/psudocodes provided)

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.