Verification Tests

Test Structure

A Test is composed of a few different attributes:

• The functions necessary to facilitate the test, such as runTest() or endTest()
• Variables needed throughout the test, such as a reference to the media app’s MediaController
• A list of TestStep objects to run (hereafter referred to as steps).

The Test runs the current step when the MediaController observes a change in either the PlaybackState or Metadata. A TestStep can return one of three possible values:

• STEP_FAIL immediately ends the test because something went wrong.
• STEP_CONTINUE means everything is okay but the conditions haven’t been met to move to the next step.
• STEP_PASS indicates that the test can proceed to the next step.

The Test succeeds when its last step returns STEP_PASS.

Adding a test

This testing framework is designed to make adding new tests straightforward. This page will guide you through adding the “Play Test” as an example.

The first task is to define the steps that the Test will run. To define a new step, create a class that implements the TestStep interface located in the MediaAppTestDetails file. There are two items each TestStep must implement:

• The logTag String, an identifier that will prepend log messages from this TestStep
• The execute() function, which is called to run a TestStep

The first step you need is ConfigurePlay. “Configure” steps have the following responsibilities:

1. Store the PlaybackState and Metadata to keep a reference to their original values
2. If applicable, parse the query text into a value usable by the TransportControl request
3. Check if the requested action is marked as supported in the PlaybackState
4. Send the TransportControl request

Below is the completed ConfigurePlay step:

/**
 * No query input. This step checks if ACTION_PLAY is supported and sends the play() request.
 * Always returns STEP_PASS.
 */
class ConfigurePlay(override val test: Test) : TestStep {
    override val logTag = "${test.name}.CP"
    override fun execute(
            currState: PlaybackStateCompat?,
            currMetadata: MediaMetadataCompat?
    ): TestStepStatus {
        test.origState = test.mediaController.playbackState
        test.origMetadata = test.mediaController.metadata
 
        checkActionSupported(currState, PlaybackStateCompat.ACTION_PLAY)
        test.testLogger(logTag, androidResources.getString(
                R.string.test_running_request,
                "play()"
        ))
        test.mediaController.transportControls.play()
        return TestStepStatus.STEP_PASS
    }
}

The second step you need is WaitForPlaying. This step is in charge of monitoring the MediaController’s Metadata and PlaybackState to verify that the play() request was properly executed. For this test, the media item is not expected to change, so Metadata should remain the same as it was originally in the ConfigurePlay step. The PlaybackState State should be STATE_PLAYING at the end of this test. Any other Terminal State (such as STATE_PAUSED or STATE_STOPPED) will result in the test failing. The step will return STEP_CONTINUE for any Transition State (such as STATE_BUFFERING or STATE_CONNECTING).

Below is the completed WaitForPlaying step:

/**
 * PASS: metadata must not change, and state must be STATE_PLAYING
 * CONTINUE: null state, original state, transition states
 * FAIL: metadata changes, any other terminal state
 */
class WaitForPlaying(override val test: Test) : TestStep {
    override val logTag = "${test.name}.WFP"
    override fun execute(
            currState: PlaybackStateCompat?,
            currMetadata: MediaMetadataCompat?
    ): TestStepStatus {
        test.testLogger(logTag, androidResources.getString(
                R.string.test_compare_metadata,
                test.origMetadata.toBasicString(),
                currMetadata.toBasicString()
        ))
        // Metadata should not change for this step, but some apps "update" the Metadata with the
        // same media item.
        if (test.origMetadata != null && !test.origMetadata.isContentSameAs(currMetadata)) {
            test.testLogger(logTag, androidResources.getString(R.string.test_error_metadata))
            return TestStepStatus.STEP_FAIL
        }
 
        return when {
            currState?.state == null -> {
                test.testLogger(logTag, androidResources.getString(R.string.test_warn_state_null))
                TestStepStatus.STEP_CONTINUE
            }
            currState.state == PlaybackStateCompat.STATE_PLAYING -> {
                TestStepStatus.STEP_PASS
            }
            currState.state == test.origState?.state
                    || transitionStates.contains(currState.state) -> {
                // Sometimes apps "update" the Playback State without any changes or may enter an
                // unexpected transition state
                TestStepStatus.STEP_CONTINUE
            }
            else -> {
                // All terminal states other than STATE_PLAYING
                TestStepStatus.STEP_FAIL
            }
        }
    }
}

With the steps done, next you need to define the actual Test in the MediaAppTests file. This file holds the functions that run when the Run Test button is clicked in the MCT UI. The function is where you’ll instantiate the Test, add the two steps you defined, and run the test.

Below is the completed runPlayTest function:

fun runPlayTest(
        controller: MediaControllerCompat,
        logger: (tag: String, message: String) -> Unit?
) = Test(Test.androidResources.getString(R.string.play_test_logs_title), controller, logger)
        .apply {
            addStep(ConfigurePlay(this))
            addStep(WaitForPlaying(this))
        }.runTest()

With this, all of the components for a test are defined. Note that with this system, steps can be reused across tests. For example, to define a more complex test that emulates a user requesting a play action, followed by a seek-to action, followed by a skip-to-next action, the only new thing required is the below runChainTest function:

fun runChainTest(
        query: String
        controller: MediaControllerCompat,
        logger: (tag: String, message: String) -> Unit?
) = Test(Test.androidResources.getString(R.string.chain_test_logs_title), controller, logger)
        .apply {
            addStep(ConfigurePlay(this))
            addStep(WaitForPlaying(this))
            addStep(ConfigureSeekTo(this, query))
            addStep(WaitForTerminalAtTarget(this))
            addStep(ConfigureSkipToNext(this))
            addStep(WaitForSkip(this))
        }.runTest()

Mobile App

To use the test in the Android mobile app, you need to add the test to the MediaAppTestingActivity file in the setupTests() function. The TestOptionDetails class is used to display Tests in the MCT UI. It takes a display name and description for the test, as well as a function to run when the Run Test button is clicked (in this case, the runPlayTest function defined above). Remember to add this playTest object to the testOptionAdapter list to add a card for the Play Test in the MCT UI.

Below is the completed playTest object:

/**
 * Tests the play() transport control. The test can start in any state, might enter a
 * transition state, but must eventually end in STATE_PLAYING. The test will fail for
 * any terminal state other than the starting state and STATE_PLAYING. The test
 * will also fail if the metadata changes unless the test began with null metadata.
 */
val playTest = TestOptionDetails(
        getString(R.string.play_test_title),
        getString(R.string.play_test_desc)
) { _ ->
    Test(
            getString(R.string.play_test_logs_title),
            controller,
            ::logTestUpdate
    ).apply {
        addStep(ConfigurePlay(this))
        addStep(WaitForPlaying(this))
    }.runTest()
}

Android TV

Adding a test to the Android TV app takes a couple extra steps. First, add an action to the TvTestingGuidedStepFragment in the onCreateActions function. An action requires an ID (defined in the companion object), a test title, a test description, and an optional Boolean that should be set to true for tests that use a query.

Below is the code to add an action for the Play Test:

override fun onCreateActions(actions: MutableList<GuidedAction>, savedInstanceState: Bundle?) {
    actions.add(buildAction(
            PLAY_TEST,
            getString(R.string.play_test_title),
            getString(R.string.play_test_desc)
    ))
    ...
}

Next, add a case for the new test in the onGuidedActionClicked function. This case should simply run the test function you defined earlier (runPlayTest in this case).

Below is the case used to execute the Play Test:

override fun onGuidedActionClicked(action: GuidedAction?) {
    ...
    mediaController?.let {
        when (action?.id) {
            PLAY_TEST -> runPlayTest(it, ::logTestUpdate)
            ...
        }
    }
}

Note that for tests that use a query, instead of adding a case to the onGuidedActionClicked function, complete the following:

• Add the ID of the new test to the queryTests list in onGuidedActionClicked
• Add a case for the new test to onGuidedActionEditedAndProceed. This case should reset the title, update the description with the query, and run the requested test.
• Add a case for the new test to onGuidedActionEditCanceled. This case should reset the title and description of the action.