Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remote Playback API test automation #92

Open
Honry opened this issue Jun 21, 2017 · 14 comments

Comments

@Honry
Copy link

commented Jun 21, 2017

Import comment(#87 (comment)) from @mounirlamouri:

The issue with this API is that you will have to use a Chrome Cast compatible device to test the API on Chrome but another browser might not support Chrome Cast. The test preconditions will have to start with something a bit hand-wavy like "Have a device compatible with the browser on your WiFi". Because only Chrome (maybe other Chrome-based browsers) has an implementation for this at the moment, it would be hard to be clearer.

@Honry

This comment has been minimized.

Copy link
Author

commented Jun 21, 2017

@louaybassbouss, I know you did really good on test automation for Presentation API, the problem we encountered in Remote Playback should be similar, could you please share your experience for us?

@Honry

This comment has been minimized.

Copy link
Author

commented Oct 13, 2017

@louaybassbouss, friendly ping.

@louaybassbouss

This comment has been minimized.

Copy link

commented Oct 13, 2017

@Honry sorry I missed your request from 21 Jun thanks for the reminder. Yes the setup for testing the Presentation API and RemotePlayback API Tests on Chromecast should be similar. I would say that the tests of RemotePlayback API are easier to implement since you will not deal with different Presentation URLs as in the Presentation API (You can have a look in [1] where two different Presentation URLs are specified, one for cast receivers and one for other receivers that can render a web page). In RemotePlayback API you don't need to pass any URL which makes a lot of things easier. To test on Chromecast (or any other cast receiver like Android TV) you need just to make sure that your PC and the Receiver devices are in the same Network. But you need to keep in mind when you are writing the tests to not use any code that is specific for chromecast or any specific technology, but only according the API specification. To create a test report of RemotePlayback API e.g. for Chrome Browser, the tester (for example the person who is responsible to create the report) must have the information about which receiver devices are supported by the Browser under test. The tester musst also ensure that a receiver device is available during the test. I would recommend to habe a look to the tests of Presentation API in [2] and the current Test Report in [3].

PS: when you are writing tests please keep in mind that you are testing the API Implementation and NOT the underlying protocols.

@Honry

This comment has been minimized.

Copy link
Author

commented Oct 13, 2017

@louaybassbouss, really appreciate for your exhaustive sharing, I agree with you that tests should be strictly compliance with API specification without any dependence. I will dig into tests of Presentation API as a reference to write tests for Remote Playback API.

@anssiko

This comment has been minimized.

Copy link
Member

commented Oct 24, 2017

@Honry, we're likely to discuss this topic at our TPAC F2F on Monday 6 November, so any input, testing-related issues, even if preliminary, prior to that would be appreciated.

@Honry

This comment has been minimized.

Copy link
Author

commented Oct 25, 2017

Can we talk about the possibility to implement Testing API(quite similar to w3c/presentation-api#440) to break through test automation?

@Honry

This comment has been minimized.

Copy link
Author

commented Oct 25, 2017

Considering the Remote Playback API is much more simple then Presentation API, that is the amount of manual tests is much less. But implement a Testing API would bring much more efforts.

@louaybassbouss

This comment has been minimized.

Copy link

commented Oct 25, 2017

@Honry please let me know if can help on this

@mfoltzgoogle

This comment has been minimized.

Copy link
Contributor

commented Oct 28, 2017

If the test API is strictly about testing the RemotePlayback interface, it could be very simple:

<video>.remote.test.deviceAvailable = true to simulate a remote playback device being available for that element, and false to simulate a device not being available.

If we want to test behavior of the media element itself in remote playback scenarios, that could mean creating a fake remote playback device with specific capabilities.

@Honry

This comment has been minimized.

Copy link
Author

commented Nov 2, 2017

.remote.test.deviceAvailable = true to simulate a remote playback device being available for that element, and false to simulate a device not being available.

I think this should be enough. @louaybassbouss, what's your opinion?

@louaybassbouss

This comment has been minimized.

Copy link

commented Nov 2, 2017

@Honry <video>.remote.test.deviceAvailable = true is enough for Observing remote playback device availability but I am not sure how to test the RemotePlayback Connection State. We may need a 'FakePlaybackScreen' similar to this proposal from @mfoltzgoogle. The 'FakePlaybackScreen' can have some additional attributes like playing, paused, etc. to test the transition between the different remote playback connection states. What do you think?

@Honry

This comment has been minimized.

Copy link
Author

commented Nov 2, 2017

@louaybassbouss, I was thinking using <video>.remote.test.deviceAvailable = true/false to monitor state change should be enough. But yes, I agree with you that we should also test the video state must be align with the remote playback.

@mfoltzgoogle mfoltzgoogle added the F2F label Nov 2, 2017
@mfoltzgoogle

This comment has been minimized.

Copy link
Contributor

commented Nov 2, 2017

We do need to be able to test a failure to connect to a remote playback device (step 4 of "Establishing a Connection.")

As far as testing the behavior of the media element, the mandatory requirements in the spec are:

  • The user agent MUST send media commands to the remote playback device - but how would this be observable from a test?
  • The media element MUST reflect changes to the remote playback state, which is similar to testing media playback in general.

Let's discuss this further at the F2F.

@mounirlamouri

This comment has been minimized.

Copy link
Member

commented Nov 6, 2017

Blink is using a method to force update the device availability. Because we run this only in our tests which do not have a Remote Playback backend, I think the WPT method should be different as there will be an idea of forcing the value on top of the backend. Because of this, I would recommend the method to take true, false and null in order to unset the override. A method might make things clearer to (such as setOverrideDeviceAvailability()). Same could apply for connection state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.