Initial pass of WebVR 1.1 conformance tests #5535
Conversation
An initial drop of WebVR V1.1 conformance tests. These tests require an additional file - VRSimulator.js - that includes browser specific implementations of simulated VR actions. (Examples include: AttachWebVRDisplay and DetachWebVRDisplay. See the Readme.md for the full list.) A larger suite of tests are coming soon.
|
Notifying @klausw. (Learn how reviewing works.) |
Firefox (nightly channel)Testing web-platform-tests at revision badb452 All results3 tests ran/webvr/frozenArray_activeVRDisplays.html
/webvr/getLayers.html
/webvr/requestPresent.html
|
Chrome (unstable channel)Testing web-platform-tests at revision badb452 All results3 tests ran/webvr/frozenArray_activeVRDisplays.html
/webvr/getLayers.html
/webvr/requestPresent.html
|
|
Reviewed 6 of 6 files at r1. webvr/frozenArray_activeVRDisplays.html, line 32 at r1 (raw file):
This test looks good to me. I am just curious why you wanna verify the array is not frozen in WebVR test. That seems to be not related to the field of WebVR. Comments from Reviewable |
|
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed. webvr/frozenArray_activeVRDisplays.html, line 32 at r1 (raw file): Previously, daoshengmu (Daosheng Mu) wrote…
The reason for the test is to verify the array returned from the API is truly Frozen and cannot have any elements add or removed. Comments from Reviewable |
|
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed. webvr/frozenArray_activeVRDisplays.html, line 32 at r1 (raw file): Previously, RafaelCintron (Rafael Cintron) wrote…
ok. That makes sense. Comments from Reviewable |
|
Can you please fix the lint failure that's leading to Travis to fail? i.e., |
|
These tests are now available on w3c-test.org |
LintPassed |
|
@SamiraAtMicrosoft, did you check the "Allow edits from maintainers" checkbox when opening this PR? See https://github.com/blog/2247-improving-collaboration-with-forks. I tried to fix the Readme.md problem, but couldn't push the fix. The problem is that it's a UTF-16 file, can be fixed with |
|
@SamiraAtMicrosoft, this approach is similar to one that has been discussed on public-test-infra and in Chromium recently: And for Bluetooth specifically: I think what's needed here is a discussion between the implementers of WebVR to agree on a framework for tests. For Chromium, that's @toji and @kenrussell. |
|
FWIW, I think that if the testing APIs were written into a spec and well defined, then an approach like this should be workable. Chromium's layout tests in https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/vr/ seem to use a similar approach. |
|
@foolip , we have been discussing WebVR tests in the WebVR group as part of the Interoperable WebVR Testing #187. At Mozilla, @daoshengmu has already integrated a subset of the tests into Mozilla's testing infrastructure. For Chromium, @ddorwin expressed interest in developing a web server that communicates with a OpenVR test driver. At Microsoft, we've been running these tests internally using WebDriver and Perception Simulation I think the current structure of the tests provides flexibility in how browsers structure the backend simulation of device. But, we're happy to hear feedback from others. |
Codecov Report
@@ Coverage Diff @@
## master #5535 +/- ##
=======================================
Coverage 87.22% 87.22%
=======================================
Files 24 24
Lines 2418 2418
Branches 406 406
=======================================
Hits 2109 2109
- Misses 239 240 +1
+ Partials 70 69 -1
Continue to review full report at Codecov.
|
|
@SamiraAtMicrosoft @RafaelCintron, I've looked over immersive-web/webxr#187 and find that very encouraging. Do you think you could send a message to public-test-infra about the proposed setup, maybe just linking to this PR and the previous discussion? If something in the shape of a spec is written that defines these APIs, and web-platform-tests just assumes those APIs exist, that would be a good start I think. There should also be some way of enabling such test APIs that's done in wptrunner, so that the tests can run using the actual releases of various browsers. @toji, what do you think about all of this? |
|
(It looks like @toji will be OOO for about a week.) |
|
@foolip , the readme file in the pull request has description of the APIs the test runner is meant to implement. We'll send an e-mail to public-test-infra. |
|
I see https://lists.w3.org/Archives/Public/public-test-infra/2017AprJun/0033.html was sent a week ago now, but it didn't show up in my inbox so I can't respond there. I'm curious about what the split between WebDriver and VRSimulator.js looks like here. I would have expected that if the WebDriver API covered all testing needs, then no implementation-defined extra VRSimulator.js would be required, just some shared utility code that would wrap the common WebDriver API. @ddorwin, do you have a better understanding? (I fear I may just be adding confusion when y'all know what to do already.) |
|
@foolip, The VRSimulator.js file actually leverages WebDriver to perform platform specific actions relating to VR simulation, for example AttachVRDisplay. I am borrowing a snippet from @RafaelCintron who posted on this thread to provide further details:
This approach allows a single test page to test both the DOM API while interacting with the VR simulation layer. Other browser vendors are welcome to adopt our WebDriver approach, however Mozilla has already integrated our tests using their own custom test harness. @ddorwin has also expressed interest in adopting a custom webserver. With each browser taking their own approach with VR simulation, we cannot avoid platform specific behavior in VRSimulator.js. The initial goal of this pull request is to agree upon the VRSimulator API and release some basic WebVR tests. We have several additional tests in the pipeline that we would like to release in order to weed out any potential interop bugs as soon as possible. |
|
In terms of VRSimulator.js, Mozilla uses a test-only WebAPI for running these tests, and I am planning to move the custom test harness to our WPT suite folder after these tests are merged. Currently, Mozilla didn't run WebDriver on our try servers. That's the reason I wanna implement the test-only WebAPI at the first stage. But, I am still opened to provide these test APIs from VRSimulator.js for WebDriver after I assure our test suites are very stable. |
|
@SamiraAtMicrosoft, what would you think about changing wptrunner to interact with WebDriver in exactly the way you describe? @gsnedders and @NavidZ are looking into something similar for input automation. The UserConsentRequestPresent command mentioned is something I would love to use for Fullscreen as well. I don't want to stall this indefinitely, but it sounds like the mechanism should eventually be consolidated with that we'll be doing to input automation and eventually all other kinds of automation that are backed by WebDriver. If this is merged first to iterate, it'd be great if someone takes on the responsibility of merging the two in the end. I defer to @ddorwin to represent what makes sense for the Chromium project. |
|
@foolip – updating WPTRunner to work with WebDriver in this way would be a positive step for the community. I gather that the time estimate on this effort would not be trivial. Also, given that Mozilla has already merged our tests into their internal testing framework, I would lean towards merging our tests first so that we can begin to have some basic conformance across platforms. However I am curious to learn more about the efforts of @gsnedders and @NavidZ that you describe, and how they feel about the approach we have taken. |
|
@gsnedders, care to take a look at this? If one ignores everything about WebDriver, then this boils down to a testing API. https://lists.w3.org/Archives/Public/public-test-infra/2017AprJun/0080.html is relevant. If the behavior of the test API is specified in sufficient detail and two vendors (Microsoft and Mozilla) want to use it, then we should go ahead and merge it. I would make two changes:
|
|
What Mozilla did is very similar with The only thing users have to do is getting the VRServiceTest via |
|
Ping @gsnedders to take a look. @SamiraAtMicrosoft, any thoughts on my two bullet points above? |
|
Removing Defining a test API in the browser is an interesting idea and we are still exploring this on our end. |
|
@foolip oh, I hadn't responded because I was just agreed with what you said above! :) |
|
Very sorry for the delayed reply. Had an email organization mixup that buried this pull request. :( I think the approach that we'd like to take for Chromium is to continue using the abstract The one thing I would request is that we have some concrete definition of what the expected interface of VRSimulator.js is checked into the repo somewhere. This could be an example implementation, IDL, or even just a file full of well commented function stubs. That way we have a canonical reference of what needs to be exposed rather than browsing through the tests themselves to determine the required interface. |
|
@toji , @SamiraAtMicrosoft 's pull request contains a Readme.md file with the API along with a brief description of each function. Today, there are only three functions. As we (all) submit additional tests, we'll expand the API. |
|
I saw the description in the readme file, which is helpful, but (anticipating that the interface will need to grow over time) I still think that a file which clearly lists the expected interface in terms of function signatures would be valuable. |
|
I also agree with @toji that we should put VRSimulator.js back to our tests. @RafaelCintron I am curious I also feel these three interfaces are not enough for us to fulfill tests. We might need some configuration interfaces for supporting different targeted devices. Currently, Mozilla's APIs is like this, https://dxr.mozilla.org/mozilla-central/source/dom/vr/test/mochitest/VRSimulationDriver.js. |
An initial pass of WebVR V1.1 conformance tests. More tests to come.
These tests require an additional file - VRSimulator.js - that includes browser specific implementations of simulated VR actions. (Examples include: AttachWebVRDisplay and DetachWebVRDisplay. See the Readme.md for the full list.)
Note that these WebVR tests will fail during bot runs because VRSimulator.js is not present.