New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial pass of WebVR 1.1 conformance tests #5535

Open
wants to merge 4 commits into
from

Conversation

Projects
None yet
9 participants
@SamiraAtMicrosoft

SamiraAtMicrosoft commented Apr 11, 2017

An initial pass of WebVR V1.1 conformance tests. More tests to come.

These tests require an additional file - VRSimulator.js - that includes browser specific implementations of simulated VR actions. (Examples include: AttachWebVRDisplay and DetachWebVRDisplay. See the Readme.md for the full list.)

Note that these WebVR tests will fail during bot runs because VRSimulator.js is not present.

First pass WebVR conformance tests
An initial drop of WebVR V1.1 conformance tests. These tests require an additional
file - VRSimulator.js - that includes browser specific implementations of simulated
VR actions. (Examples include: AttachWebVRDisplay and DetachWebVRDisplay. See the
Readme.md for the full list.)

A larger suite of tests are coming soon.
@wpt-pr-bot

This comment has been minimized.

Show comment
Hide comment
Collaborator

wpt-pr-bot commented Apr 11, 2017

@wpt-pr-bot wpt-pr-bot added the webvr label Apr 11, 2017

@w3c-bots

This comment has been minimized.

Show comment
Hide comment
@w3c-bots

w3c-bots Apr 11, 2017

View the complete job log.

Firefox (nightly channel)

Testing web-platform-tests at revision badb452
Using browser at version BuildID 20170430100638; SourceStamp 2fe636103d7167f3a5d57f61bd19fddcc878ca3c
Starting 10 test iterations
All results were stable

All results

3 tests ran
/webvr/frozenArray_activeVRDisplays.html
Subtest Results Messages
OK
FrozenArray<VRDisplay> Navigator.activeVRDisplays Test FAIL VRSimulator is not defined
/webvr/getLayers.html
Subtest Results Messages
OK
Test VRDisplay.getLayers() with leftBounds = undefined, rightBounds = undefined FAIL VRSimulator is not defined
Test VRDisplay.getLayers() with leftBounds = 0.3,0.2,0.5,0.7, rightBounds = 0.8,0.25,0.2,0.75 FAIL VRSimulator is not defined
Test VRDisplay.getLayers() with leftBounds = 0.3,0.2,0.5,0.7, rightBounds = undefined FAIL VRSimulator is not defined
Test VRDisplay.getLayers() with leftBounds = undefined, rightBounds = 0.8,0.25,0.2,0.75 FAIL VRSimulator is not defined
/webvr/requestPresent.html
Subtest Results Messages
OK
WebVR requestPresent fulfilled FAIL VRSimulator is not defined
WebVR requestPresent rejected with empty layers FAIL VRSimulator is not defined
WebVR requestPresent rejected with incorrect bounds (bounds arrays must be 0 or 4 long) FAIL VRSimulator is not defined
WebVR requestPresent rejected with invalid source (must be canvas element) FAIL VRSimulator is not defined
WebVR requestPresent rejected with invalid bounds data type (must be able to convert to float array) FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [2,0,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [2,0,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [0,2,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [0,2,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [0,0,2,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [0,0,2,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [0,0,0,2] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [0,0,0,2] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [-1,0,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [-1,0,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [0,-1,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [0,-1,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [0,0,-1,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [0,0,-1,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [0,0,0,-1] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [0,0,0,-1] FAIL VRSimulator is not defined
WebVR requestPresent rejected without user initiated action FAIL VRSimulator is not defined
WebVR requestPresent rejected with more layers than max layers FAIL VRSimulator is not defined
WebVR requestPresent fails while another one is in progress FAIL VRSimulator is not defined

w3c-bots commented Apr 11, 2017

View the complete job log.

Firefox (nightly channel)

Testing web-platform-tests at revision badb452
Using browser at version BuildID 20170430100638; SourceStamp 2fe636103d7167f3a5d57f61bd19fddcc878ca3c
Starting 10 test iterations
All results were stable

All results

3 tests ran
/webvr/frozenArray_activeVRDisplays.html
Subtest Results Messages
OK
FrozenArray<VRDisplay> Navigator.activeVRDisplays Test FAIL VRSimulator is not defined
/webvr/getLayers.html
Subtest Results Messages
OK
Test VRDisplay.getLayers() with leftBounds = undefined, rightBounds = undefined FAIL VRSimulator is not defined
Test VRDisplay.getLayers() with leftBounds = 0.3,0.2,0.5,0.7, rightBounds = 0.8,0.25,0.2,0.75 FAIL VRSimulator is not defined
Test VRDisplay.getLayers() with leftBounds = 0.3,0.2,0.5,0.7, rightBounds = undefined FAIL VRSimulator is not defined
Test VRDisplay.getLayers() with leftBounds = undefined, rightBounds = 0.8,0.25,0.2,0.75 FAIL VRSimulator is not defined
/webvr/requestPresent.html
Subtest Results Messages
OK
WebVR requestPresent fulfilled FAIL VRSimulator is not defined
WebVR requestPresent rejected with empty layers FAIL VRSimulator is not defined
WebVR requestPresent rejected with incorrect bounds (bounds arrays must be 0 or 4 long) FAIL VRSimulator is not defined
WebVR requestPresent rejected with invalid source (must be canvas element) FAIL VRSimulator is not defined
WebVR requestPresent rejected with invalid bounds data type (must be able to convert to float array) FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [2,0,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [2,0,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [0,2,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [0,2,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [0,0,2,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [0,0,2,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [0,0,0,2] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [0,0,0,2] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [-1,0,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [-1,0,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [0,-1,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [0,-1,0,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [0,0,-1,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [0,0,-1,0] FAIL VRSimulator is not defined
WebVR requestPresent rejected with left bounds in invalid range: [0,0,0,-1] FAIL VRSimulator is not defined
WebVR requestPresent rejected with right bounds in invalid range: [0,0,0,-1] FAIL VRSimulator is not defined
WebVR requestPresent rejected without user initiated action FAIL VRSimulator is not defined
WebVR requestPresent rejected with more layers than max layers FAIL VRSimulator is not defined
WebVR requestPresent fails while another one is in progress FAIL VRSimulator is not defined
@w3c-bots

This comment has been minimized.

Show comment
Hide comment
@w3c-bots

w3c-bots Apr 11, 2017

View the complete job log.

Chrome (unstable channel)

Testing web-platform-tests at revision badb452
Using browser at version 60.0.3080.5 dev
Starting 10 test iterations
All results were stable

All results

3 tests ran
/webvr/frozenArray_activeVRDisplays.html
Subtest Results Messages
OK
FrozenArray<VRDisplay> Navigator.activeVRDisplays Test FAIL VRSimulator is not defined
/webvr/getLayers.html
Subtest Results Messages
OK
Test VRDisplay.getLayers() with leftBounds = undefined, rightBounds = undefined FAIL VRSimulator is not defined
Test VRDisplay.getLayers() with leftBounds = 0.3,0.2,0.5,0.7, rightBounds = 0.8,0.25,0.2,0.75 FAIL VRSimulator is not defined
Test VRDisplay.getLayers() with leftBounds = 0.3,0.2,0.5,0.7, rightBounds = undefined FAIL VRSimulator is not defined
Test VRDisplay.getLayers() with leftBounds = undefined, rightBounds = 0.8,0.25,0.2,0.75 FAIL VRSimulator is not defined
/webvr/requestPresent.html
Subtest Results Messages
OK
WebVR requestPresent fulfilled FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with empty layers FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with incorrect bounds (bounds arrays must be 0 or 4 long) FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with invalid source (must be canvas element) FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with invalid bounds data type (must be able to convert to float array) FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [2,0,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [2,0,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [0,2,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [0,2,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [0,0,2,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [0,0,2,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [0,0,0,2] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [0,0,0,2] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [-1,0,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [-1,0,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [0,-1,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [0,-1,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [0,0,-1,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [0,0,-1,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [0,0,0,-1] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [0,0,0,-1] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected without user initiated action FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with more layers than max layers FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent fails while another one is in progress FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"

w3c-bots commented Apr 11, 2017

View the complete job log.

Chrome (unstable channel)

Testing web-platform-tests at revision badb452
Using browser at version 60.0.3080.5 dev
Starting 10 test iterations
All results were stable

All results

3 tests ran
/webvr/frozenArray_activeVRDisplays.html
Subtest Results Messages
OK
FrozenArray<VRDisplay> Navigator.activeVRDisplays Test FAIL VRSimulator is not defined
/webvr/getLayers.html
Subtest Results Messages
OK
Test VRDisplay.getLayers() with leftBounds = undefined, rightBounds = undefined FAIL VRSimulator is not defined
Test VRDisplay.getLayers() with leftBounds = 0.3,0.2,0.5,0.7, rightBounds = 0.8,0.25,0.2,0.75 FAIL VRSimulator is not defined
Test VRDisplay.getLayers() with leftBounds = 0.3,0.2,0.5,0.7, rightBounds = undefined FAIL VRSimulator is not defined
Test VRDisplay.getLayers() with leftBounds = undefined, rightBounds = 0.8,0.25,0.2,0.75 FAIL VRSimulator is not defined
/webvr/requestPresent.html
Subtest Results Messages
OK
WebVR requestPresent fulfilled FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with empty layers FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with incorrect bounds (bounds arrays must be 0 or 4 long) FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with invalid source (must be canvas element) FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with invalid bounds data type (must be able to convert to float array) FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [2,0,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [2,0,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [0,2,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [0,2,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [0,0,2,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [0,0,2,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [0,0,0,2] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [0,0,0,2] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [-1,0,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [-1,0,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [0,-1,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [0,-1,0,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [0,0,-1,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [0,0,-1,0] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with left bounds in invalid range: [0,0,0,-1] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with right bounds in invalid range: [0,0,0,-1] FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected without user initiated action FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent rejected with more layers than max layers FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
WebVR requestPresent fails while another one is in progress FAIL assert_equals: 'navigator.getVRDisplays()' must be defined. expected "function" but got "undefined"
@daoshengmu

This comment has been minimized.

Show comment
Hide comment
@daoshengmu

daoshengmu Apr 13, 2017

Reviewed 6 of 6 files at r1.
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.


webvr/frozenArray_activeVRDisplays.html, line 32 at r1 (raw file):

                verifyArrayNotFrozen(displays);

This test looks good to me. I am just curious why you wanna verify the array is not frozen in WebVR test. That seems to be not related to the field of WebVR.


Comments from Reviewable

Reviewed 6 of 6 files at r1.
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.


webvr/frozenArray_activeVRDisplays.html, line 32 at r1 (raw file):

                verifyArrayNotFrozen(displays);

This test looks good to me. I am just curious why you wanna verify the array is not frozen in WebVR test. That seems to be not related to the field of WebVR.


Comments from Reviewable

@RafaelCintron

This comment has been minimized.

Show comment
Hide comment
@RafaelCintron

RafaelCintron Apr 14, 2017

Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.


webvr/frozenArray_activeVRDisplays.html, line 32 at r1 (raw file):

Previously, daoshengmu (Daosheng Mu) wrote…

This test looks good to me. I am just curious why you wanna verify the array is not frozen in WebVR test. That seems to be not related to the field of WebVR.

The reason for the test is to verify the array returned from the API is truly Frozen and cannot have any elements add or removed.


Comments from Reviewable

Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.


webvr/frozenArray_activeVRDisplays.html, line 32 at r1 (raw file):

Previously, daoshengmu (Daosheng Mu) wrote…

This test looks good to me. I am just curious why you wanna verify the array is not frozen in WebVR test. That seems to be not related to the field of WebVR.

The reason for the test is to verify the array returned from the API is truly Frozen and cannot have any elements add or removed.


Comments from Reviewable

@daoshengmu

This comment has been minimized.

Show comment
Hide comment
@daoshengmu

daoshengmu Apr 19, 2017

:lgtm:


Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.


webvr/frozenArray_activeVRDisplays.html, line 32 at r1 (raw file):

Previously, RafaelCintron (Rafael Cintron) wrote…

The reason for the test is to verify the array returned from the API is truly Frozen and cannot have any elements add or removed.

ok. That makes sense.


Comments from Reviewable

:lgtm:


Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.


webvr/frozenArray_activeVRDisplays.html, line 32 at r1 (raw file):

Previously, RafaelCintron (Rafael Cintron) wrote…

The reason for the test is to verify the array returned from the API is truly Frozen and cannot have any elements add or removed.

ok. That makes sense.


Comments from Reviewable

@gsnedders

This comment has been minimized.

Show comment
Hide comment
@gsnedders

gsnedders Apr 24, 2017

Contributor

Can you please fix the lint failure that's leading to Travis to fail? i.e.,

webvr/requestPresent.html:21: Whitespace at EOL (TRAILING WHITESPACE)
Contributor

gsnedders commented Apr 24, 2017

Can you please fix the lint failure that's leading to Travis to fail? i.e.,

webvr/requestPresent.html:21: Whitespace at EOL (TRAILING WHITESPACE)
@w3c-bots

This comment has been minimized.

Show comment
Hide comment
@w3c-bots

w3c-bots Apr 25, 2017

These tests are now available on w3c-test.org

w3c-bots commented Apr 25, 2017

These tests are now available on w3c-test.org

@w3c-bots

This comment has been minimized.

Show comment
Hide comment

w3c-bots commented Apr 25, 2017

@gsnedders

This comment has been minimized.

Show comment
Hide comment
@gsnedders

gsnedders Apr 25, 2017

Contributor

cc/ @jgraham @foolip

Similar to discussions on public-test-infra about automating new web UIs, this relies on a utils/VRSimulator.js (not provided) to actually run the tests.

GitHub seems to think the Readme.md is a binary file (and probably git does as well); any idea why?

Contributor

gsnedders commented Apr 25, 2017

cc/ @jgraham @foolip

Similar to discussions on public-test-infra about automating new web UIs, this relies on a utils/VRSimulator.js (not provided) to actually run the tests.

GitHub seems to think the Readme.md is a binary file (and probably git does as well); any idea why?

@foolip

This comment has been minimized.

Show comment
Hide comment
@foolip

foolip Apr 26, 2017

Contributor

@SamiraAtMicrosoft, did you check the "Allow edits from maintainers" checkbox when opening this PR? See https://github.com/blog/2247-improving-collaboration-with-forks. I tried to fix the Readme.md problem, but couldn't push the fix. The problem is that it's a UTF-16 file, can be fixed with iconv -f UTF-16 -t ASCII.

Contributor

foolip commented Apr 26, 2017

@SamiraAtMicrosoft, did you check the "Allow edits from maintainers" checkbox when opening this PR? See https://github.com/blog/2247-improving-collaboration-with-forks. I tried to fix the Readme.md problem, but couldn't push the fix. The problem is that it's a UTF-16 file, can be fixed with iconv -f UTF-16 -t ASCII.

@foolip

This comment has been minimized.

Show comment
Hide comment
@foolip

foolip Apr 26, 2017

Contributor

@SamiraAtMicrosoft, this approach is similar to one that has been discussed on public-test-infra and in Chromium recently:
https://lists.w3.org/Archives/Public/public-test-infra/2017JanMar/thread.html#msg30
https://lists.w3.org/Archives/Public/public-test-infra/2017AprJun/thread.html#msg0
https://groups.google.com/a/chromium.org/d/msg/platform-predictability/TZhAWLpInFs/Y50c2mBeAwAJ

And for Bluetooth specifically:
https://lists.w3.org/Archives/Public/public-test-infra/2017AprJun/thread.html#msg18

I think what's needed here is a discussion between the implementers of WebVR to agree on a framework for tests. For Chromium, that's @toji and @kenrussell.

Contributor

foolip commented Apr 26, 2017

@SamiraAtMicrosoft, this approach is similar to one that has been discussed on public-test-infra and in Chromium recently:
https://lists.w3.org/Archives/Public/public-test-infra/2017JanMar/thread.html#msg30
https://lists.w3.org/Archives/Public/public-test-infra/2017AprJun/thread.html#msg0
https://groups.google.com/a/chromium.org/d/msg/platform-predictability/TZhAWLpInFs/Y50c2mBeAwAJ

And for Bluetooth specifically:
https://lists.w3.org/Archives/Public/public-test-infra/2017AprJun/thread.html#msg18

I think what's needed here is a discussion between the implementers of WebVR to agree on a framework for tests. For Chromium, that's @toji and @kenrussell.

@foolip

This comment has been minimized.

Show comment
Hide comment
@foolip

foolip Apr 26, 2017

Contributor

FWIW, I think that if the testing APIs were written into a spec and well defined, then an approach like this should be workable. Chromium's layout tests in https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/vr/ seem to use a similar approach.

Contributor

foolip commented Apr 26, 2017

FWIW, I think that if the testing APIs were written into a spec and well defined, then an approach like this should be workable. Chromium's layout tests in https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/vr/ seem to use a similar approach.

@RafaelCintron

This comment has been minimized.

Show comment
Hide comment
@RafaelCintron

RafaelCintron May 1, 2017

@foolip , we have been discussing WebVR tests in the WebVR group as part of the Interoperable WebVR Testing #187.

At Mozilla, @daoshengmu has already integrated a subset of the tests into Mozilla's testing infrastructure.

For Chromium, @ddorwin expressed interest in developing a web server that communicates with a OpenVR test driver.

At Microsoft, we've been running these tests internally using WebDriver and Perception Simulation

I think the current structure of the tests provides flexibility in how browsers structure the backend simulation of device. But, we're happy to hear feedback from others.

RafaelCintron commented May 1, 2017

@foolip , we have been discussing WebVR tests in the WebVR group as part of the Interoperable WebVR Testing #187.

At Mozilla, @daoshengmu has already integrated a subset of the tests into Mozilla's testing infrastructure.

For Chromium, @ddorwin expressed interest in developing a web server that communicates with a OpenVR test driver.

At Microsoft, we've been running these tests internally using WebDriver and Perception Simulation

I think the current structure of the tests provides flexibility in how browsers structure the backend simulation of device. But, we're happy to hear feedback from others.

@codecov-io

This comment has been minimized.

Show comment
Hide comment
@codecov-io

codecov-io May 1, 2017

Codecov Report

Merging #5535 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #5535   +/-   ##
=======================================
  Coverage   87.22%   87.22%           
=======================================
  Files          24       24           
  Lines        2418     2418           
  Branches      406      406           
=======================================
  Hits         2109     2109           
- Misses        239      240    +1     
+ Partials       70       69    -1
Impacted Files Coverage Δ
tools/lint/fnmatch.py 71.42% <0%> (ø) ⬆️
...ebsocket/src/mod_pywebsocket/handshake/__init__.py
...ple/assertion/global_testmodule_config/conftest.py
tools/__init__.py 100% <0%> (ø)
tools/conftest.py 100% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a69bb1a...8cc3904. Read the comment docs.

Codecov Report

Merging #5535 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #5535   +/-   ##
=======================================
  Coverage   87.22%   87.22%           
=======================================
  Files          24       24           
  Lines        2418     2418           
  Branches      406      406           
=======================================
  Hits         2109     2109           
- Misses        239      240    +1     
+ Partials       70       69    -1
Impacted Files Coverage Δ
tools/lint/fnmatch.py 71.42% <0%> (ø) ⬆️
...ebsocket/src/mod_pywebsocket/handshake/__init__.py
...ple/assertion/global_testmodule_config/conftest.py
tools/__init__.py 100% <0%> (ø)
tools/conftest.py 100% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a69bb1a...8cc3904. Read the comment docs.

@foolip

This comment has been minimized.

Show comment
Hide comment
@foolip

foolip May 3, 2017

Contributor

@SamiraAtMicrosoft @RafaelCintron, I've looked over immersive-web/webxr#187 and find that very encouraging. Do you think you could send a message to public-test-infra about the proposed setup, maybe just linking to this PR and the previous discussion?

If something in the shape of a spec is written that defines these APIs, and web-platform-tests just assumes those APIs exist, that would be a good start I think. There should also be some way of enabling such test APIs that's done in wptrunner, so that the tests can run using the actual releases of various browsers.

@toji, what do you think about all of this?

Contributor

foolip commented May 3, 2017

@SamiraAtMicrosoft @RafaelCintron, I've looked over immersive-web/webxr#187 and find that very encouraging. Do you think you could send a message to public-test-infra about the proposed setup, maybe just linking to this PR and the previous discussion?

If something in the shape of a spec is written that defines these APIs, and web-platform-tests just assumes those APIs exist, that would be a good start I think. There should also be some way of enabling such test APIs that's done in wptrunner, so that the tests can run using the actual releases of various browsers.

@toji, what do you think about all of this?

@foolip

This comment has been minimized.

Show comment
Hide comment
@foolip

foolip May 3, 2017

Contributor

(It looks like @toji will be OOO for about a week.)

Contributor

foolip commented May 3, 2017

(It looks like @toji will be OOO for about a week.)

@RafaelCintron

This comment has been minimized.

Show comment
Hide comment
@RafaelCintron

RafaelCintron May 3, 2017

@foolip , the readme file in the pull request has description of the APIs the test runner is meant to implement.

We'll send an e-mail to public-test-infra.

@foolip , the readme file in the pull request has description of the APIs the test runner is meant to implement.

We'll send an e-mail to public-test-infra.

@foolip

This comment has been minimized.

Show comment
Hide comment
@foolip

foolip May 12, 2017

Contributor

I see https://lists.w3.org/Archives/Public/public-test-infra/2017AprJun/0033.html was sent a week ago now, but it didn't show up in my inbox so I can't respond there.

I'm curious about what the split between WebDriver and VRSimulator.js looks like here. I would have expected that if the WebDriver API covered all testing needs, then no implementation-defined extra VRSimulator.js would be required, just some shared utility code that would wrap the common WebDriver API.

@ddorwin, do you have a better understanding? (I fear I may just be adding confusion when y'all know what to do already.)

Contributor

foolip commented May 12, 2017

I see https://lists.w3.org/Archives/Public/public-test-infra/2017AprJun/0033.html was sent a week ago now, but it didn't show up in my inbox so I can't respond there.

I'm curious about what the split between WebDriver and VRSimulator.js looks like here. I would have expected that if the WebDriver API covered all testing needs, then no implementation-defined extra VRSimulator.js would be required, just some shared utility code that would wrap the common WebDriver API.

@ddorwin, do you have a better understanding? (I fear I may just be adding confusion when y'all know what to do already.)

@SamiraAtMicrosoft

This comment has been minimized.

Show comment
Hide comment
@SamiraAtMicrosoft

SamiraAtMicrosoft Jun 5, 2017

@foolip, The VRSimulator.js file actually leverages WebDriver to perform platform specific actions relating to VR simulation, for example AttachVRDisplay. I am borrowing a snippet from @RafaelCintron who posted on this thread to provide further details:

Edge interacts with WebDriver in a unique way compared to standard WebDriver. At the beginning of a test, the WebDriver client sends the ExecuteAsyncScript WebDriver command to browser. The client patiently waits for a JSON response. For a simple test, the response is typically, "I passed", or "I failed". But, the response can also be AttachVRDisplay, DetachVRDisplay, SetPositionAndRotation, etc. As you can read from my test sample, all of this is done using async promises in the test page itself. Once the client runs the command, it sends another ExecuteAsyncScriptScript command to the browser to continue the test. The test ends when the final pass/fail response gets sent to the WebDriver client by the browser.

This approach allows a single test page to test both the DOM API while interacting with the VR simulation layer. Other browser vendors are welcome to adopt our WebDriver approach, however Mozilla has already integrated our tests using their own custom test harness. @ddorwin has also expressed interest in adopting a custom webserver. With each browser taking their own approach with VR simulation, we cannot avoid platform specific behavior in VRSimulator.js.

The initial goal of this pull request is to agree upon the VRSimulator API and release some basic WebVR tests. We have several additional tests in the pipeline that we would like to release in order to weed out any potential interop bugs as soon as possible.

SamiraAtMicrosoft commented Jun 5, 2017

@foolip, The VRSimulator.js file actually leverages WebDriver to perform platform specific actions relating to VR simulation, for example AttachVRDisplay. I am borrowing a snippet from @RafaelCintron who posted on this thread to provide further details:

Edge interacts with WebDriver in a unique way compared to standard WebDriver. At the beginning of a test, the WebDriver client sends the ExecuteAsyncScript WebDriver command to browser. The client patiently waits for a JSON response. For a simple test, the response is typically, "I passed", or "I failed". But, the response can also be AttachVRDisplay, DetachVRDisplay, SetPositionAndRotation, etc. As you can read from my test sample, all of this is done using async promises in the test page itself. Once the client runs the command, it sends another ExecuteAsyncScriptScript command to the browser to continue the test. The test ends when the final pass/fail response gets sent to the WebDriver client by the browser.

This approach allows a single test page to test both the DOM API while interacting with the VR simulation layer. Other browser vendors are welcome to adopt our WebDriver approach, however Mozilla has already integrated our tests using their own custom test harness. @ddorwin has also expressed interest in adopting a custom webserver. With each browser taking their own approach with VR simulation, we cannot avoid platform specific behavior in VRSimulator.js.

The initial goal of this pull request is to agree upon the VRSimulator API and release some basic WebVR tests. We have several additional tests in the pipeline that we would like to release in order to weed out any potential interop bugs as soon as possible.

@daoshengmu

This comment has been minimized.

Show comment
Hide comment
@daoshengmu

daoshengmu Jun 7, 2017

In terms of VRSimulator.js, Mozilla uses a test-only WebAPI for running these tests, and I am planning to move the custom test harness to our WPT suite folder after these tests are merged. Currently, Mozilla didn't run WebDriver on our try servers. That's the reason I wanna implement the test-only WebAPI at the first stage. But, I am still opened to provide these test APIs from VRSimulator.js for WebDriver after I assure our test suites are very stable.

daoshengmu commented Jun 7, 2017

In terms of VRSimulator.js, Mozilla uses a test-only WebAPI for running these tests, and I am planning to move the custom test harness to our WPT suite folder after these tests are merged. Currently, Mozilla didn't run WebDriver on our try servers. That's the reason I wanna implement the test-only WebAPI at the first stage. But, I am still opened to provide these test APIs from VRSimulator.js for WebDriver after I assure our test suites are very stable.

@foolip

This comment has been minimized.

Show comment
Hide comment
@foolip

foolip Jun 7, 2017

Contributor

@SamiraAtMicrosoft, what would you think about changing wptrunner to interact with WebDriver in exactly the way you describe? @gsnedders and @NavidZ are looking into something similar for input automation. The UserConsentRequestPresent command mentioned is something I would love to use for Fullscreen as well.

I don't want to stall this indefinitely, but it sounds like the mechanism should eventually be consolidated with that we'll be doing to input automation and eventually all other kinds of automation that are backed by WebDriver. If this is merged first to iterate, it'd be great if someone takes on the responsibility of merging the two in the end.

I defer to @ddorwin to represent what makes sense for the Chromium project.

Contributor

foolip commented Jun 7, 2017

@SamiraAtMicrosoft, what would you think about changing wptrunner to interact with WebDriver in exactly the way you describe? @gsnedders and @NavidZ are looking into something similar for input automation. The UserConsentRequestPresent command mentioned is something I would love to use for Fullscreen as well.

I don't want to stall this indefinitely, but it sounds like the mechanism should eventually be consolidated with that we'll be doing to input automation and eventually all other kinds of automation that are backed by WebDriver. If this is merged first to iterate, it'd be great if someone takes on the responsibility of merging the two in the end.

I defer to @ddorwin to represent what makes sense for the Chromium project.

@SamiraAtMicrosoft

This comment has been minimized.

Show comment
Hide comment
@SamiraAtMicrosoft

SamiraAtMicrosoft Jun 14, 2017

@foolip – updating WPTRunner to work with WebDriver in this way would be a positive step for the community. I gather that the time estimate on this effort would not be trivial. Also, given that Mozilla has already merged our tests into their internal testing framework, I would lean towards merging our tests first so that we can begin to have some basic conformance across platforms. However I am curious to learn more about the efforts of @gsnedders and @NavidZ that you describe, and how they feel about the approach we have taken.

@foolip – updating WPTRunner to work with WebDriver in this way would be a positive step for the community. I gather that the time estimate on this effort would not be trivial. Also, given that Mozilla has already merged our tests into their internal testing framework, I would lean towards merging our tests first so that we can begin to have some basic conformance across platforms. However I am curious to learn more about the efforts of @gsnedders and @NavidZ that you describe, and how they feel about the approach we have taken.

@foolip

This comment has been minimized.

Show comment
Hide comment
@foolip

foolip Jun 16, 2017

Contributor

@gsnedders, care to take a look at this?

If one ignores everything about WebDriver, then this boils down to a testing API. https://lists.w3.org/Archives/Public/public-test-infra/2017AprJun/0080.html is relevant.

If the behavior of the test API is specified in sufficient detail and two vendors (Microsoft and Mozilla) want to use it, then we should go ahead and merge it. I would make two changes:

  • Remove <script src="utils/VRSimulator.js"></script> since it isn't actually there, instead just assume the API exists.
  • Following the style of https://wicg.github.io/webusb/test/, put all the testing APIs under navigator.testVR or perhaps a static VRDisplay.test.
Contributor

foolip commented Jun 16, 2017

@gsnedders, care to take a look at this?

If one ignores everything about WebDriver, then this boils down to a testing API. https://lists.w3.org/Archives/Public/public-test-infra/2017AprJun/0080.html is relevant.

If the behavior of the test API is specified in sufficient detail and two vendors (Microsoft and Mozilla) want to use it, then we should go ahead and merge it. I would make two changes:

  • Remove <script src="utils/VRSimulator.js"></script> since it isn't actually there, instead just assume the API exists.
  • Following the style of https://wicg.github.io/webusb/test/, put all the testing APIs under navigator.testVR or perhaps a static VRDisplay.test.
@daoshengmu

This comment has been minimized.

Show comment
Hide comment
@daoshengmu

daoshengmu Jun 20, 2017

What Mozilla did is very similar with webusb, we create a VRServiceTest.webidl, https://dxr.mozilla.org/mozilla-central/rev/72346a4d6afcc17504f911190dbf470100e879ec/dom/webidl/VRServiceTest.webidl, for defining all test APIs, and we call them in VRSimulationDriver.js.

The only thing users have to do is getting the VRServiceTest via VRServiceTest = navigator.requestVRServiceTest();

What Mozilla did is very similar with webusb, we create a VRServiceTest.webidl, https://dxr.mozilla.org/mozilla-central/rev/72346a4d6afcc17504f911190dbf470100e879ec/dom/webidl/VRServiceTest.webidl, for defining all test APIs, and we call them in VRSimulationDriver.js.

The only thing users have to do is getting the VRServiceTest via VRServiceTest = navigator.requestVRServiceTest();

@foolip

This comment has been minimized.

Show comment
Hide comment
@foolip

foolip Jun 30, 2017

Contributor

Ping @gsnedders to take a look. @SamiraAtMicrosoft, any thoughts on my two bullet points above?

Contributor

foolip commented Jun 30, 2017

Ping @gsnedders to take a look. @SamiraAtMicrosoft, any thoughts on my two bullet points above?

@SamiraAtMicrosoft

This comment has been minimized.

Show comment
Hide comment
@SamiraAtMicrosoft

SamiraAtMicrosoft Jun 30, 2017

Removing <script src="utils/VRSimulator.js"></script> could be fine. Though I prefer reducing divergence from the test files that we run, I don't have a strong opinion in this particular case.

Defining a test API in the browser is an interesting idea and we are still exploring this on our end.

Removing <script src="utils/VRSimulator.js"></script> could be fine. Though I prefer reducing divergence from the test files that we run, I don't have a strong opinion in this particular case.

Defining a test API in the browser is an interesting idea and we are still exploring this on our end.

@gsnedders

This comment has been minimized.

Show comment
Hide comment
@gsnedders

gsnedders Jun 30, 2017

Contributor

@foolip oh, I hadn't responded because I was just agreed with what you said above! :)

Contributor

gsnedders commented Jun 30, 2017

@foolip oh, I hadn't responded because I was just agreed with what you said above! :)

@toji

This comment has been minimized.

Show comment
Hide comment
@toji

toji Jul 17, 2017

Contributor

Very sorry for the delayed reply. Had an email organization mixup that buried this pull request. :(

I think the approach that we'd like to take for Chromium is to continue using the abstract VRSimulator.js and have our own implementation that's backed by our Mojo mocking interface. That turns out to be cheaper in terms of binary distribution size than continuing to make our webdriver interface ever larger.

The one thing I would request is that we have some concrete definition of what the expected interface of VRSimulator.js is checked into the repo somewhere. This could be an example implementation, IDL, or even just a file full of well commented function stubs. That way we have a canonical reference of what needs to be exposed rather than browsing through the tests themselves to determine the required interface.

Contributor

toji commented Jul 17, 2017

Very sorry for the delayed reply. Had an email organization mixup that buried this pull request. :(

I think the approach that we'd like to take for Chromium is to continue using the abstract VRSimulator.js and have our own implementation that's backed by our Mojo mocking interface. That turns out to be cheaper in terms of binary distribution size than continuing to make our webdriver interface ever larger.

The one thing I would request is that we have some concrete definition of what the expected interface of VRSimulator.js is checked into the repo somewhere. This could be an example implementation, IDL, or even just a file full of well commented function stubs. That way we have a canonical reference of what needs to be exposed rather than browsing through the tests themselves to determine the required interface.

@RafaelCintron

This comment has been minimized.

Show comment
Hide comment
@RafaelCintron

RafaelCintron Jul 18, 2017

@toji , @SamiraAtMicrosoft 's pull request contains a Readme.md file with the API along with a brief description of each function. Today, there are only three functions. As we (all) submit additional tests, we'll expand the API.

RafaelCintron commented Jul 18, 2017

@toji , @SamiraAtMicrosoft 's pull request contains a Readme.md file with the API along with a brief description of each function. Today, there are only three functions. As we (all) submit additional tests, we'll expand the API.

@toji

This comment has been minimized.

Show comment
Hide comment
@toji

toji Jul 18, 2017

Contributor

I saw the description in the readme file, which is helpful, but (anticipating that the interface will need to grow over time) I still think that a file which clearly lists the expected interface in terms of function signatures would be valuable.

Contributor

toji commented Jul 18, 2017

I saw the description in the readme file, which is helpful, but (anticipating that the interface will need to grow over time) I still think that a file which clearly lists the expected interface in terms of function signatures would be valuable.

@daoshengmu

This comment has been minimized.

Show comment
Hide comment
@daoshengmu

daoshengmu Jul 19, 2017

I also agree with @toji that we should put VRSimulator.js back to our tests.

@RafaelCintron I am curious UserConsentRequestPresent interface. It seems to take no parameters as well, right?

I also feel these three interfaces are not enough for us to fulfill tests. We might need some configuration interfaces for supporting different targeted devices. Currently, Mozilla's APIs is like this, https://dxr.mozilla.org/mozilla-central/source/dom/vr/test/mochitest/VRSimulationDriver.js.

I also agree with @toji that we should put VRSimulator.js back to our tests.

@RafaelCintron I am curious UserConsentRequestPresent interface. It seems to take no parameters as well, right?

I also feel these three interfaces are not enough for us to fulfill tests. We might need some configuration interfaces for supporting different targeted devices. Currently, Mozilla's APIs is like this, https://dxr.mozilla.org/mozilla-central/source/dom/vr/test/mochitest/VRSimulationDriver.js.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment