New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove "interact" flags for tests that can be run automatically #5370

Merged
merged 3 commits into from Apr 7, 2017

Conversation

Projects
None yet
5 participants
@mrego
Member

mrego commented Apr 4, 2017

There were a bunch of tests that have both a reference file and
the "interact" flag.
They use some small JavaScript to execute some steps
before checking the output against reference file.
This caused them to be wrongly considered manual tests
when they can be run automatically.

For more info check w3c/wpt-tools#196 and w3c/wpt-tools#197.


This change is Reviewable

@mrego mrego requested review from gsnedders and jgraham Apr 4, 2017

@w3c-bots

This comment has been minimized.

Show comment
Hide comment
@w3c-bots

w3c-bots Apr 4, 2017

View the complete job log.

Firefox (nightly channel)

Testing web-platform-tests at revision 4e42c4c
Using browser at version BuildID 20170406100208; SourceStamp 3c68d659c2b715f811708f043a1e7169d77be2ba
Starting 10 test iterations
All results were stable

All results

29 tests ran
/css/CSS2/backgrounds/background-root-012b.xht
Subtest Results Messages
PASS
/css/CSS2/backgrounds/background-root-013b.xht
Subtest Results Messages
PASS
/css/CSS2/backgrounds/background-root-014b.xht
Subtest Results Messages
PASS
/css/CSS2/backgrounds/background-root-017.xht
Subtest Results Messages
PASS
/css/css-regions-1/contentEditable/contentEditable-001.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-002.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-003.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-004.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-005.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-006.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-007.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-008.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-009.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-010.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-011.html
Subtest Results Messages
FAIL
/css/css-writing-modes-3/background-position-vrl-018.xht
Subtest Results Messages
PASS
/css/css-writing-modes-3/background-position-vrl-020.xht
Subtest Results Messages
PASS
/css/css-writing-modes-3/background-position-vrl-022.xht
Subtest Results Messages
PASS
/css/selectors4/focus-within-001.html
Subtest Results Messages
PASS
/css/selectors4/focus-within-002.html
Subtest Results Messages
PASS
/css/selectors4/focus-within-003.html
Subtest Results Messages
PASS
/css/selectors4/focus-within-004.html
Subtest Results Messages
PASS
/css/selectors4/focus-within-005.html
Subtest Results Messages
PASS
/css/selectors4/focus-within-006.html
Subtest Results Messages
PASS
/css/selectors4/focus-within-shadow-001.html
Subtest Results Messages
TIMEOUT
/css/selectors4/focus-within-shadow-002.html
Subtest Results Messages
TIMEOUT
/css/selectors4/focus-within-shadow-003.html
Subtest Results Messages
TIMEOUT
/css/selectors4/focus-within-shadow-004.html
Subtest Results Messages
TIMEOUT
/css/selectors4/focus-within-shadow-005.html
Subtest Results Messages
TIMEOUT

w3c-bots commented Apr 4, 2017

View the complete job log.

Firefox (nightly channel)

Testing web-platform-tests at revision 4e42c4c
Using browser at version BuildID 20170406100208; SourceStamp 3c68d659c2b715f811708f043a1e7169d77be2ba
Starting 10 test iterations
All results were stable

All results

29 tests ran
/css/CSS2/backgrounds/background-root-012b.xht
Subtest Results Messages
PASS
/css/CSS2/backgrounds/background-root-013b.xht
Subtest Results Messages
PASS
/css/CSS2/backgrounds/background-root-014b.xht
Subtest Results Messages
PASS
/css/CSS2/backgrounds/background-root-017.xht
Subtest Results Messages
PASS
/css/css-regions-1/contentEditable/contentEditable-001.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-002.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-003.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-004.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-005.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-006.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-007.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-008.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-009.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-010.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-011.html
Subtest Results Messages
FAIL
/css/css-writing-modes-3/background-position-vrl-018.xht
Subtest Results Messages
PASS
/css/css-writing-modes-3/background-position-vrl-020.xht
Subtest Results Messages
PASS
/css/css-writing-modes-3/background-position-vrl-022.xht
Subtest Results Messages
PASS
/css/selectors4/focus-within-001.html
Subtest Results Messages
PASS
/css/selectors4/focus-within-002.html
Subtest Results Messages
PASS
/css/selectors4/focus-within-003.html
Subtest Results Messages
PASS
/css/selectors4/focus-within-004.html
Subtest Results Messages
PASS
/css/selectors4/focus-within-005.html
Subtest Results Messages
PASS
/css/selectors4/focus-within-006.html
Subtest Results Messages
PASS
/css/selectors4/focus-within-shadow-001.html
Subtest Results Messages
TIMEOUT
/css/selectors4/focus-within-shadow-002.html
Subtest Results Messages
TIMEOUT
/css/selectors4/focus-within-shadow-003.html
Subtest Results Messages
TIMEOUT
/css/selectors4/focus-within-shadow-004.html
Subtest Results Messages
TIMEOUT
/css/selectors4/focus-within-shadow-005.html
Subtest Results Messages
TIMEOUT
@w3c-bots

This comment has been minimized.

Show comment
Hide comment
@w3c-bots

w3c-bots Apr 4, 2017

View the complete job log.

Chrome (unstable channel)

Testing web-platform-tests at revision 4e42c4c
Using browser at version 59.0.3063.4 dev
Starting 10 test iterations
All results were stable

All results

29 tests ran
/css/CSS2/backgrounds/background-root-012b.xht
Subtest Results Messages
PASS
/css/CSS2/backgrounds/background-root-013b.xht
Subtest Results Messages
PASS
/css/CSS2/backgrounds/background-root-014b.xht
Subtest Results Messages
PASS
/css/CSS2/backgrounds/background-root-017.xht
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-001.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-002.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-003.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-004.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-005.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-006.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-007.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-008.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-009.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-010.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-011.html
Subtest Results Messages
FAIL
/css/css-writing-modes-3/background-position-vrl-018.xht
Subtest Results Messages
PASS
/css/css-writing-modes-3/background-position-vrl-020.xht
Subtest Results Messages
PASS
/css/css-writing-modes-3/background-position-vrl-022.xht
Subtest Results Messages
PASS
/css/selectors4/focus-within-001.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-002.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-003.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-004.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-005.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-006.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-shadow-001.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-shadow-002.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-shadow-003.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-shadow-004.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-shadow-005.html
Subtest Results Messages
FAIL

w3c-bots commented Apr 4, 2017

View the complete job log.

Chrome (unstable channel)

Testing web-platform-tests at revision 4e42c4c
Using browser at version 59.0.3063.4 dev
Starting 10 test iterations
All results were stable

All results

29 tests ran
/css/CSS2/backgrounds/background-root-012b.xht
Subtest Results Messages
PASS
/css/CSS2/backgrounds/background-root-013b.xht
Subtest Results Messages
PASS
/css/CSS2/backgrounds/background-root-014b.xht
Subtest Results Messages
PASS
/css/CSS2/backgrounds/background-root-017.xht
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-001.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-002.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-003.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-004.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-005.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-006.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-007.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-008.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-009.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-010.html
Subtest Results Messages
FAIL
/css/css-regions-1/contentEditable/contentEditable-011.html
Subtest Results Messages
FAIL
/css/css-writing-modes-3/background-position-vrl-018.xht
Subtest Results Messages
PASS
/css/css-writing-modes-3/background-position-vrl-020.xht
Subtest Results Messages
PASS
/css/css-writing-modes-3/background-position-vrl-022.xht
Subtest Results Messages
PASS
/css/selectors4/focus-within-001.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-002.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-003.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-004.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-005.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-006.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-shadow-001.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-shadow-002.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-shadow-003.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-shadow-004.html
Subtest Results Messages
FAIL
/css/selectors4/focus-within-shadow-005.html
Subtest Results Messages
FAIL
@frivoal

I've reviewed this because some tests for focus within where in there, and I've worked on that feature and some of these tests. I've reviewed the rest of the PR while I was at it, but feedback from someone familiar with the tests / specs being tests would probably be good.

Please see the individual comments from adjustments that should be made before merging.

@@ -6,7 +6,6 @@
<link rel="author" title="Florian Rivoal" href="mailto:florian@rivoal.net">
<link rel="help" href="https://drafts.csswg.org/selectors-4/#focus-within-pseudo">
<link rel="match" href="focus-within-001-ref.html">
<meta name="flags" content="interact">

This comment has been minimized.

@frivoal

frivoal Apr 5, 2017

Contributor
  • It is better to switch to <meta name="flags" content=""> than to remove the line. Makes it clear that you have considered the question and decided that there was no need for flags, instead of maybe forgetting to think about it.

  • If you remove the interact flag, the comment above the javascript about it being an optional convenience becomes wrong and needs to be removed: either the test is interactive and the js is a optional helper, or it is automated and js is required.

  • As we switch to js required, it would be good to add class="reftest-wait" on the root, and remove it once js has run, so that we make sure that the ref comparison is done at the right time, and that if js fails to run for some reason, we detect it as a test that failed to run rather than a failure to implement focus-within correctly.

@frivoal

frivoal Apr 5, 2017

Contributor
  • It is better to switch to <meta name="flags" content=""> than to remove the line. Makes it clear that you have considered the question and decided that there was no need for flags, instead of maybe forgetting to think about it.

  • If you remove the interact flag, the comment above the javascript about it being an optional convenience becomes wrong and needs to be removed: either the test is interactive and the js is a optional helper, or it is automated and js is required.

  • As we switch to js required, it would be good to add class="reftest-wait" on the root, and remove it once js has run, so that we make sure that the ref comparison is done at the right time, and that if js fails to run for some reason, we detect it as a test that failed to run rather than a failure to implement focus-within correctly.

This comment has been minimized.

@mrego

mrego Apr 5, 2017

Member

Ok, I've done this in all the tests.

I'm still removing the empty flags, but I can add it if it's a blocker to land this. I have seen many tests that don't have it.

@mrego

mrego Apr 5, 2017

Member

Ok, I've done this in all the tests.

I'm still removing the empty flags, but I can add it if it's a blocker to land this. I have seen many tests that don't have it.

@@ -6,7 +6,6 @@
<link rel="author" title="Florian Rivoal" href="mailto:florian@rivoal.net">
<link rel="help" href="https://drafts.csswg.org/selectors-4/#focus-within-pseudo">
<link rel="match" href="focus-within-001-ref.html">
<meta name="flags" content="interact">

This comment has been minimized.

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as for focus-within-001

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as for focus-within-001

@@ -6,7 +6,6 @@
<link rel="author" title="Florian Rivoal" href="mailto:florian@rivoal.net">
<link rel="help" href="https://drafts.csswg.org/selectors-4/#focus-within-pseudo">
<link rel="match" href="focus-within-001-ref.html">
<meta name="flags" content="interact">

This comment has been minimized.

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as for focus-within-001

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as for focus-within-001

@@ -6,7 +6,6 @@
<link rel="author" title="Florian Rivoal" href="mailto:florian@rivoal.net">
<link rel="help" href="https://drafts.csswg.org/selectors-4/#focus-within-pseudo">
<link rel="match" href="focus-within-001-ref.html">
<meta name="flags" content="interact">

This comment has been minimized.

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as for focus-within-001

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as for focus-within-001

@@ -6,7 +6,6 @@
<link rel="author" title="Florian Rivoal" href="mailto:florian@rivoal.net">
<link rel="help" href="https://drafts.csswg.org/selectors-4/#focus-within-pseudo">
<link rel="match" href="focus-within-001-ref.html">
<meta name="flags" content="interact">

This comment has been minimized.

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as for focus-within-001

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as for focus-within-001

@@ -8,7 +8,7 @@
<link rel="help" href="http://www.w3.org/TR/css3-regions/#the-flow-into-property">
<link rel="help" href="http://www.w3.org/TR/css3-regions/#flow-from">
<link rel="help" href="http://www.w3.org/TR/html5/editing.html#contenteditable">
<meta name="flags" content="dom interact">

This comment has been minimized.

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as css-regions-1/contentEditable/contentEditable-001.html

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as css-regions-1/contentEditable/contentEditable-001.html

@@ -7,7 +7,7 @@
<link rel="help" href="http://www.w3.org/TR/css3-regions/#the-flow-into-property">
<link rel="help" href="http://www.w3.org/TR/css3-regions/#flow-from">
<link rel="help" href="http://www.w3.org/TR/html5/editing.html#contenteditable">
<meta name="flags" content="dom interact">

This comment has been minimized.

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as css-regions-1/contentEditable/contentEditable-001.html

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as css-regions-1/contentEditable/contentEditable-001.html

@@ -7,7 +7,7 @@
<link rel="help" href="http://www.w3.org/TR/css3-regions/#the-flow-into-property">
<link rel="help" href="http://www.w3.org/TR/css3-regions/#flow-from">
<link rel="help" href="http://www.w3.org/TR/html5/editing.html#contenteditable">
<meta name="flags" content="dom interact">

This comment has been minimized.

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as css-regions-1/contentEditable/contentEditable-001.html

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as css-regions-1/contentEditable/contentEditable-001.html

@@ -7,7 +7,7 @@
<link rel="help" href="http://www.w3.org/TR/css3-regions/#the-flow-into-property">
<link rel="help" href="http://www.w3.org/TR/css3-regions/#flow-from">
<link rel="help" href="http://www.w3.org/TR/html5/editing.html#contenteditable">
<meta name="flags" content="dom interact">

This comment has been minimized.

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as css-regions-1/contentEditable/contentEditable-001.html

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as css-regions-1/contentEditable/contentEditable-001.html

@@ -7,7 +7,7 @@
<link rel="help" href="http://www.w3.org/TR/css3-regions/#the-flow-into-property">
<link rel="help" href="http://www.w3.org/TR/css3-regions/#flow-from">
<link rel="help" href="http://www.w3.org/TR/html5/editing.html#contenteditable">
<meta name="flags" content="dom interact">

This comment has been minimized.

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as css-regions-1/contentEditable/contentEditable-001.html

@frivoal

frivoal Apr 5, 2017

Contributor

same comment as css-regions-1/contentEditable/contentEditable-001.html

@mrego

This comment has been minimized.

Show comment
Hide comment
@mrego

mrego Apr 5, 2017

Member

Thanks @frivoal for the thorough review. I'll do the changes, but I'm not 100% sure about the empty flags stuff:

  <meta name="flags" content="">

I cannot find any documentation saying that this is mandatory, and the initial examples in the doc don't use it.
@gsnedders is there any agreement regarding this topic?

Member

mrego commented Apr 5, 2017

Thanks @frivoal for the thorough review. I'll do the changes, but I'm not 100% sure about the empty flags stuff:

  <meta name="flags" content="">

I cannot find any documentation saying that this is mandatory, and the initial examples in the doc don't use it.
@gsnedders is there any agreement regarding this topic?

@w3c-bots

This comment has been minimized.

Show comment
Hide comment
@w3c-bots

w3c-bots Apr 5, 2017

These tests are now available on w3c-test.org

w3c-bots commented Apr 5, 2017

These tests are now available on w3c-test.org

@mrego

This comment has been minimized.

Show comment
Hide comment
@mrego

mrego Apr 5, 2017

Member

@frivoal I've applied the suggested changes, expect keeping the empty flags, and uploaded a new version.

Member

mrego commented Apr 5, 2017

@frivoal I've applied the suggested changes, expect keeping the empty flags, and uploaded a new version.

@gsnedders

This comment has been minimized.

Show comment
Hide comment
@gsnedders

gsnedders Apr 5, 2017

Contributor

Per the readme, we shouldn't be editing things in css/vendor-import. That said, I've got no idea if @dbaron is going to be importing any upstream changes any time soon, and how much of a pain exporting changes here in the meantime will be?

Contributor

gsnedders commented Apr 5, 2017

Per the readme, we shouldn't be editing things in css/vendor-import. That said, I've got no idea if @dbaron is going to be importing any upstream changes any time soon, and how much of a pain exporting changes here in the meantime will be?

@gsnedders

This comment has been minimized.

Show comment
Hide comment
@gsnedders

gsnedders Apr 5, 2017

Contributor

@mrego I don't think there's any real preference either way. I'd rather not have them bloating the file, on the whole.

Contributor

gsnedders commented Apr 5, 2017

@mrego I don't think there's any real preference either way. I'd rather not have them bloating the file, on the whole.

@frivoal

This comment has been minimized.

Show comment
Hide comment
@frivoal

frivoal Apr 5, 2017

Contributor

@mrego: based on @gsnedders's comment, can you back out the changes to the tests under vendor-imports/ These are meant to be fixed upstream, not here. Let's not cause any merge conflict for next time Mozilla wants to sync things. Maybe you want to send this fix directly to them? I don't know if you have access to Mozilla's repos.

Once that's done, I'll merge.

Contributor

frivoal commented Apr 5, 2017

@mrego: based on @gsnedders's comment, can you back out the changes to the tests under vendor-imports/ These are meant to be fixed upstream, not here. Let's not cause any merge conflict for next time Mozilla wants to sync things. Maybe you want to send this fix directly to them? I don't know if you have access to Mozilla's repos.

Once that's done, I'll merge.

@mrego

This comment has been minimized.

Show comment
Hide comment
@mrego

mrego Apr 5, 2017

Member

Ok, I've just reverted the changes on the Mozilla tests, I'll send them upstream to Firefox bugzilla.

Member

mrego commented Apr 5, 2017

Ok, I've just reverted the changes on the Mozilla tests, I'll send them upstream to Firefox bugzilla.

@frivoal

frivoal approved these changes Apr 5, 2017

@frivoal

This comment has been minimized.

Show comment
Hide comment
@frivoal

frivoal Apr 5, 2017

Contributor

@mrego Having some problems with travis failing tests. Can you rebase on top of the latest master to catch the fix for #5330 ?

Contributor

frivoal commented Apr 5, 2017

@mrego Having some problems with travis failing tests. Can you rebase on top of the latest master to catch the fix for #5330 ?

@frivoal

This comment has been minimized.

Show comment
Hide comment
@frivoal

frivoal Apr 5, 2017

Contributor

@mrego actually, I see that you've already merged master in. @gsnedders, any idea what travis is still complaining about?

Contributor

frivoal commented Apr 5, 2017

@mrego actually, I see that you've already merged master in. @gsnedders, any idea what travis is still complaining about?

@gsnedders

This comment has been minimized.

Show comment
Hide comment
@gsnedders

This comment has been minimized.

Show comment
Hide comment
@gsnedders

gsnedders Apr 5, 2017

Contributor

@frivoal the failure mode of #5330 is getting the "Unstable results" with nothing below it, for reference.

Contributor

gsnedders commented Apr 5, 2017

@frivoal the failure mode of #5330 is getting the "Unstable results" with nothing below it, for reference.

@mrego

This comment has been minimized.

Show comment
Hide comment
@mrego

mrego Apr 5, 2017

Member

I'm not sure if I'm checking the right thing, but the results I see don't make a lot of sense to me.
Specially the focus-within ones:

ERROR:check_stability:| `/css/selectors4/focus-within-001.html`                       |         | **FAIL: 6/10, PASS: 4/10** |          |

The feature is not implemented yet, so it's impossible that the test is passing.

Member

mrego commented Apr 5, 2017

I'm not sure if I'm checking the right thing, but the results I see don't make a lot of sense to me.
Specially the focus-within ones:

ERROR:check_stability:| `/css/selectors4/focus-within-001.html`                       |         | **FAIL: 6/10, PASS: 4/10** |          |

The feature is not implemented yet, so it's impossible that the test is passing.

@frivoal

This comment has been minimized.

Show comment
Hide comment
@frivoal

frivoal Apr 6, 2017

Contributor

@gsnedders @jugglinmike @mrego

The feature is not implemented yet, so it's impossible that the test is passing.

Right, the test results don't make sense to me either. Here and in other ref-tests PR I've been working on (#5285, #5283). If the Chrome stability test isn't working right due to jugglinmike/chrome-screenshot-race#1 or whatever, shouldn't we disable it until it is fixed? For now it's doing more harm than good.

Contributor

frivoal commented Apr 6, 2017

@gsnedders @jugglinmike @mrego

The feature is not implemented yet, so it's impossible that the test is passing.

Right, the test results don't make sense to me either. Here and in other ref-tests PR I've been working on (#5285, #5283). If the Chrome stability test isn't working right due to jugglinmike/chrome-screenshot-race#1 or whatever, shouldn't we disable it until it is fixed? For now it's doing more harm than good.

@gsnedders

This comment has been minimized.

Show comment
Hide comment
@gsnedders

gsnedders Apr 6, 2017

Contributor

The latest Travis build seems to have run into #5407

Contributor

gsnedders commented Apr 6, 2017

The latest Travis build seems to have run into #5407

xeonchen pushed a commit to mozilla-necko/gecko that referenced this pull request Apr 7, 2017

Bug 1354006 - [selectors4] Remove "interact" flag from :focus-within …
…tests. r=xidorn

This patch has been extracted from:
web-platform-tests/wpt#5370

The problem is that the "interact" flag on these tests
causes that they are considered manual,
when they can be run automatically.

This patch removes the "interact" flag and uses "reftest-wait" class
to be sure that the test has been completed.
@mrego

This comment has been minimized.

Show comment
Hide comment
@mrego

mrego Apr 7, 2017

Member

JFTR, Mozilla tests have been modified upstream: https://bugzilla.mozilla.org/show_bug.cgi?id=1354006

Member

mrego commented Apr 7, 2017

JFTR, Mozilla tests have been modified upstream: https://bugzilla.mozilla.org/show_bug.cgi?id=1354006

mrego added some commits Apr 4, 2017

Remove "interact" flags for tests that can be run automatically
There were a bunch of tests that have both a reference file and
the "interact" flag.
They use some small JavaScript to execute some steps
before checking the output against reference file.
This caused them to be wrongly considered manual tests
when they can be run automatically.

For more info check w3c/wpt-tools#196 and w3c/wpt-tools#197.
Apply suggested comments on review for patch that removes "interact" …
…flag

Basically this patch does the following:
* Duplicate some tests so now they have an automatic part and a manual one
  (as it's required to resize the window).
* Use "reftest-wait" class to be sure that the JavaScript steps
  have been completed before checking the reference.
* Remove some wrong "reftest-wait" used outside the root element.
* Remove some wrong comments now that the tests are automatic.

moz-v2v-gh pushed a commit to mozilla/gecko-dev that referenced this pull request Apr 7, 2017

Bug 1354006 - [selectors4] Remove "interact" flag from :focus-within …
…tests. r=xidorn

This patch has been extracted from:
web-platform-tests/wpt#5370

The problem is that the "interact" flag on these tests
causes that they are considered manual,
when they can be run automatically.

This patch removes the "interact" flag and uses "reftest-wait" class
to be sure that the test has been completed.
@mrego

This comment has been minimized.

Show comment
Hide comment
@mrego

mrego Apr 7, 2017

Member

Ok, it seems we can merge this now. Nice!

Member

mrego commented Apr 7, 2017

Ok, it seems we can merge this now. Nice!

@mrego mrego merged commit a460e79 into web-platform-tests:master Apr 7, 2017

1 of 2 checks passed

code-review/reviewable 38 files left
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@mrego mrego deleted the mrego:remove-interact-flag branch Apr 10, 2017

Manishearth pushed a commit to Manishearth/gecko-dev that referenced this pull request Apr 11, 2017

Bug 1354006 - [selectors4] Remove "interact" flag from :focus-within …
…tests. r=xidorn

This patch has been extracted from:
web-platform-tests/wpt#5370

The problem is that the "interact" flag on these tests
causes that they are considered manual,
when they can be run automatically.

This patch removes the "interact" flag and uses "reftest-wait" class
to be sure that the test has been completed.

JerryShih pushed a commit to JerryShih/gecko-dev that referenced this pull request Apr 12, 2017

Bug 1354006 - [selectors4] Remove "interact" flag from :focus-within …
…tests. r=xidorn

This patch has been extracted from:
web-platform-tests/wpt#5370

The problem is that the "interact" flag on these tests
causes that they are considered manual,
when they can be run automatically.

This patch removes the "interact" flag and uses "reftest-wait" class
to be sure that the test has been completed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment