New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Web Platform Test: upstreaming test suites from vendors #1388

Open
hoch opened this Issue Oct 2, 2017 · 19 comments

Comments

Projects
6 participants
@hoch
Member

hoch commented Oct 2, 2017

I realized we don't have an open issue for this one. I leave this note as a future reference.

Upstream target: https://github.com/w3c/web-platform-tests/tree/master/webaudio

@hoch hoch changed the title from Web Platform Tests: upstreaming test suites from vendors to Web Platform Test: upstreaming test suites from vendors Oct 2, 2017

annevk added a commit to whatwg/html that referenced this issue Oct 3, 2017

@rtoy

This comment has been minimized.

Show comment
Hide comment
@rtoy

rtoy Oct 5, 2017

Contributor

Initial start: Create a Chrome/Firefox/Edge, etc., directories for the current vendor-specific tests.

Then move the tests around as needed.

Contributor

rtoy commented Oct 5, 2017

Initial start: Create a Chrome/Firefox/Edge, etc., directories for the current vendor-specific tests.

Then move the tests around as needed.

@mdjp mdjp added this to the Web Audio V1 milestone Oct 5, 2017

@mdjp mdjp added the V1 Blocker label Oct 5, 2017

@svgeesus

This comment has been minimized.

Show comment
Hide comment
@svgeesus

svgeesus Oct 22, 2017

Contributor

@padenot you mentioned 4 AnalyserNode tests would be upstreamed soon. I guess these will show up in https://github.com/w3c/web-platform-tests/tree/master/webaudio/the-audio-api/the-analysernode-interface ?

Contributor

svgeesus commented Oct 22, 2017

@padenot you mentioned 4 AnalyserNode tests would be upstreamed soon. I guess these will show up in https://github.com/w3c/web-platform-tests/tree/master/webaudio/the-audio-api/the-analysernode-interface ?

@hoch

This comment has been minimized.

Show comment
Hide comment
@hoch

hoch Oct 22, 2017

Member

I thought it would rather be something like:
https://github.com/w3c/web-platform-tests/tree/master/webaudio/firefox/analyser

Isn't this the consensus for WPT? Collecting the vendor-specific tests in different locations and then merge them later?

Member

hoch commented Oct 22, 2017

I thought it would rather be something like:
https://github.com/w3c/web-platform-tests/tree/master/webaudio/firefox/analyser

Isn't this the consensus for WPT? Collecting the vendor-specific tests in different locations and then merge them later?

@padenot

This comment has been minimized.

Show comment
Hide comment
@padenot

padenot Oct 23, 2017

Member

I thought it would rather be something like:
https://github.com/w3c/web-platform-tests/tree/master/webaudio/firefox/analyser

Isn't this the consensus for WPT? Collecting the vendor-specific tests in different locations and then merge them later?

Talking with @jgraham, he told me the first form (without vendor directories) is preferred.

Member

padenot commented Oct 23, 2017

I thought it would rather be something like:
https://github.com/w3c/web-platform-tests/tree/master/webaudio/firefox/analyser

Isn't this the consensus for WPT? Collecting the vendor-specific tests in different locations and then merge them later?

Talking with @jgraham, he told me the first form (without vendor directories) is preferred.

@jgraham

This comment has been minimized.

Show comment
Hide comment
@jgraham

jgraham Oct 23, 2017

Yes, we have tried quite hard to remove vendor directories. Maybe part of the confusion is that we have also tried to make it possible for vendors to upstream wpt tests from their own repositories, so in that sense they are collected in different locations and then merged, but in all cases it makes more sense to put the tests into their expected final location, rather than adding confusion from vendor directories.

jgraham commented Oct 23, 2017

Yes, we have tried quite hard to remove vendor directories. Maybe part of the confusion is that we have also tried to make it possible for vendors to upstream wpt tests from their own repositories, so in that sense they are collected in different locations and then merged, but in all cases it makes more sense to put the tests into their expected final location, rather than adding confusion from vendor directories.

@rtoy

This comment has been minimized.

Show comment
Hide comment
@rtoy

rtoy Oct 23, 2017

Contributor

I find this a bit unfortunate. We do fully expect to move all of these "vendor" directories into the final locations. But as a way to keep track of tests that are now being added, this makes life much easier to see what is new and how it breaks. (We fully expect lots of firefox tests to fail on chrome and vice versa).

In addition, chrome's tests have a wrapper around testharness (to provide much better information on failures). If this goes in, we get into a state where many tests are written in one style but many are written in another. That seems rather unfortunate to keep it this way.

I also expect lots of duplication of tests (chrome has that problem). How do we handle filename conflicts? The first one to land wins? That also seems unfortunate.

Contributor

rtoy commented Oct 23, 2017

I find this a bit unfortunate. We do fully expect to move all of these "vendor" directories into the final locations. But as a way to keep track of tests that are now being added, this makes life much easier to see what is new and how it breaks. (We fully expect lots of firefox tests to fail on chrome and vice versa).

In addition, chrome's tests have a wrapper around testharness (to provide much better information on failures). If this goes in, we get into a state where many tests are written in one style but many are written in another. That seems rather unfortunate to keep it this way.

I also expect lots of duplication of tests (chrome has that problem). How do we handle filename conflicts? The first one to land wins? That also seems unfortunate.

@svgeesus

This comment has been minimized.

Show comment
Hide comment
@svgeesus

svgeesus Oct 25, 2017

Contributor

So, are the tests going into vendor-specific directories or one WG one?

Contributor

svgeesus commented Oct 25, 2017

So, are the tests going into vendor-specific directories or one WG one?

@rtoy

This comment has been minimized.

Show comment
Hide comment
@rtoy

rtoy Oct 25, 2017

Contributor

I certainly prefer vendor-specific directories for now, as an intermediate step. You can file an issue and assignment to me to move all of the vendor tests into the final resting place, if you like.

Contributor

rtoy commented Oct 25, 2017

I certainly prefer vendor-specific directories for now, as an intermediate step. You can file an issue and assignment to me to move all of the vendor tests into the final resting place, if you like.

@svgeesus

This comment has been minimized.

Show comment
Hide comment
@svgeesus

svgeesus Oct 25, 2017

Contributor

I was mainly trying to avoid the situation where lack of clarity on where to put the tests, meant we didn't have tests for a while.
I agree that vendor-specific directories will make triaging for overlap easier.

Contributor

svgeesus commented Oct 25, 2017

I was mainly trying to avoid the situation where lack of clarity on where to put the tests, meant we didn't have tests for a while.
I agree that vendor-specific directories will make triaging for overlap easier.

@jgraham

This comment has been minimized.

Show comment
Hide comment
@jgraham

jgraham Oct 25, 2017

Again, this is something we have tried (and mostly succeeded) in eliminating from the repository, because it generally doesn't work. Whilst the tests are in vendor directories they don't help interop because everyone is happy to assume that tests from other vendors aren't supposed to pass in their implementation for whatever reason. Then the actual act of moving tests around can cause problems (because the path is the basis of the test id and so things like vendor metadata and wpt.fyi results can get lost after a move).

If you want to edit the tests before they are run by vendors then that can happen on a branch. Otherwise, just land the tests with any renaming necessary to produce unique test names. If you want to know which tests came from each vendor for triage purposes the git history will tell you. If you want to rewrite some tests in a different style it's better to do it in-place so that if the project is never completed we still end up with the tests in a sensible place, rather than with unexpectedly long-lived vendor directories that result in interop issues being ignored.

jgraham commented Oct 25, 2017

Again, this is something we have tried (and mostly succeeded) in eliminating from the repository, because it generally doesn't work. Whilst the tests are in vendor directories they don't help interop because everyone is happy to assume that tests from other vendors aren't supposed to pass in their implementation for whatever reason. Then the actual act of moving tests around can cause problems (because the path is the basis of the test id and so things like vendor metadata and wpt.fyi results can get lost after a move).

If you want to edit the tests before they are run by vendors then that can happen on a branch. Otherwise, just land the tests with any renaming necessary to produce unique test names. If you want to know which tests came from each vendor for triage purposes the git history will tell you. If you want to rewrite some tests in a different style it's better to do it in-place so that if the project is never completed we still end up with the tests in a sensible place, rather than with unexpectedly long-lived vendor directories that result in interop issues being ignored.

@rtoy

This comment has been minimized.

Show comment
Hide comment
@rtoy

rtoy Oct 25, 2017

Contributor

I understand your point and I agree that vendor directories are bad.

Sadly, when I look at, say, https://wpt.fyi/webaudio/the-audio-api/the-audioparam-interface/retrospective-exponentialRampToValueAtTime.html, two of four browsers are marked as passing. The two browsers that fail the test fail because they timed out. The two browsers that pass the test pass because of an error: ConstantSourceNode not implemented or AudioContext not implemented (prefixed version). (That seems wrong.)

I definitely do want to fix this, either be fixing the test itself or by removing it. And piling on tons of new tests in this directory isn't going to help me fix things. And having to look through git logs just makes my life that much more difficult.

It's also pretty clear that no one is paying any attention to these tests, including me. :-( Probably because everyone has his own set of tests with probably better coverage than the existing tests.

I do want to pay attention to these tests. I just don't want to have to deal with everything all at once, everywhere.

Contributor

rtoy commented Oct 25, 2017

I understand your point and I agree that vendor directories are bad.

Sadly, when I look at, say, https://wpt.fyi/webaudio/the-audio-api/the-audioparam-interface/retrospective-exponentialRampToValueAtTime.html, two of four browsers are marked as passing. The two browsers that fail the test fail because they timed out. The two browsers that pass the test pass because of an error: ConstantSourceNode not implemented or AudioContext not implemented (prefixed version). (That seems wrong.)

I definitely do want to fix this, either be fixing the test itself or by removing it. And piling on tons of new tests in this directory isn't going to help me fix things. And having to look through git logs just makes my life that much more difficult.

It's also pretty clear that no one is paying any attention to these tests, including me. :-( Probably because everyone has his own set of tests with probably better coverage than the existing tests.

I do want to pay attention to these tests. I just don't want to have to deal with everything all at once, everywhere.

@rtoy

This comment has been minimized.

Show comment
Hide comment
@rtoy

rtoy Oct 25, 2017

Contributor

Oops. I think I misinterpreted the results for the two browsers that pass. These two both pass the first test but fail the second with errors. That seems ok. Why the other two timeout needs investigation.

Contributor

rtoy commented Oct 25, 2017

Oops. I think I misinterpreted the results for the two browsers that pass. These two both pass the first test but fail the second with errors. That seems ok. Why the other two timeout needs investigation.

@padenot

This comment has been minimized.

Show comment
Hide comment
@padenot

padenot Nov 20, 2017

Member

Some gecko tests are now at 0.

Member

padenot commented Nov 20, 2017

Some gecko tests are now at 0.

@rtoy rtoy added this to To Do in V1 Jan 23, 2018

@rtoy rtoy removed the V1 Blocker label Feb 1, 2018

@rtoy

This comment has been minimized.

Show comment
Hide comment
@rtoy

rtoy Feb 1, 2018

Contributor

Not a blocker. Leave open until every one has "finished" upstreaming their tests.

Contributor

rtoy commented Feb 1, 2018

Not a blocker. Leave open until every one has "finished" upstreaming their tests.

@rtoy

This comment has been minimized.

Show comment
Hide comment
@rtoy

rtoy Mar 2, 2018

Contributor

From yesterday's teleconf, Chrome has upstreamed their tests for all nodes except those from A to C.

Contributor

rtoy commented Mar 2, 2018

From yesterday's teleconf, Chrome has upstreamed their tests for all nodes except those from A to C.

@mdjp

This comment has been minimized.

Show comment
Hide comment
@mdjp

mdjp Mar 26, 2018

Member

Current test coverage can be viewed here
https://wpt.fyi/webaudio/the-audio-api

Member

mdjp commented Mar 26, 2018

Current test coverage can be viewed here
https://wpt.fyi/webaudio/the-audio-api

@rtoy

This comment has been minimized.

Show comment
Hide comment
@rtoy

rtoy Jun 28, 2018

Contributor

Chrome has pretty much upstreamed all tests that can be easily moved to WPT. Chrome still has a lot of tests, but most have chrome-specific thresholds or internals.

For all intents, chrome is done with this task.

Contributor

rtoy commented Jun 28, 2018

Chrome has pretty much upstreamed all tests that can be easily moved to WPT. Chrome still has a lot of tests, but most have chrome-specific thresholds or internals.

For all intents, chrome is done with this task.

@mdjp

This comment has been minimized.

Show comment
Hide comment
@mdjp

mdjp Jun 28, 2018

Member

@padenot please could you give us an update of the progress from your side?

Member

mdjp commented Jun 28, 2018

@padenot please could you give us an update of the progress from your side?

@padenot

This comment has been minimized.

Show comment
Hide comment
@padenot

padenot Jul 12, 2018

Member

We've recently started to look closely at Firefox's results on the current set of wpt, and we have some tests that can complement those. We still have quite some work to be compliant though.

Member

padenot commented Jul 12, 2018

We've recently started to look closely at Firefox's results on the current set of wpt, and we have some tests that can complement those. We still have quite some work to be compliant though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment