Do not allocate points for controversial and experimental specifications

[fbender]

E. g. for Video/DRM, there is no agreement among the "Big 4" to implement it at all(!), and strong critique by various other entities.

Same applies for WebGL 2 (there's only a highly experimental implementation in Firefox, and I think some interest – no code! – from Google) where agreement is likely, but the spec is in a very early state, and succeeding or failing the test should not change the score. The problem with early-stage specs is that APIs can change substantially during development (in that case you'd have to change the test anyway, so "future-proofing" the test is not a valid argument).

These APIs should not be awarded points if they exist because testing for them defies the purpose of HTML5Test.com. You may keep the tests, but move them to the (formerly) "bonus points" section (where failing or passing the test does not change the score). When and if an agreement on spec & impl exists at some point in the future, you can then add the tests back to their original section and count the points.

[NielsLeenheer]

There are no points awarded for DRM support. It is just informational. I haven't decided yet what to do with WebGL2.

[fbender]

OK, thanks. In that case, please move the test to the extra section to avoid confusion. The note can be reworded like:

The following tests go beyond the requirements of the HTML5 specification, or do not have sufficient agreement among browser vendors. Thus, they are not counted towards the total score.

For WebGL2, I strongly advise to not count the test, but keep it as an extra nonetheless (and also saying so).

[fbender]

Looks like the Path primitive and Dashed Line support are still under discussion, so I'd not account for them, too. The test could produce a false positive in the near future (because exact implementation/API may change).

Here are some discussions (I started looking from August):
http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-August/thread.html#40511
http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-August/thread.html#40523
http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-September/thread.html#40710
http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-September/thread.html#40717
http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-September/thread.html#40872
http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2013-October/thread.html#41005

[NielsLeenheer]

I don't mind that an API will change. We can update the test accordingly. That has always been the idea behind HTML5test. Nothing is set in stone and that the spec will change is fully expected. I will update the test to remove deprecated features and add new features.

But HTML5test should not introduce new features that have no implementations and completely lost traction or new features that have already been outright dropped from the spec.

I do plan on giving more information about each test. If you click on a test right now a popup will appear with links to documentation and specs. But I also want to include info about the maturity of the spec and how many points can be earned by passing the test.

[fbender]

Yes, but I'd avoid testing for stuff where the API might change in the near term, otherwise this will create a false sense of compliance (even if you include information about stability & what spec version you are testing for; people don't read!). At least, don't count, and add a disclaimer before them, like:

These APIs are not finalized. A succeeding test may not indicate specification compliance. Therefore, there are currently no points awarded for these tests.

Once specs have stabilized a bit more, you can move them back to the "counting area".

But I also want to include info about the maturity of the spec and how many points can be earned by passing the test.

Awesome! Do you reflect the maturity in the number of points to be awarded, i. e. do not-fully-stable features get less points than if they were stable?

Also, I was always curious on how you wheight features and actually set the number of points a test has (seems a bit arbitrary at times). Can you provide information on that (on the site)?