WebIDL tests

[heycam]

Here are the rest of the tests I have for Web IDL v1.

[jgraham]

How long do you expect making followups to take? At the very least the files need to be in the right place in the tree. And you don't want to loose the review comments. Is there anything really substantive to fix?

[plehegar]

There are additional comments in https://critic.hoppipolla.co.uk/r/247

[sideshowbarker]

This PR has become kinda stale and it's not clear at this point who it's waiting on action from. @plehegar?

[AFBarstow]

@sideshowbarker @plehegar - my understanding is that @ylafon agreed to address all of the comments for this PR although I don't know if his plan is to update the PR or to submit a new one.

Yves - what is the plan and timeframe to address this PR?

[jgraham]

@tobie: You are working on WebIDL, right? What do you want to happen with these tests?

[ylafon]

Note that there is a PR on this PR, comments on this one were addressed in https://github.com/ylafon/web-platform-tests/commits/submission/heycam/WebIDL-tests-1

[tobie]

@jgraham: wow. I wasn't aware this test suite existed.

I'm debating the value of reviewing and maintaining such a test suite compared to investing this time improving idlharness in both depth (keep the harness in sync with spec changes, adding more tests to it) and breadth (e.g. adding it to the +50% of specs in wpt who don't have idlharness tests yet, keeping the existing ones up to date, etc.).

@domenic, @annevk, @bzbarsky, @heycam: would love your thoughts on this.

[annevk]

It's not clear to me why we dropped the ball here. These look like good tests. Someone should run them in browsers to see if they're reasonably accurate and then just land them (using the branch from @ylafon as that seems to contain all the fixes).

[tobie]

@annevk:

These look like good tests.

I'm not debating this. However, given constrained resources (which is clearly demonstrated by the fact no one had enough cycles to follow through on merging those), I'm wondering:

1. who will maintain those tests going forward, and
2. whether they offer something idlharness tests don't.

[annevk]

@Ms2ger might be able to judge whether they offer something in addition to idlharness. Maintenance will generally fall out of implementations finding that tests start failing or passing.

[tobie]

@gsnedders your input would also be valuable here.

[tobie]

@annevk:

Maintenance will generally fall out of implementations finding that tests start failing or passing.

In doing that, don't we loose the ability to help implementors fix their WebIDL bindings by maintaining the tests ourselves?

[tobie]

On irc, @Ms2ger said:

tobie, my general thought is that most of those tests don't belong in a specific WebIDL test suite; either they should go in idlharness or they should be sorted according to the spec that defines the interface.

[annevk]

In doing that, don't we loose the ability to help implementors fix their WebIDL bindings by maintaining the tests ourselves?

Sure, but as you said, you can't have it all.

[bzbarsky]

My personal experience with modifying idlharness is that it's a huge pain. The code is scattered all over, there's lots of repetition, no particular rhyme or reason to where certain things are tested, etc.

Also, idlharness assertions are not that well factored into test() blocks, so it's easy to get no coverage of all sorts of stuff because you don't implement some one thing somewhere that has nothing to do with the other stuff idlharness is testing.

Fixing these problems is a pretty huge undertaking. So I actually somewhat favor less monolithic, more maintainable tests to some extent.

In principle, I'm not sure anything prevents us from rolling these tests into idlharness and ensuring that idlharness runs on interfaces that exercise the relevant functionality.

In practice, let's consider the workflow around adding a test for a new IDL feature. With small tests like this, you pick an interface that uses the feature, write a test against it. With idlharness, you pick an interface that uses the feature. You make sure idlharness is running against that interface. You make sure the idlharness parser can parse the feature (want to test record<>? have to go add it; it's not supported there yet). You make sure idlharness can auto-detect which bits use the feature. You add some tests conditioned on that auto-detection. This is a lot more pain...

And that's assuming you can do it at all. If you have some function that exercises the feature you want, and it needs some specific arguments, auto-figuring-out how it gets those arguments is quite nontrivial. Can they be constructed via constructor calls based on their IDL interface types? What if those have no constructors but we have factory methods? etc.

As a concrete example, how would you write a [Clamp] test in idlharness? You need to observe the value that gets passed through to the underlying implementation, but you need to know something about the underlying implementation to manage to do that, right?

I guess the upshot of all that rambling is that I think we should definitely get targeted tests of specific features written, outside of idlharness. idlharness is then somewhat useful for checking that the IDL in browsers matches the spec IDL, effectively, for the things idharness can test...

[Ms2ger]

All true, and certainly it's not possible to test all of WebIDL's requirements with idlharness alone. I do think that it is also valuable for things that can be tested in idlharness, to be tested there.

[bzbarsky]

Sure.

I think it's more important to have tests than anything else. So what I think we should do in this concrete instance is get these tests landed, then see about porting those of them that can be ported to idlharness.