Prerequisites

[bterlson]

Test262 has many tests that test the interaction between two features. For example, proxies are generally interesting values and so any API should be tested with proxy values. This has the unfortunate effect of failing those API tests when proxies aren't available, which doesn't make much sense.

One way to solve this problem is specifying a prerequisite.

Two ways I can think of handling this:

1. specify a function (somehow) that must return true before a test is run

2. specify prerequisites in the front-matter

   a. map feature names to existence checks for said feature
   b. map feature names to the collateral for that feature, and only run the test if the tests have a pass rate higher than a certain threshold.

1 is hard to support because we need some way to pass a function from the test to the harness, which isn't something we have today and seems difficult (especially when running tests out-of-proc where you can't just pass the harness a reference to the prereq function).

2.a shouldn't be too hard to support, but since we want to support mapping feature names to test collateral for that feature (#107), 2.b seems somewhat interesting to pursue.

Thoughts?

[domenic]

This seems overcomplicated. I would just split out tests that use proxies into a separate file, e.g. Array.prototype.contains-proxies-this-arg or something.

[bterlson]

Sure, but then that test fails and it's up to the person running the test to know that it's an expected failure in a host without proxies. Imagine testing a polyfill in an ES5 implementation - it would be nice to have the proxy integration tests automatically filtered out.

I agree, though, that this isn't super critical, but it would be nice to have IMO :)

[smikes]

Having an es6 feature tag on the test that says "Proxy" would be enough to tell us to skip that test if the host doesn't have proxies. 

Is that enough to address both the prerequisite and the interaction issue? 

Annotate a test with the es6 features and abstract operations it touches. That's enough to get us an idea of coverage/lack of and to rule out tests that hit uncovered operations. 

es6: [Array#find, IsCallable, arrowfunctions]

On Mon, Oct 27, 2014 at 4:56 PM, Brian Terlson notifications@github.com
wrote:

Sure, but then that test fails and it's up to the person running the test to know that it's an expected failure in a host without proxies. Imagine testing a polyfill in an ES5 implementation - it would be nice to have the proxy integration tests automatically filtered out.

I agree, though, that this isn't super critical, but it would be nice to have IMO :)

Reply to this email directly or view it on GitHub:
#111 (comment)

[bterlson]

Is that enough to address both the prerequisite and the interaction issue?

It's not enough I don't think. You still need some way of determining if the feature is present, aka a mapping of "Proxy" to return typeof Proxy === 'function'.

[smikes]

It's not enough I don't think. You still need some way of determining if the feature is present, aka a mapping of "Proxy" to return typeof Proxy === 'function'.

Oh, I thought we were going to do that manually. So if I'm testing v8 today I would have some sort of features.json file with Proxy: false.

Even so, whether it's manual or automatic, tagging a test as features: [Proxy] is a good start in being able to filter out these cross-es6-feature dependencies

[leobalter]

We ended up using the features tag, which is not really consumed, but at this point I think it's too late to fix unless we have a new plan. Closing it for now.