Add jQuery test suite #2281

Open
scottgonzalez opened this Issue Oct 26, 2015 · 7 comments

Comments

Projects
None yet
5 participants
@scottgonzalez
Contributor

scottgonzalez commented Oct 26, 2015

At least some browser vendors already run some version of the jQuery test suite as part of their tests to ensure that new releases of the browsers don't break existing releases of jQuery. In a recent discussion with some Mozilla developers, we had the idea of adding the jQuery test suite to web platform tests since browser vendors already import and run these tests.

This would provide a good sanity check for web compatibility, and updating the test suite in one location would be ideal. I think we'd just need to write a small wrapper to pipe the QUnit results to the existing test harness.

Thoughts? Concerns?

@gsnedders

This comment has been minimized.

Show comment
Hide comment
@gsnedders

gsnedders Oct 27, 2015

Contributor

What's the advantage of having them in wpt v. having the vendors grab them individually?

The big advantage is we can help everyone keep up-to-date wrt versions of jQuery.

The disadvantages are numerous:

  • jQuery is large compared with almost ever other test in the repo, and is comparatively hard to reduce failures.
  • jQuery allows non-conforming behaviours (normally behind bug detection, etc.).
  • We don't want to end up with ancient versions of jQuery, but at the same time we don't want to be updating them endlessly (because it'll cause problems by invalidating expectations, potentially).
  • How many JS libraries do we end up adding? Some we can't bundle because of licensing. Others we can.

Anyone else have any opinions? I'm not entirely opposed but I'm not exactly keen.

Contributor

gsnedders commented Oct 27, 2015

What's the advantage of having them in wpt v. having the vendors grab them individually?

The big advantage is we can help everyone keep up-to-date wrt versions of jQuery.

The disadvantages are numerous:

  • jQuery is large compared with almost ever other test in the repo, and is comparatively hard to reduce failures.
  • jQuery allows non-conforming behaviours (normally behind bug detection, etc.).
  • We don't want to end up with ancient versions of jQuery, but at the same time we don't want to be updating them endlessly (because it'll cause problems by invalidating expectations, potentially).
  • How many JS libraries do we end up adding? Some we can't bundle because of licensing. Others we can.

Anyone else have any opinions? I'm not entirely opposed but I'm not exactly keen.

@jgraham

This comment has been minimized.

Show comment
Hide comment
@jgraham

jgraham Nov 10, 2015

Contributor

I feel the same way as @gsnedders fwiw. I see the appeal of running these tests, but it feels like they're a bad fit specifically because they are trying to paper over the differences between browsers rather than expose them.

Contributor

jgraham commented Nov 10, 2015

I feel the same way as @gsnedders fwiw. I see the appeal of running these tests, but it feels like they're a bad fit specifically because they are trying to paper over the differences between browsers rather than expose them.

@gsnedders

This comment has been minimized.

Show comment
Hide comment
@gsnedders

gsnedders Nov 10, 2015

Contributor

I think if we can get agreement across all browsers as to policies as to what JS library test suites people want to run (inc. how to deal with new library releases, etc.), then perhaps it's worth hacking at them in some shared repo to use testharness.js such that testharnessreporter.js can be used for all of them. I'd lean against putting them in w-p-t though.

Contributor

gsnedders commented Nov 10, 2015

I think if we can get agreement across all browsers as to policies as to what JS library test suites people want to run (inc. how to deal with new library releases, etc.), then perhaps it's worth hacking at them in some shared repo to use testharness.js such that testharnessreporter.js can be used for all of them. I'd lean against putting them in w-p-t though.

@leobalter

This comment has been minimized.

Show comment
Hide comment
@leobalter

leobalter Nov 11, 2015

Contributor

jQuery is large compared with almost ever other test in the repo, and is comparatively hard to reduce failures.

the tests for jQuery are not lot large compared to the official tests for ES5/6 (test262), not even close if you regard the time it takes.

ES6 tests fail on most browsers for many features, anything else showing they break is not a problem, but an opportunity to fix their implementation.

jQuery allows non-conforming behaviours (normally behind bug detection, etc.).

It does not mean it supports only non-conforming behaviors, otherwise that would be a serious bug on jQuery. This statement might only bring FUD.

The jQuery tests would be useful to find any unintended retro-compatibility issue on new browser versions.


@scottgonzalez this is interesting as it reminds how it would be great to have headless version of browser apis, as we already have with the JS runtimes, as V8, Chakra and SpiderMonkey.

Having tests for jQuery wouldn't hurt: it's not hard to update it to the last released versions and this project already test features beyond the web api, as JS built-in methods, see https://github.com/w3c/web-platform-tests/tree/master/js/builtins

Contributor

leobalter commented Nov 11, 2015

jQuery is large compared with almost ever other test in the repo, and is comparatively hard to reduce failures.

the tests for jQuery are not lot large compared to the official tests for ES5/6 (test262), not even close if you regard the time it takes.

ES6 tests fail on most browsers for many features, anything else showing they break is not a problem, but an opportunity to fix their implementation.

jQuery allows non-conforming behaviours (normally behind bug detection, etc.).

It does not mean it supports only non-conforming behaviors, otherwise that would be a serious bug on jQuery. This statement might only bring FUD.

The jQuery tests would be useful to find any unintended retro-compatibility issue on new browser versions.


@scottgonzalez this is interesting as it reminds how it would be great to have headless version of browser apis, as we already have with the JS runtimes, as V8, Chakra and SpiderMonkey.

Having tests for jQuery wouldn't hurt: it's not hard to update it to the last released versions and this project already test features beyond the web api, as JS built-in methods, see https://github.com/w3c/web-platform-tests/tree/master/js/builtins

@scottgonzalez

This comment has been minimized.

Show comment
Hide comment
@scottgonzalez

scottgonzalez Nov 11, 2015

Contributor

jQuery is large compared with almost ever other test in the repo, and is comparatively hard to reduce failures.

jQuery's API surface is certainly larger than most individual W3C specs, but the test suite isn't really that large.

As for being hard to reduce failures, are you referring to reducing a single method in jQuery not working as expected or web pages that do complex things using jQuery which uncover bugs? I would expect the overwhelming majority of test failures in the jQuery test suite to be easy to reduce.

jQuery allows non-conforming behaviours (normally behind bug detection, etc.).

The intention is to check for web compatibility, so it isn't really a concern if the test suite passes while the browser has a spec compliance violation. The spec test suite should catch that.

We don't want to end up with ancient versions of jQuery, but at the same time we don't want to be updating them endlessly (because it'll cause problems by invalidating expectations, potentially).

This is actually where I think this gets tricky: determining what version(s) to test. Probably the thing most people would think of first is to always test the latest version. That's certainly a good idea to ensure that browsers aren't regressing against the current stable version. But it might be a good idea to also run the tests for the most popular versions in use.

How many JS libraries do we end up adding? Some we can't bundle because of licensing. Others we can.

I think this would be based on real world usage statistics. With jQuery being the most popular library, I think starting with just one version of jQuery and seeing how the process goes would be the first step. Then we can determine criteria for adding additional libraries and possibly additional versions.

I see the appeal of running these tests, but it feels like they're a bad fit specifically because they are trying to paper over the differences between browsers rather than expose them.

Only a fraction of jQuery is about papering over differences. A large portion is just about providing nicer APIs. But again, this is about testing for web compatibility, not for conformance to a specification.

Contributor

scottgonzalez commented Nov 11, 2015

jQuery is large compared with almost ever other test in the repo, and is comparatively hard to reduce failures.

jQuery's API surface is certainly larger than most individual W3C specs, but the test suite isn't really that large.

As for being hard to reduce failures, are you referring to reducing a single method in jQuery not working as expected or web pages that do complex things using jQuery which uncover bugs? I would expect the overwhelming majority of test failures in the jQuery test suite to be easy to reduce.

jQuery allows non-conforming behaviours (normally behind bug detection, etc.).

The intention is to check for web compatibility, so it isn't really a concern if the test suite passes while the browser has a spec compliance violation. The spec test suite should catch that.

We don't want to end up with ancient versions of jQuery, but at the same time we don't want to be updating them endlessly (because it'll cause problems by invalidating expectations, potentially).

This is actually where I think this gets tricky: determining what version(s) to test. Probably the thing most people would think of first is to always test the latest version. That's certainly a good idea to ensure that browsers aren't regressing against the current stable version. But it might be a good idea to also run the tests for the most popular versions in use.

How many JS libraries do we end up adding? Some we can't bundle because of licensing. Others we can.

I think this would be based on real world usage statistics. With jQuery being the most popular library, I think starting with just one version of jQuery and seeing how the process goes would be the first step. Then we can determine criteria for adding additional libraries and possibly additional versions.

I see the appeal of running these tests, but it feels like they're a bad fit specifically because they are trying to paper over the differences between browsers rather than expose them.

Only a fraction of jQuery is about papering over differences. A large portion is just about providing nicer APIs. But again, this is about testing for web compatibility, not for conformance to a specification.

@scottgonzalez

This comment has been minimized.

Show comment
Hide comment
@scottgonzalez

scottgonzalez Nov 11, 2015

Contributor

I think if we can get agreement across all browsers as to policies as to what JS library test suites people want to run (inc. how to deal with new library releases, etc.), then perhaps it's worth hacking at them in some shared repo to use testharness.js such that testharnessreporter.js can be used for all of them. I'd lean against putting them in w-p-t though.

That would be fine too. I think everything I outlined above would still apply the same regardless of which approach we take. If it's easier to accomplish this via a different repository, then let's do that :-)

Contributor

scottgonzalez commented Nov 11, 2015

I think if we can get agreement across all browsers as to policies as to what JS library test suites people want to run (inc. how to deal with new library releases, etc.), then perhaps it's worth hacking at them in some shared repo to use testharness.js such that testharnessreporter.js can be used for all of them. I'd lean against putting them in w-p-t though.

That would be fine too. I think everything I outlined above would still apply the same regardless of which approach we take. If it's easier to accomplish this via a different repository, then let's do that :-)

@foolip

This comment has been minimized.

Show comment
Hide comment
@foolip

foolip Mar 28, 2018

Contributor

@gsnedders, it's been quite a while since this was opened. You added the acid tests and this would be a bit similar in that it doesn't target any single spec.

I'm inclined to say that we should just not do this, unless multiple browser vendors say that they'd like the shared test coverage, or something.

Contributor

foolip commented Mar 28, 2018

@gsnedders, it's been quite a while since this was opened. You added the acid tests and this would be a bit similar in that it doesn't target any single spec.

I'm inclined to say that we should just not do this, unless multiple browser vendors say that they'd like the shared test coverage, or something.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment