New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test262 conformance as an line item the compat table #830

Open
littledan opened this Issue Jul 7, 2016 · 8 comments

Comments

Projects
None yet
6 participants
@littledan

littledan commented Jul 7, 2016

Test262 is another test suite for JavaScript. It provides a number of very detailed set of around 20k tests and aims to test every line in the specification, going into some details which the current compat table omits. However, there is not currently a place where test262 scores are surfaced, and test262 does not include categorization into features, weighting, or a pretty website.

For me as an implementer, test262 has been extremely useful in identifying non-compliant parts of V8 so that we can fix them. Although many of the bugs don't reflect something that seems likely for a user to run into, it feels to me like a good tool for making a JavaScript implementation into a reliable, consistent surface.

Maybe it would be useful for others as well to see in a comparison context. I'm wondering whether it might make sense to have one item in the compat table here be for the entire test262 score. It could be factored into the overall score, or left to the side for independent reference. Thoughts?

@ljharb

This comment has been minimized.

Show comment
Hide comment
@ljharb

ljharb Jul 7, 2016

Collaborator

I love this idea. This table is mostly checking presence and mostly not checking compliance, and compliance matters far more money than presence.

Collaborator

ljharb commented Jul 7, 2016

I love this idea. This table is mostly checking presence and mostly not checking compliance, and compliance matters far more money than presence.

@kangax

This comment has been minimized.

Show comment
Hide comment
@kangax

kangax Jul 8, 2016

Owner

This is a great idea! The fact that test262 compliance is not listed anywhere on a web for any browser seems like an unfortunate omission. We've pushed the table far enough, to the point where some engines score 100%. It's only fair to expose even more detailed metrics. I imagine a separate table would be most convenient for implementors? Something with breakdown by test group, to see what exactly is failing?

Owner

kangax commented Jul 8, 2016

This is a great idea! The fact that test262 compliance is not listed anywhere on a web for any browser seems like an unfortunate omission. We've pushed the table far enough, to the point where some engines score 100%. It's only fair to expose even more detailed metrics. I imagine a separate table would be most convenient for implementors? Something with breakdown by test group, to see what exactly is failing?

@ljharb

This comment has been minimized.

Show comment
Hide comment
@ljharb

ljharb Jul 8, 2016

Collaborator

If there is a way to automatically convert commandline test262 results to a web-consumable format, that would be ideal imo, because then we could programmatically show a breakdown with collapsible details etc.

Collaborator

ljharb commented Jul 8, 2016

If there is a way to automatically convert commandline test262 results to a web-consumable format, that would be ideal imo, because then we could programmatically show a breakdown with collapsible details etc.

@natorion

This comment has been minimized.

Show comment
Hide comment
@natorion

natorion Jul 11, 2016

Contributor

There is a web test runner for test262. It produces html and json as a result. You can have a look at it at http://v8.github.io/test262/website/default.html. The runner unfortunately is quite buggy (did not work in Safari the last time I checked).

Contributor

natorion commented Jul 11, 2016

There is a web test runner for test262. It produces html and json as a result. You can have a look at it at http://v8.github.io/test262/website/default.html. The runner unfortunately is quite buggy (did not work in Safari the last time I checked).

@littledan

This comment has been minimized.

Show comment
Hide comment
@littledan

littledan Jul 12, 2016

Yes, there's significant technical work needed to make this work, which @jugglinmike, @natorion and I have looked into to some extent. If anyone is interested in implementation along these lines, I could help you get started; I won't have the time to do this technical work myself at the moment.

littledan commented Jul 12, 2016

Yes, there's significant technical work needed to make this work, which @jugglinmike, @natorion and I have looked into to some extent. If anyone is interested in implementation along these lines, I could help you get started; I won't have the time to do this technical work myself at the moment.

@bterlson

This comment has been minimized.

Show comment
Hide comment
@bterlson

bterlson Jul 13, 2016

Contributor

I support this idea, although I still hope someone will contribute a better website to ECMA like the old test262.ecmascript.org website. I think the hardest problem with this is going to be the UI - there is a lot of data in test262 and presenting it in a useful way is hard.

FWIW, I have just released a new version of https://github.com/bterlson/test262-harness which supports running test262 in browsers from the command line.

Contributor

bterlson commented Jul 13, 2016

I support this idea, although I still hope someone will contribute a better website to ECMA like the old test262.ecmascript.org website. I think the hardest problem with this is going to be the UI - there is a lot of data in test262 and presenting it in a useful way is hard.

FWIW, I have just released a new version of https://github.com/bterlson/test262-harness which supports running test262 in browsers from the command line.

@chicoxyzzy

This comment has been minimized.

Show comment
Hide comment
@chicoxyzzy
Collaborator

chicoxyzzy commented Nov 29, 2016

This also might be useful
https://github.com/bakkot/test262-web-runner/

@chicoxyzzy chicoxyzzy referenced this issue Dec 6, 2016

Merged

WIP Post #1014

6 of 6 tasks complete
@chicoxyzzy

This comment has been minimized.

Show comment
Hide comment
@chicoxyzzy
Collaborator

chicoxyzzy commented Dec 30, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment