Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run tc39/test262 against Babel #4987

Open
xtuc opened this issue Dec 10, 2016 · 11 comments

Comments

@xtuc
Copy link
Member

commented Dec 10, 2016

Context

The idea is to run tc39/test262 tests suite against Babel.

Running the official spec tests against Babel can make sure we comply to the spec or at least know when we don’t. We’ll want to figure out how to do filtering on things we can’t compile (proxy, TCO, etc) and set up an easy way to check failing tests and file issues and PRs for them.

This issue will be used to track this.

Previous discussions:

Possible Solution

I think we should use our own test runner. We might want to skip some tests.

There are some restriction in order to run the tests. They are well explained in https://github.com/tc39/test262/blob/master/INTERPRETING.md#test-execution.

The tests could be run automaticly on the CI (they are taking some time).

/cc @kangax


Edit: 11/12

My test is available here: https://github.com/xtuc/babel-test262.

Some observations I made:

  • Running all tests takes too long (build log)
  • Mocha reporter and only the language built-ins tests (build log) and this time the ouput was too long.
  • Mocha dot reporter and only the language built-ins tests seems to work pretty well (build log)
@kangax

This comment has been minimized.

Copy link
Member

commented Dec 12, 2016

I did this couple months ago when testing Babili. Here's my script to run from within chakra-core repo — https://gist.github.com/kangax/d4e96384bdce3528d41808ae1843df14

Here's one of the issues I filed back then — #4756

@xtuc

This comment has been minimized.

Copy link
Member Author

commented Dec 17, 2016

The project now allows to run tests with given Babel transformations (for convienience I choose to use babel-preset-env).

I wanted it to be as simple and flexible as possible. See the README to see its usage.

I used an environement variable to configure the target browser.

TARGET_BROWSERS="last 200 versions"

Please tell me if this make sense to you and/or way to improve it.

Example build job: https://travis-ci.org/xtuc/babel-test262/builds/184788391.

TODO

  • @kangax, would this suffice to run your tests on Babili?
  • Define tests
    • Target browsers? Polyfill? Plugins? Presets?
    • Which test suite in test262?
  • Could it be integrated within Babel's CI?
  • There are still some bugs with the runner
@kangax

This comment has been minimized.

Copy link
Member

commented Dec 20, 2016

@xtuc looking good! For our use case, one simple thing we could try as Babili smoke test is to try running jQuery unit tests after transpiling jQuery with Babili. It's a small/simple repo with easy to run unit tests so should be a good starting point.

@gsathya

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

@littledan can our test262 test infra be used by babel or is it tightly coupled with our CI?

@hzoo

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

I know there's https://github.com/bakkot/test262-web-runner and https://github.com/bterlson/test262-harness too but haven't looked into it that much

@littledan

This comment has been minimized.

Copy link

commented Jan 18, 2017

@gsathya It would take some infrastructure work to hook everything up. I'm not sure what you'd get out of V8's infrastructure. I'd recommend starting with either the web runner or the official Node-based command-line runner.

@chicoxyzzy

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

I didn't see this issue earlier. I ❤️ the idea and will be happy to help if I can.

Also we can add babel/babel-preset-env#117 to discussions in first message

@gsathya

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

I'm not sure what you'd get out of V8's infrastructure.

Mostly the harness -- ability to skip tests + cli

@littledan

This comment has been minimized.

Copy link

commented Jan 19, 2017

@gsathya I think V8's infrastructure for this is a bit heavy for this purpose. It would probably be simpler to rebuild something different than use that.

@hzoo

This comment has been minimized.

Copy link
Member

commented Aug 10, 2017

Babylon has this now courtesy of @jugglinmike babel/babylon#654, will need to do something else for Babel transforms

@hzoo

This comment has been minimized.

Copy link
Member

commented Sep 27, 2019

Update: working via babel/babel-test262-runner and we also have #10500 as a per to test in circle. We can ask about being adding to http://test262.report too

@hzoo hzoo pinned this issue Oct 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.