Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a Tokenizer API. #163

Closed
wants to merge 1 commit into from
Closed

Add a Tokenizer API. #163

wants to merge 1 commit into from

Conversation

espadrine
Copy link
Contributor

This adds a tokenize(code, options) function to the exported object.

It uses the same API as when options.tokens is set.

It also uses Tim Disney's algorithm documented here:
https://github.com/mozilla/sweet.js/wiki/design

http://code.google.com/p/esprima/issues/detail?id=398

This adds a `tokenize(code, options)` function to the exported object.

It uses the same API as when `options.tokens` is set.

It also uses Tim Disney's algorithm documented here:
https://github.com/mozilla/sweet.js/wiki/design

http://code.google.com/p/esprima/issues/detail?id=398
@ariya
Copy link
Contributor

ariya commented Mar 9, 2013

Landed. Thank you!

@ariya ariya closed this Mar 9, 2013
@espadrine
Copy link
Contributor Author

Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants