New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve tokenization #233

merged 6 commits into from Jan 27, 2018


None yet
1 participant

50Wliu commented Jan 24, 2018


  • Filling out the template is required. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
  • All new code requires tests to ensure against regressions

Description of the Change

  • Tokenize function names for function and method calls
  • Tokenize objects and properties
Before After
language-python-before language-python-after

Quite the difference if I do say so myself. This brings language-python in line with other languages such as language-javascript, language-coffee-script, language-php, etc.

Alternate Designs




Possible Drawbacks


Applicable Issues

Fixes #170
Fixes #141

50Wliu added some commits Jan 23, 2018

50Wliu added some commits Jan 24, 2018

@50Wliu 50Wliu merged commit 847186a into master Jan 27, 2018

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
continuous-integration/travis-ci/pr The Travis CI build passed

@50Wliu 50Wliu deleted the wl-better-tokenization branch Jan 27, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment