Skip to content

code_tokenize v0.2.0

Latest
Compare
Choose a tag to compare
@cedricrupb cedricrupb released this 28 Jun 11:07
· 2 commits to main since this release
2705388

Major API redesign

code_tokenize in v0.2.0 makes now mainly use of the visitor pattern for parsing the AST

Changes

  • tokenize parses source code now by parsing the AST and traversing the AST via a visitor
  • custom tokenizing visitors can be defined per language
  • For Python, we correct the tokenization process: the indentation is now AST based computed
  • Code is extensively tested in parsing large libraries (Python and Java)
  • more languages are closer integrated