Skip to content

Conversation

@bact
Copy link
Member

@bact bact commented Apr 20, 2019

  • Reduce duplicated code.
  • dict_word_tokenize() is a word_tokenize() with a custom_dict argument.
  • dict_word_tokenize() is currently kept (inside is just a call to word_tokenize()) but should be deprecated
  • word_tokenize() argument whitespaces is now keep_whitespace to make is more explicit, default behavior is to keep whitespaces
  • fix word_tokenize() call in FrequencySummarize to properly pass tokenizer engine value

@coveralls
Copy link

coveralls commented Apr 20, 2019

Coverage Status

Coverage increased (+0.2%) to 81.382% when pulling 7671f0e on bact:dev into 94dd259 on PyThaiNLP:dev.

@bact bact merged commit 7e9b82e into PyThaiNLP:dev Apr 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants