Fixes #628 Greek word tokenizer added #629
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Smart addition. FYI the build server error'ed out on the pass (looks like a problem on Travis CI's side), so I'll merge as soon as I see that finished. |
Merged
Codecov Report
@@ Coverage Diff @@
## master #629 +/- ##
==========================================
+ Coverage 86.55% 86.57% +0.01%
==========================================
Files 138 138
Lines 8385 8394 +9
==========================================
+ Hits 7258 7267 +9
Misses 1127 1127
Continue to review full report at Codecov.
|
|
Saw that last night (problem seems to have been with Travis this time?)—looks like it’s passing on my end now. |
|
Fwiw I have seen a lot of interest in the Greek tools in recent months and hope to spend more time—starting with this PR—on building up/advising a Greek core development team. |
kylepjohnson
approved these changes
Jan 28, 2018
kylepjohnson
added a commit
that referenced
this pull request
Jan 28, 2018
API addition of Greek by Patrick Burns PR #629
|
Yes, the hangups were all on Travis's side. Everything passed fine, eventually. |
inishchith
added a commit
to inishchith/cltk
that referenced
this pull request
Jan 29, 2018
* add odia language (cltk#606) * add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Bump vers for Odia alphabet * alpha odia * Fixes cltk#628 Greek word tokenizer added (cltk#629) * Check that tokens exist before handling them in Latin word tokenizer * Update files * Reset master * Reset master * rm whitespace * Add default Greek tokenizer * Comment on word tokenizer * Cleanup order of languages/functions, alphabetical * Update docs for Greek word tokenizer * Add unittest for Greek word tokenizer * Bump vers for tokenizer API addition of Greek by Patrick Burns PR cltk#629 * Fix Typo (cltk#627) * Fix Typo mrathi -> marathi * Update stops.py
kylepjohnson
pushed a commit
that referenced
this pull request
Jan 30, 2018
* add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Fix typo UNSTRUCTURESCONSONANTS -> UNSTRUCTURED_CONSONANTS followed the usual convention of CLTK to separate with an _ ( underscore) * sync (#1) * add odia language (#606) * add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Bump vers for Odia alphabet * alpha odia * Fixes #628 Greek word tokenizer added (#629) * Check that tokens exist before handling them in Latin word tokenizer * Update files * Reset master * Reset master * rm whitespace * Add default Greek tokenizer * Comment on word tokenizer * Cleanup order of languages/functions, alphabetical * Update docs for Greek word tokenizer * Add unittest for Greek word tokenizer * Bump vers for tokenizer API addition of Greek by Patrick Burns PR #629 * Fix Typo (#627) * Fix Typo mrathi -> marathi * Update stops.py
Merged
inishchith
added a commit
to inishchith/cltk
that referenced
this pull request
Jan 31, 2018
* add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Fix typo UNSTRUCTURESCONSONANTS -> UNSTRUCTURED_CONSONANTS followed the usual convention of CLTK to separate with an _ ( underscore) * sync (#1) * add odia language (cltk#606) * add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Bump vers for Odia alphabet * alpha odia * Fixes cltk#628 Greek word tokenizer added (cltk#629) * Check that tokens exist before handling them in Latin word tokenizer * Update files * Reset master * Reset master * rm whitespace * Add default Greek tokenizer * Comment on word tokenizer * Cleanup order of languages/functions, alphabetical * Update docs for Greek word tokenizer * Add unittest for Greek word tokenizer * Bump vers for tokenizer API addition of Greek by Patrick Burns PR cltk#629 * Fix Typo (cltk#627) * Fix Typo mrathi -> marathi * Update stops.py
kylepjohnson
pushed a commit
that referenced
this pull request
Feb 8, 2018
* add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Fix typo UNSTRUCTURESCONSONANTS -> UNSTRUCTURED_CONSONANTS followed the usual convention of CLTK to separate with an _ ( underscore) * sync (#1) * add odia language (#606) * add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Bump vers for Odia alphabet * alpha odia * Fixes #628 Greek word tokenizer added (#629) * Check that tokens exist before handling them in Latin word tokenizer * Update files * Reset master * Reset master * rm whitespace * Add default Greek tokenizer * Comment on word tokenizer * Cleanup order of languages/functions, alphabetical * Update docs for Greek word tokenizer * Add unittest for Greek word tokenizer * Bump vers for tokenizer API addition of Greek by Patrick Burns PR #629 * Fix Typo (#627) * Fix Typo mrathi -> marathi * Update stops.py * add classical_hindi stops.py Signed-off-by: inishchith <inishchith@gmail.com> * add classical_hindi stops.py Signed-off-by: inishchith <inishchith@gmail.com>
kylepjohnson
pushed a commit
that referenced
this pull request
Feb 19, 2018
* add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Fix typo UNSTRUCTURESCONSONANTS -> UNSTRUCTURED_CONSONANTS followed the usual convention of CLTK to separate with an _ ( underscore) * sync (#1) * add odia language (#606) * add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Bump vers for Odia alphabet * alpha odia * Fixes #628 Greek word tokenizer added (#629) * Check that tokens exist before handling them in Latin word tokenizer * Update files * Reset master * Reset master * rm whitespace * Add default Greek tokenizer * Comment on word tokenizer * Cleanup order of languages/functions, alphabetical * Update docs for Greek word tokenizer * Add unittest for Greek word tokenizer * Bump vers for tokenizer API addition of Greek by Patrick Burns PR #629 * Fix Typo (#627) * Fix Typo mrathi -> marathi * Update stops.py * add classical_hindi stops.py Signed-off-by: inishchith <inishchith@gmail.com> * add classical_hindi stops.py Signed-off-by: inishchith <inishchith@gmail.com> * add swadesh_hi words and docs Signed-off-by: inishchith <inishchith@gmail.com> * Update hindi.rst * Update hindi.rst * Update hindi.rst
kylepjohnson
pushed a commit
that referenced
this pull request
Feb 22, 2018
* add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Fix typo UNSTRUCTURESCONSONANTS -> UNSTRUCTURED_CONSONANTS followed the usual convention of CLTK to separate with an _ ( underscore) * sync (#1) * add odia language (#606) * add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Bump vers for Odia alphabet * alpha odia * Fixes #628 Greek word tokenizer added (#629) * Check that tokens exist before handling them in Latin word tokenizer * Update files * Reset master * Reset master * rm whitespace * Add default Greek tokenizer * Comment on word tokenizer * Cleanup order of languages/functions, alphabetical * Update docs for Greek word tokenizer * Add unittest for Greek word tokenizer * Bump vers for tokenizer API addition of Greek by Patrick Burns PR #629 * Fix Typo (#627) * Fix Typo mrathi -> marathi * Update stops.py * add classical_hindi stops.py Signed-off-by: inishchith <inishchith@gmail.com> * add classical_hindi stops.py Signed-off-by: inishchith <inishchith@gmail.com> * add swadesh_hi words and docs Signed-off-by: inishchith <inishchith@gmail.com> * remove duplicates * remove duplicates #2 remove duplicates + docs formatting
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a default word tokenizer for Greek (nb: I could have sworn this already existed, but apparently not; e.g. not in the docs). Until language-specific options are added, this word tokenizer just uses the existing NLTK word tokenizer wrapper split string into list of tokens. Also adds unittest and documentation.