Dev #2
Merged
Dev #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kylepjohnson
added a commit
that referenced
this pull request
Jun 15, 2014
kylepjohnson
pushed a commit
that referenced
this pull request
Sep 6, 2015
kylepjohnson
pushed a commit
that referenced
this pull request
Aug 29, 2016
kylepjohnson
pushed a commit
that referenced
this pull request
Nov 9, 2016
kylepjohnson
pushed a commit
that referenced
this pull request
Nov 10, 2016
* Check that tokens exist before handling them in Latin word tokenizer * Add helper function 'tokenize' to sentence tokenizer * Add custom word/sentence tokenizers to Latinlibrary 'book' loader * Add alternative sentence tokenizer for latinlibrary loader if CLTK_DATA is not present * Add GSoC Lemmatize files * Minor updates to lookup files, tests for main file * Improved docs for backoff.py; more consistent naming conventions throughout * More general updates to GSoC lemmatizer code and models. * Add test module for lemmatizer * Add more tests for base lemmatizer classes * Remove test_distance_sentences temporarily * Added a readme with GSoC 2016 info * Fixed markdown link * Update readme.md * do light cleanup * Fixes #2, adds English words to exceptions list for Latin tokenizer * Remove * import for nltk tag module; load old lemma model from cltk_data as pickle. * Deleted old_model; moved to cltk_data * Renamed ModelLemmatizer to TrainLemmatizer tests * Make all model imports check cltk_data * Fixed default settings--mostly preloading regex patterns--for various lemmatizers; modified tests accordingly * Updated docstrings for backoff.py * Moved model files out of module * Fix bad print statement * Clean up Greek lemmatize module * Remove extra readme files * Comment out TrigramPOSLemmatizer * Cleanup backoff.py; add test for BigramPOS lemmatizer * Remove function for old version of lemmatizer * Updated tests * Fixed typo in test_lemmatize imports * Fixes #417: Check token against full regex before iterating in Latin backoff lemmatizer * Updated docs for Latin lemmatzier
kylepjohnson
pushed a commit
that referenced
this pull request
Feb 6, 2017
* Check that tokens exist before handling them in Latin word tokenizer * Add helper function 'tokenize' to sentence tokenizer * Add custom word/sentence tokenizers to Latinlibrary 'book' loader * Add alternative sentence tokenizer for latinlibrary loader if CLTK_DATA is not present * Add GSoC Lemmatize files * Minor updates to lookup files, tests for main file * Improved docs for backoff.py; more consistent naming conventions throughout * More general updates to GSoC lemmatizer code and models. * Add test module for lemmatizer * Add more tests for base lemmatizer classes * Remove test_distance_sentences temporarily * Added a readme with GSoC 2016 info * Fixed markdown link * Update readme.md * do light cleanup * Fixes #2, adds English words to exceptions list for Latin tokenizer * Remove * import for nltk tag module; load old lemma model from cltk_data as pickle. * Deleted old_model; moved to cltk_data * Renamed ModelLemmatizer to TrainLemmatizer tests * Make all model imports check cltk_data * Fixed default settings--mostly preloading regex patterns--for various lemmatizers; modified tests accordingly * Updated docstrings for backoff.py * Moved model files out of module * Fix bad print statement * Clean up Greek lemmatize module * Remove extra readme files * Comment out TrigramPOSLemmatizer * Cleanup backoff.py; add test for BigramPOS lemmatizer * Remove function for old version of lemmatizer * Updated tests * Fixed typo in test_lemmatize imports * Fixes #417: Check token against full regex before iterating in Latin backoff lemmatizer * Updated docs for Latin lemmatzier * Add seed parameter so that Lemmatizer runs can be replicable
kylepjohnson
pushed a commit
that referenced
this pull request
Feb 22, 2018
* add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Fix typo UNSTRUCTURESCONSONANTS -> UNSTRUCTURED_CONSONANTS followed the usual convention of CLTK to separate with an _ ( underscore) * sync (#1) * add odia language (#606) * add odia language * Fix-Typo * add odia docs * Update wiki summary to emphasize history * Bump vers for Odia alphabet * alpha odia * Fixes #628 Greek word tokenizer added (#629) * Check that tokens exist before handling them in Latin word tokenizer * Update files * Reset master * Reset master * rm whitespace * Add default Greek tokenizer * Comment on word tokenizer * Cleanup order of languages/functions, alphabetical * Update docs for Greek word tokenizer * Add unittest for Greek word tokenizer * Bump vers for tokenizer API addition of Greek by Patrick Burns PR #629 * Fix Typo (#627) * Fix Typo mrathi -> marathi * Update stops.py * add classical_hindi stops.py Signed-off-by: inishchith <inishchith@gmail.com> * add classical_hindi stops.py Signed-off-by: inishchith <inishchith@gmail.com> * add swadesh_hi words and docs Signed-off-by: inishchith <inishchith@gmail.com> * remove duplicates * remove duplicates #2 remove duplicates + docs formatting
clemsciences
pushed a commit
that referenced
this pull request
Sep 3, 2018
remove duplicates + docs formatting
kylepjohnson
pushed a commit
to kylepjohnson/cltk
that referenced
this pull request
Aug 21, 2020
fully refactor greek prosody, also reformat
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
dev branch merge for training set fix