Add KenLM wrapper #71
Conversation
|
|
||
| .. code-block:: python | ||
|
|
||
| pip install https://github.com/kpu/kenlm/archive/master.zip |
There was a problem hiding this comment.
Is it possible to move it to "requirements.txt"?
|
|
||
| pip install https://github.com/kpu/kenlm/archive/master.zip | ||
|
|
||
| Please refer to **Scalable Modified Kneser-Ney Language Model Estimation** for |
There was a problem hiding this comment.
- Add a reference link.
- Add a dot at the end of the sentence.
| class KenLMModel(object): | ||
| """ | ||
| Wrapper for KenLM language model. | ||
| You should install KenLM python interface first. |
There was a problem hiding this comment.
You should install --> Please install.
|
|
||
| class KenLMModel(object): | ||
| """ | ||
| Wrapper for KenLM language model. |
There was a problem hiding this comment.
Is there any full name for KenLM ? Add a reference link.
| """ | ||
| Initialize variables and load model. | ||
|
|
||
| :param model_path: Path of language model |
| :param id_list: Id list of word. | ||
| :param id_str_dict: list | ||
| """ | ||
| assert len(id_list) > 0, 'invalid id list' |
There was a problem hiding this comment.
Replace it with raise ValueError.
| import kenlm | ||
| self._model = kenlm.LanguageModel(self._model_path) | ||
|
|
||
| def score_sentence_ids(self, id_list): |
There was a problem hiding this comment.
A string type is better than an id list.
And this class should be responsible for tokenization, dictionary loading and dictionary looking-up. Don't leave it to users.
| end token explicitly if the input sentence has been completed. | ||
|
|
||
| :param id_list: Id list of word. | ||
| :param id_str_dict: list |
There was a problem hiding this comment.
Add return and rtype in the comments.
| return math.pow(10, score) | ||
|
|
||
|
|
||
| if __name__ == '__main__': |
There was a problem hiding this comment.
Remove below or add this to a unit test file.
| id_str_dict, | ||
| verbose=False): | ||
| """ | ||
| Initialize variables and load model. |
There was a problem hiding this comment.
This function comments should be moved above to class comments.
resolves #2229