Add support for Chinese and Japanese stop words #507
Conversation
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
nemo_curator/download/commoncrawl.py
Outdated
| stop_lists: A dictionary stop lists, where the keys are languages (e.g., "ENGLISH") | ||
| and the values are Python frozensets denoting the list of stop words for that language. | ||
| If None, it defaults to jusText's stop lists: https://github.com/miso-belica/jusText/tree/main/justext/stoplists, | ||
| with added Thai, Chinese, and Japanese support. |
There was a problem hiding this comment.
While I think it's important for NeMo Curator to support Thai, Chinese, and Japanese, I also think it would be a good idea for us to allow users to pass in their own stop lists as a workaround.
This way, if a language is not already supported, the user can do it themselves. Additionally, a user might not like the stop lists provided by jusText and want to pass in their own custom stop lists for that reason, too.
nemo_curator/download/commoncrawl.py
Outdated
| paragraphs = handler.paragraphs | ||
|
|
||
| # Context free classification | ||
| # TODO: Check Thai, Chinese, Japanese, and Korean |
There was a problem hiding this comment.
Words in Thai, Chinese, Japanese, and Korean are not separated by spaces. I need to make sure to either (1) raise a warning about the stop word logic used by our jusText/Resiliparse extractors (such as suggesting to modify the stopwords_low/stopwords_high/required_stopword_density parameters?) or (2) adding word splitting logic like I did for #320.
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
nemo_curator/download/commoncrawl.py
Outdated
| from .thai_stopwords import thai_stopwords | ||
|
|
||
| stop_list_dict["THAI"] = thai_stopwords | ||
| if lang_key in ["THAI", "CHINESE", "JAPANESE"]: |
There was a problem hiding this comment.
Nit: Make this conditional if lang_key in custom_stopwords
nemo_curator/download/commoncrawl.py
Outdated
|
|
||
| return [p.text for p in paragraphs if not p.is_boilerplate] | ||
| if self.is_boilerplate is None: | ||
| if language in ["THAI", "CHINESE", "JAPANESE", "KOREAN"]: |
There was a problem hiding this comment.
can you abstract away this list of 4 languages to a global var that is shared between all of the extractors?
There was a problem hiding this comment.
Updated, thanks!
Signed-off-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
docs/user-guide/download.rst
Outdated
| 1. Decode the HTML within the record from binary to text. | ||
| 2. If the HTML can be properly decoded, then with `pyCLD2 <https://github.com/aboSamoor/pycld2>`_, perform language detection on the input HTML. | ||
| 3. Finally, the extract the relevant text with `jusText <https://github.com/miso-belica/jusText>`_ or `Resiliparse <https://github.com/chatnoir-eu/chatnoir-resiliparse>`_ from the HTML and write it out as a single string within the 'text' field of a json entry within a `.jsonl` file. | ||
| 1. Decode the HTML within the record from binary to text. |
There was a problem hiding this comment.
I just took a peak at the rendered version of this page and it looks a little messed up. Can you fix it?
There was a problem hiding this comment.
Updated, let me know what you think.
Signed-off-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
| if language in NON_SPACED_LANGUAGES: | ||
| warnings.warn( | ||
| "stopword_density is ignored for non-space-separated languages." | ||
| ) | ||
| result = paragraphs |
There was a problem hiding this comment.
Since #431 was just merged, I added this logic to the Trafilatura extractor. It is the same as the Resiliparse logic.
* add zh and ja stopwords Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> * run isort Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> * edit doc Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> * indent? Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> * rst file Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> * rst? Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> * more indents? Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> * fix todos and add pytests Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> * run black Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> * add Ryan's suggestions Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> * run isort Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> * edit rst file Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> * add trafilatura support Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> --------- Signed-off-by: Sarah Yurick <sarahyurick@gmail.com> Signed-off-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
2 participants
Closes #459.
Related PRs:
get_word_splitter#320