PyThaiNLP · wannaphong · Sep 24, 2022 · Sep 4, 2022 · Sep 7, 2022 · Sep 7, 2022
diff --git a/.github/workflows/windows-test.yml b/.github/workflows/windows-test.yml
@@ -0,0 +1,59 @@
+name: Windows Unit test and code coverage
+
+on:
+  push:
+    paths-ignore:
+      - '**.md'
+      - 'docs/**'
+  pull_request:
+    branches:
+      - dev
+    paths-ignore:
+      - '**.md'
+      - 'docs/**'
+
+jobs:
+  build:
+
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [windows-latest]
+        python-version: [3.8]
+
+    steps:
+    - uses: actions/checkout@v2
+    - uses: conda-incubator/setup-miniconda@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+        auto-activate-base: true
+        auto-update-conda: true
+    - shell: powershell
+      run: |
+        conda info
+        conda list
+    - name: Install PyTorch
+      shell: powershell
+      run: |
+        pip install torch==1.8.1
+    - name: Install dependencies
+      shell: powershell
+      run: |
+        python -m pip install --disable-pip-version-check --user --upgrade pip setuptools
+        python -m pip --version
+        python -m pip install pytest coverage coveralls
+        conda install -y -c conda-forge fairseq
+        python -m pip install https://www.dropbox.com/s/o6p2sj5z50iim1e/PyICU-2.3.1-cp38-cp38-win_amd64.whl?dl=1
+        python -m pip install -r docker_requirements.txt
+        python -m pip install .[full]
+        python -m nltk.downloader omw-1.4
+        python -m pip install spacy deepcut
+    - name: Test
+      shell: powershell
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        COVERALLS_SERVICE_NAME: github
+      run: |
+        coverage run -m unittest discover
+        coveralls
diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-FROM python:3.7-slim-buster
+FROM python:3.8-slim-buster
 
 COPY . .
 

diff --git a/README.md b/README.md
@@ -20,12 +20,12 @@ PyThaiNLP เป็นไลบารีภาษาไพทอนสำหร
 
 **News**
 
-> Now, You can contact or ask any questions you encounter with the PyThaiNLP team. <a href="https://matrix.to/#/#thainlp:matrix.org" rel="noopener" target="_blank"><img src="https://matrix.to/img/matrix-badge.svg" alt="Chat on Matrix"></a>
+> Now, You can contact or ask any questions with the PyThaiNLP team. <a href="https://matrix.to/#/#thainlp:matrix.org" rel="noopener" target="_blank"><img src="https://matrix.to/img/matrix-badge.svg" alt="Chat on Matrix"></a>
 
 | Version | Description | Status |
 |:------:|:--:|:------:|
-| [3.0](https://github.com/PyThaiNLP/pythainlp/releases) | Stable | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/545) |
-| [`dev`](https://github.com/PyThaiNLP/pythainlp/tree/dev) | Release Candidate for 3.1  | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/643) |
+| [3.1](https://github.com/PyThaiNLP/pythainlp/releases) | Stable | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/643) |
+| [`dev`](https://github.com/PyThaiNLP/pythainlp/tree/dev) | Release Candidate for 3.2  | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/714) |
 
 
 ## Getting Started

diff --git a/README_TH.md b/README_TH.md
@@ -16,12 +16,12 @@ PyThaiNLP เป็นไลบารีภาษาไพทอนสำหร
 
 **ข่าวสาร**
 
->นับตั้งแต่ PyThaiNLP 3.0 พวกเราจะยุติการสนับสนุน Python 3.6 แล้ว หากคุณจำเป็นต้องใช้ PyThaiNLP บน Python 3.6 คุณสามารถใช้ PyThaiNLP 2.3.1 ได้
+> คุณสามารถพูดคุยหรือแชทกับทีม PyThaiNLP หรือผู้สนับสนุนคนอื่น ๆ ได้ที่ <a href="https://matrix.to/#/#thainlp:matrix.org" rel="noopener" target="_blank"><img src="https://matrix.to/img/matrix-badge.svg" alt="Chat on Matrix"></a>
 
 | รุ่น | คำอธิบาย | สถานะ |
 |:------:|:--:|:------:|
-| [3.0](https://github.com/PyThaiNLP/pythainlp/releases) | Stable | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/545 |
-| [`dev`](https://github.com/PyThaiNLP/pythainlp/tree/dev) | Release Candidate for 3.1  | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/643) |
+| [3.1](https://github.com/PyThaiNLP/pythainlp/releases) | Stable | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/643) |
+| [`dev`](https://github.com/PyThaiNLP/pythainlp/tree/dev) | Release Candidate for 3.2  | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/714) |
 
 ติดตามพวกเราบน [PyThaiNLP Facebook page](https://www.facebook.com/pythainlp/) เพื่อรับข่าวสารเพิ่มเติม
 

diff --git a/SECURITY.md b/SECURITY.md
@@ -4,6 +4,7 @@
 
 | Version | Supported          |
 | ------- | ------------------ |
+| 3.1.x   | :white_check_mark: |
 | 3.0.x   | :white_check_mark: |
 | 2.3.x   | :white_check_mark: |
 | 2.2.x   | :x:                |

diff --git a/docker_requirements.txt b/docker_requirements.txt
@@ -12,7 +12,7 @@ sentencepiece==0.1.91
 ssg==0.0.8
 torch==1.8.1
 fastai==1.0.61
-transformers==4.8.2
+transformers==4.22.1
 phunspell==0.1.6
 spylls==0.1.5
 symspellpy==6.7.6
@@ -31,3 +31,6 @@ thai-nner==0.3
 spacy==2.3.*
 wunsen==0.0.3
 khanaa==0.0.6
+spacy_thai==0.7.1
+esupar==1.3.8
+ufal.chu-liu-edmonds==1.0.2
diff --git a/docs/api/parse.rst b/docs/api/parse.rst
@@ -0,0 +1,10 @@
+.. currentmodule:: pythainlp.parse
+
+pythainlp.parse
+===============
+The :class:`pythainlp.parse` is dependency parsing for Thai.
+
+Modules
+-------
+
+.. autofunction:: dependency_parsing
diff --git a/docs/api/soundex.rst b/docs/api/soundex.rst
@@ -11,6 +11,7 @@ Modules
 .. autofunction:: lk82
 .. autofunction:: udom83
 .. autofunction:: metasound
+.. autofunction:: prayut_and_somchaip
 
 References
 ----------
@@ -23,3 +24,5 @@ References
         Master Thesis. Chulalongkorn University, Thailand.
 
 .. [#lk82] วิชิต หล่อจีระชุณห์กุล และ เจริญ คุวินทร์พันธุ์. `โปรแกรมการสืบค้นคำไทยตามเสียงอ่าน (Thai Soundex) <http://guru.sanook.com/1520/>`_.
+
+.. [#prayut_and_somchaip] Prayut Suwanvisat, Somchai Prasitjutrakul. Thai-English Cross-Language Transliterated Word Retrieval using Soundex Technique. In 1998 [cited 2022 Sep 8]. Available from: https://www.cp.eng.chula.ac.th/~somchai/spj/papers/ThaiText/ncsec98-clir.pdf
diff --git a/docs/api/tokenize.rst b/docs/api/tokenize.rst
@@ -12,6 +12,7 @@ Modules
 .. autofunction:: sent_tokenize
 .. autofunction:: subword_tokenize
 .. autofunction:: word_tokenize
+.. autofunction:: word_detokenize
 .. autoclass:: Tokenizer
    :members:
 

diff --git a/docs/notes/installation.rst b/docs/notes/installation.rst
@@ -31,7 +31,11 @@ where ``extras`` can be
   - ``tltk`` (to support tltk)
   - ``textaugment`` (to support text augmentation)
   - ``oskut`` (to support OSKUT)
-  - ``nlpo3`` (to support nlpo3 enging)
+  - ``nlpo3`` (to support nlpo3 engine)
+  - ``spacy_thai`` (to support spacy_thai engine)
+  - ``esupar`` (to support esupar engine)
+  - ``transformers_ud`` (to support transformers_ud engine)
+  - ``dependency_parsing`` (to support dependency parsing with all engine)
   - ``full`` (install everything)
 
 For dependency details, look at `extras` variable in `setup.py <https://github.com/PyThaiNLP/pythainlp/blob/dev/setup.py>`_.

diff --git a/notebooks/word_detokenize.ipynb b/notebooks/word_detokenize.ipynb
@@ -0,0 +1,147 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pythainlp.tokenize import word_detokenize"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'ผมเลี้ยง 5 ตัว'"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "word_detokenize([\"ผม\",\"เลี้ยง\",\"5\",\"ตัว\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[['ผม', 'เลี้ยง', ' ', '5', ' ', 'ตัว']]"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "word_detokenize([\"ผม\",\"เลี้ยง\",\" \",\"5\",\"ตัว\"],\"list\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'ผมเลี้ยง 5 5 ตัว'"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "word_detokenize([\"ผม\",\"เลี้ยง\",\"5\",\"5\",\"ตัว\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'ภาษาไทยหรือภาษาไทยกลางเป็นภาษาในกลุ่มภาษาไทซึ่งเป็นกลุ่มย่อยของตระกูลภาษาขร้า - ไท และเป็นภาษาราชการและภาษาประจำชาติของประเทศไทย'"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "word_detokenize(['ภาษาไทย', 'หรือ', 'ภาษาไทย', 'กลาง', 'เป็น', 'ภาษา', 'ใน', 'กลุ่ม', 'ภาษา', 'ไท', 'ซึ่ง', 'เป็น', 'กลุ่มย่อย', 'ของ', 'ตระกูล', 'ภาษา', 'ข', 'ร้า', '-', 'ไท', 'และ', 'เป็น', 'ภาษาราชการ', 'และ', 'ภาษาประจำชาติ', 'ของ', 'ประเทศ', 'ไทย'])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'ผมเลี้ยง 5 5 ตัว ๆ คนดี'"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "word_detokenize([\"ผม\",\"เลี้ยง\",\"5\",\"5\",\"ตัว\",\"ๆ\",\"คน\",\"ดี\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.9.12 ('base')",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.12"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "48b90c76b600d2ec6cf3e350b23a5df9176e3eef7b22ad90377f14c1de9c1bf6"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/pythainlp/__init__.py b/pythainlp/__init__.py
@@ -4,7 +4,7 @@
 # Copyright (C) 2016-2022 PyThaiNLP Project
 # URL: <https://pythainlp.github.io/>
 # For license information, see LICENSE
-__version__ = "3.1.0-dev1"
+__version__ = "3.1.0"
 
 thai_consonants = "กขฃคฅฆงจฉชซฌญฎฏฐฑฒณดตถทธนบปผฝพฟภมยรลวศษสหฬอฮ"  # 44 chars
 

diff --git a/pythainlp/corpus/core.py b/pythainlp/corpus/core.py
@@ -47,7 +47,7 @@ def get_corpus_db_detail(name: str, version: str = None) -> dict:
     :return: details about a corpus
     :rtype: dict
     """
-    with open(corpus_db_path(), "r", encoding="utf-8") as f:
+    with open(corpus_db_path(), "r", encoding="utf-8-sig") as f:
         local_db = json.load(f)
 
     if version is None:
@@ -148,11 +148,13 @@ def get_corpus_default_db(name: str, version: str = None) -> Union[str, None]:
             )
 
 
-def get_corpus_path(name: str,  version: str = None) -> Union[str, None]:
+def get_corpus_path(name: str,  version: str = None, force: bool = False) -> Union[str, None]:
     """
     Get corpus path.
 
     :param str name: corpus name
+    :param str version: version
+    :param bool force: force download
     :return: path to the corpus or **None** of the corpus doesn't \
              exist in the device
     :rtype: str
@@ -202,7 +204,7 @@ def get_corpus_path(name: str,  version: str = None) -> Union[str, None]:
     corpus_db_detail = get_corpus_db_detail(name, version=version)
 
     if not corpus_db_detail or not corpus_db_detail.get("filename"):
-        download(name,  version=version)
+        download(name,  version=version, force=force)
         corpus_db_detail = get_corpus_db_detail(name, version=version)
 
     if corpus_db_detail and corpus_db_detail.get("filename"):
@@ -213,7 +215,7 @@ def get_corpus_path(name: str,  version: str = None) -> Union[str, None]:
             path = get_full_data_path(corpus_db_detail.get("filename"))
         # check if the corpus file actually exists, download if not
         if not os.path.exists(path):
-            download(name)
+            download(name,  version=version, force=force)
         if os.path.exists(path):
             return path
 
@@ -378,7 +380,7 @@ def download(
 
     # check if corpus is available
     if name in corpus_db:
-        with open(corpus_db_path(), "r", encoding="utf-8") as f:
+        with open(corpus_db_path(), "r", encoding="utf-8-sig") as f:
             local_db = json.load(f)
 
         corpus = corpus_db[name]
@@ -509,7 +511,7 @@ def remove(name: str) -> bool:
     if _CHECK_MODE == "1":
         print("PyThaiNLP is read-only mode. It can't remove corpus.")
         return False
-    with open(corpus_db_path(), "r", encoding="utf-8") as f:
+    with open(corpus_db_path(), "r", encoding="utf-8-sig") as f:
         db = json.load(f)
     data = [
         corpus for corpus in db["_default"].values() if corpus["name"] == name

diff --git a/pythainlp/corpus/oscar.py b/pythainlp/corpus/oscar.py
@@ -25,7 +25,7 @@ def word_freqs() -> List[Tuple[str, int]]:
     """
     word_freqs = []
     _path = get_corpus_path(_FILENAME)
-    with open(_path, "r", encoding="utf-8") as f:
+    with open(_path, "r", encoding="utf-8-sig") as f:
         _data = [i for i in f.readlines()]
         del _data[0]
         for line in _data:

diff --git a/pythainlp/corpus/pos_lst20_perceptron-v0.2.3.json b/pythainlp/corpus/pos_lst20_perceptron-v0.2.3.json