Update 2.1 branch from dev #274

bact · 2019-09-21T06:40:59Z

No description provided.

Merge from 2.0.5 release

issue_71: add documentation

1. Move all functions related to text processing to the new file `ulmfit/rules.py` 2. For the rules imported from `fastai` library, we copied the code to the pythainlp library 3. Use the code of BaseTokenizer class from `fastai` librarry

Function ungroup_emoji has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.

changes in this commit: - assert the output of the method `get_ner` when argument `tag` is set to True for all 13 tags as described in (https://github.com/wannaphongcom/thai-ner/tree/master/model/1.2) - assert the output of the method `get_ner` when argument `pos` is set to True - asser the output of the method `get_ner` when argument `pos` is set to False

As thainer version 1.2 provide different results from previous version

- Add new function `process_thai` as the function of `ulmfit` module. This function process Thai texts (such as replace repetitive character/words). More detail will be explained in the release note.

@bact

according to @bact (#252 (comment))

Change in this commit: - add new test case: test_remove_space - add new test case: test_replace_wrep_post_nonum - add new test case: test_replace_wrep_post - add new test case: test_replace_rep_nonum - add new test case: test_replace_rep_after - remove a test case: test_replace_all_caps (as the function `replace_all_caps` was removed from /pythianlp/ulmfit/__init__.py)

Fix tokenization benchmark issue

…eploy docs) Update command_line.rst

Add test cases for NER

pep8speaks · 2019-09-21T06:41:05Z

Hello @bact! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file docs/conf.py:

Line 31:80: E501 line too long (194 > 79 characters)
Line 32:80: E501 line too long (182 > 79 characters)
Line 33:80: E501 line too long (174 > 79 characters)
Line 41:80: E501 line too long (83 > 79 characters)

In the file notebooks/visualize.py:

Line 13:1: E302 expected 2 blank lines, found 1
Line 13:80: E501 line too long (97 > 79 characters)
Line 14:72: E225 missing whitespace around operator
Line 14:80: E501 line too long (86 > 79 characters)
Line 16:80: E501 line too long (87 > 79 characters)
Line 17:80: E501 line too long (96 > 79 characters)
Line 23:80: E501 line too long (85 > 79 characters)
Line 29:80: E501 line too long (85 > 79 characters)
Line 30:28: E231 missing whitespace after ','
Line 34:1: E302 expected 2 blank lines, found 1
Line 34:80: E501 line too long (96 > 79 characters)
Line 34:97: W291 trailing whitespace
Line 35:48: E225 missing whitespace around operator
Line 37:80: E501 line too long (87 > 79 characters)
Line 38:80: E501 line too long (112 > 79 characters)
Line 44:80: E501 line too long (85 > 79 characters)
Line 45:80: E501 line too long (106 > 79 characters)
Line 50:23: E225 missing whitespace around operator
Line 51:31: E231 missing whitespace after ','
Line 51:40: E231 missing whitespace after ','
Line 51:50: E231 missing whitespace after ','
Line 51:58: E231 missing whitespace after ','
Line 51:80: E501 line too long (81 > 79 characters)
Line 53:29: E231 missing whitespace after ','
Line 53:39: E231 missing whitespace after ','
Line 53:47: E231 missing whitespace after ','
Line 53:55: E231 missing whitespace after ','
Line 57:1: E302 expected 2 blank lines, found 1
Line 57:80: E501 line too long (110 > 79 characters)
Line 57:90: E231 missing whitespace after ','
Line 57:95: E252 missing whitespace around parameter equals
Line 57:96: E252 missing whitespace around parameter equals
Line 57:98: E231 missing whitespace after ','
Line 57:103: E225 missing whitespace around operator
Line 59:80: E501 line too long (87 > 79 characters)
Line 69:26: E225 missing whitespace around operator
Line 69:43: E227 missing whitespace around bitwise or shift operator
Line 69:53: E225 missing whitespace around operator
Line 78:65: E231 missing whitespace after ','
Line 84:80: E501 line too long (86 > 79 characters)

In the file pythainlp/tag/named_entity.py:

Line 97:1: W293 blank line contains whitespace

In the file pythainlp/tokenize/__init__.py:

Line 40:80: E501 line too long (80 > 79 characters)
Line 66:80: E501 line too long (83 > 79 characters)
Line 435:1: W293 blank line contains whitespace
Line 435:1: W391 blank line at end of file

In the file pythainlp/tokenize/attacut.py:

Line 14:1: W293 blank line contains whitespace

In the file pythainlp/tokenize/deepcut.py:

Line 27:1: W391 blank line at end of file

In the file pythainlp/ulmfit/__init__.py:

Line 30:9: E225 missing whitespace around operator
Line 40:1: E302 expected 2 blank lines, found 1
Line 54:8: W291 trailing whitespace
Line 55:80: E501 line too long (2015 > 79 characters)
Line 386:80: E501 line too long (83 > 79 characters)

In the file tests/test_tokenize.py:

Line 182:5: E303 too many blank lines (2)

coveralls · 2019-09-21T06:47:35Z

Coverage increased (+2.9%) to 90.069% when pulling b9025aa on dev into d61bfa3 on 2.1.

bact and others added 30 commits May 10, 2019 07:45

Merge pull request #221 from PyThaiNLP/dev

3e19fd3

Merge from 2.0.5 release

Merge pull request #232 from PyThaiNLP/issue71_add_documentation

0a57ae9

issue_71: add documentation

add test cases for text processing rules and BaseTokenizer

9784fe0

add a blank line

a180a88

format docstring

e53c018

remove fastai from the library dependencies

064f791

refactor code due to conitive complexity

eb29b90

Function ungroup_emoji has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.

fix bug, ungroup emoji

9135559

NER : Add output like html tag

d44b03f

add attacut to pythainlp/tokrnize

1302a85

fix format

9d4fab3

fix PEP8 issues

def7388

fix typo

723a052

change input texts in two cases

937aecb

As thainer version 1.2 provide different results from previous version

Merge changes related to ulmfit module from dev branch

fd20e0b

- Add new function `process_thai` as the function of `ulmfit` module. This function process Thai texts (such as replace repetitive character/words). More detail will be explained in the release note.

refactor the unittest

888aa78

Add change in appveyor.yml (to fix build error)

074a391

according to @bact (#252 (comment))

add a test case for pythainlp.ulmfit.process_text

d6ccb2c

edit test case description

c20ec56

update test cases description

3e90cc5

use high level api in attacut

3292ba3

merge

3e0d4d6

Merge branch 'dev' into add-attacut

73ba1ed

fixed merge conflict in and

7adc2ea

fixed merge conflict

70b0a3b

fixed file format

9822e10

Update requirements.txt

44a9f6b

bact and others added 27 commits September 8, 2019 05:47

Update test_benchmarks.py

d7115f8

remove unused imports

7b7f92c

change open mode for sentences.yml to "rb"

fcc9d28

Merge branch 'dev' into fix-tokenization-benchmark-issue

21ea11e

Update test_benchmarks.py

ceb167f

sort imports

fd5817d

write with utf-8 encoding

08c365d

fix naming consistency (tokenization instead of tokenisation)

3e630c4

Update word-tokenization-benchmark

a853b75

Update benchmarks.rst

b4ee5d6

Delete build.sh

cfb529a

Delete bld.bat

729b886

Delete buildall.sh

c49f4f6

Merge pull request #269 from PyThaiNLP/fix-tokenization-benchmark-issue

21871bc

Fix tokenization benchmark issue

Update attacut.py

23ba97e

Update command_line.rst

206a115

Merge pull request #271 from PyThaiNLP/Command-Line-Docs (build and d…

49fe5b0

…eploy docs) Update command_line.rst

add replace_url

323bca1

fixed replace_url

b9b7792

add process_thai to wongnai

93348ae

add word2vec

2dfee52

add visualize.py to notebooks

4353e65

Update .travis.yml

d0bc6c2

PyThaiNLP 2.1.dev4

1d26919

Merge branch 'dev' into ner-tag

eb09bc9

replace_url should be in pre_rules

90bb505

Merge pull request #273 from PyThaiNLP/ner-tag

b9025aa

Add test cases for NER

bact merged commit 984f3f7 into 2.1 Sep 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update 2.1 branch from dev #274

Update 2.1 branch from dev #274

Uh oh!

bact commented Sep 21, 2019

Uh oh!

pep8speaks commented Sep 21, 2019

Uh oh!

coveralls commented Sep 21, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Update 2.1 branch from dev #274

Update 2.1 branch from dev #274

Uh oh!

Conversation

bact commented Sep 21, 2019

Uh oh!

pep8speaks commented Sep 21, 2019

Uh oh!

coveralls commented Sep 21, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants