Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add SBD tests from Pragmatic Segmenter #24

Merged
merged 55 commits into from
Jun 8, 2019
Merged

Add SBD tests from Pragmatic Segmenter #24

merged 55 commits into from
Jun 8, 2019

Conversation

nipunsadvilkar
Copy link
Owner

  • Update cleaner and abbreviation functionalities
  • All tests passing
    • with clean=True
    • without cleaning
    • for doctype PDF

@nipunsadvilkar
[WIP] abbreviation replacer
a650313
@nipunsadvilkar
BugFix: Amend false variable return
6e840e8 
Expected return -> self.text
@nipunsadvilkar
[WIP] multiple period abbreviation replacer
7ec3e4a 
18 test for english passing
@nipunsadvilkar
21 tests for english passing
b73c95a
@nipunsadvilkar
[BugFix] "\\1." instead of r"\\1." in replace_abbreviation_as_sentenc…
b7735cb 
…e_boundary

25 english test passing
@nipunsadvilkar
[BugFix] Fixed post_process_segments & Add WithMultiplePeriodsAndEmai…
daf0355 
…lRule

26 english test passing
@nipunsadvilkar
33 tests for english passing
fcb8c89
@nipunsadvilkar
45 tests for english passing
015fbe0
@nipunsadvilkar
[BugFix] Quotation text list flattening after post_process_segments
373609c 
46 tests for english passing
@nipunsadvilkar
Add GeoLocationRule rule
3ed8fb2
@nipunsadvilkar
Add FileFormatRule rule
1271de5
@nipunsadvilkar
Init clean rule: Add Newline rule
62f4807
@nipunsadvilkar
Setup segmenter class" -m "Check desired output with clean=True
beb3382
@nipunsadvilkar
Switch from Processor to Segmenter class
6fc865f
@nipunsadvilkar
Add new clean=True test cases
5761593
@nipunsadvilkar
Add cleaner funcs: [WIP] search_for_connected_sentences func
f623535
@nipunsadvilkar
Complete Cleaner funcs: Fixed search_for_connected_sentences func
91673df 
50 english test passing
@nipunsadvilkar
Add doc_type "pdf" test
f032d5c 
51 english test passing
@nipunsadvilkar
55 tests for english passing: Normal, PDF, CLEAN
95864df
@nipunsadvilkar
Add pytest skip for holy grail sentence
5f25991
@nipunsadvilkar
Add abbreviation replacer, clean rules python files
80d1a67
@nipunsadvilkar
[WIP] bugfix numbered list
28d80e6
@nipunsadvilkar
Fixed numbered list. 56 english test passing
7e028f9 
Example starting with
@nipunsadvilkar
63 english test passing
54e09e0
@nipunsadvilkar
68 english test passing
990310d
@nipunsadvilkar
74 english test passing
6658104
@nipunsadvilkar
80 english test passing
0c9a0e8
@nipunsadvilkar
95 english test passing
fbecd7e
@nipunsadvilkar
100 english test passing
b30cc72
@nipunsadvilkar
Seperate test for golden rules and clean param rules
20ee563 
106 english test passing
@nipunsadvilkar
[Major BugFix] re.sub with args ✗ -> re.IGNORECASE | *kwarg* ✔ -> fla…
3807001 
…gs=re.IGNORECASE

https://stackoverflow.com/questions/42581/python-re-sub-with-a-flag-does-not-replace-all-occurrences
@nipunsadvilkar
116 english test passing
7d341aa
@nipunsadvilkar
124 english test passing
ecc079b
@nipunsadvilkar
130 english test passing
e93234b
@nipunsadvilkar
140 english tests passing
6f41b5f
@nipunsadvilkar
150 english tests passing
498332a
@nipunsadvilkar
153 english tests passing
2f254f3
@nipunsadvilkar
154 english tests passing
d521798 
Seperate clean=False test cases
@nipunsadvilkar
[BugFix] substitute_found_list_items on matched items only
3215827 
154 english tests passing
@nipunsadvilkar
154 english tests passing
dfce6a4 
Yet 16 tests failing and need to fix
@nipunsadvilkar
Add I.V keyword in replace_abbreviation_as_sentence_boundary
5f7db13 
159 english tests passing
@nipunsadvilkar
Typo fix in SubSymbolsRules
8f282fa 
160 tests passing
@nipunsadvilkar
Regex escape chars in cleaner for ?
83df013 
163 tests passing
@nipunsadvilkar
Fix regex of QUOTATION_AT_END_OF_SENTENCE_REGEX
2eae0b0 
164 english test passing
@nipunsadvilkar
166 english tests passing
f81e9f4
@nipunsadvilkar
BugFix replacing ! -> &ᓴ& and ? -> &ᓷ&
8bb7130 
167 english tests passing
@nipunsadvilkar
Fix regex of SingleUpperCaseLetterRule
d4fe911 
168 english tests passing
@nipunsadvilkar
Fix NUMBERED_REFERENCE_REGEX
93aedec 
169 english tests passing
@nipunsadvilkar
173 english tests passing
dff14b2
@nipunsadvilkar
174 english tests passing
37ade3b
@nipunsadvilkar
179 english tests passing
05bfca9
@nipunsadvilkar
Added single big text english
eeec0ae 
[Stable] 180 english tests passing
@nipunsadvilkar
Update docstrings for tests
a9f7a6f
@nipunsadvilkar nipunsadvilkar self-assigned this Jun 8, 2019
@nipunsadvilkar nipunsadvilkar merged commit c238a75 into master Jun 8, 2019
@nipunsadvilkar
Copy link
Owner Author

Closes #4, #5, #6, #7, #8, #9, #10, #11, #12, #13, #14, #15

@nipunsadvilkar nipunsadvilkar mentioned this pull request Jun 8, 2019
Closed
@nipunsadvilkar nipunsadvilkar deleted the npn-clean-text branch October 18, 2019 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@nipunsadvilkar nipunsadvilkar

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@nipunsadvilkar