Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve IndexError in sent_tokenize #2922

Merged
merged 2 commits into from Dec 21, 2021
Merged

Conversation

tomaarsen
Copy link
Member

@tomaarsen tomaarsen commented Dec 21, 2021

Closes #2921

Hello!

Pull request overview

  • Prevent IndexError when the sent_tokenize input starts with a dot.
  • Wrote tests to prevent this from failing again.

Details

It is possible that split is empty whenever . is matched, which can happen whenever the input of sent_tokenize starts with a ., in the following snippet:

nltk/nltk/tokenize/punkt.py

Lines 1380 to 1382 in dd1494e

split = text[: match.start()].rsplit(maxsplit=1)
before_start = len(split[0]) if len(split) == 2 else 0
before_words[match] = split[-1]

This PR ought to resolve this issue.

  • Tom Aarsen

@tomaarsen tomaarsen merged commit d4d99b4 into nltk:develop Dec 21, 2021
15 of 16 checks passed
@tomaarsen tomaarsen deleted the bugfix/punkt-patch branch Dec 21, 2021
ExplorerFreda added a commit to ExplorerFreda/nltk that referenced this issue Dec 24, 2021
…develop

* 'develop' of https://github.com/ExplorerFreda/nltk:
  Temporarily pause Python 3.10 CI tests due to scikit-learn issues with Windows
  Resolve IndexError in `sent_tokenize` (nltk#2922)
  Drop support for Python 3.6, support Python 3.10 (nltk#2920)
  updates for 3.6.6
  minor clean ups
  updates for 3.6.6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant