Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve IndexError in sent_tokenize #2922

merged 2 commits into from Dec 21, 2021


Copy link

@tomaarsen tomaarsen commented Dec 21, 2021

Closes #2921


Pull request overview

  • Prevent IndexError when the sent_tokenize input starts with a dot.
  • Wrote tests to prevent this from failing again.


It is possible that split is empty whenever . is matched, which can happen whenever the input of sent_tokenize starts with a ., in the following snippet:


Lines 1380 to 1382 in dd1494e

split = text[: match.start()].rsplit(maxsplit=1)
before_start = len(split[0]) if len(split) == 2 else 0
before_words[match] = split[-1]

This PR ought to resolve this issue.

  • Tom Aarsen

@tomaarsen tomaarsen merged commit d4d99b4 into nltk:develop Dec 21, 2021
15 of 16 checks passed
@tomaarsen tomaarsen deleted the bugfix/punkt-patch branch Dec 21, 2021
ExplorerFreda added a commit to ExplorerFreda/nltk that referenced this issue Dec 24, 2021

* 'develop' of
  Temporarily pause Python 3.10 CI tests due to scikit-learn issues with Windows
  Resolve IndexError in `sent_tokenize` (nltk#2922)
  Drop support for Python 3.6, support Python 3.10 (nltk#2920)
  updates for 3.6.6
  minor clean ups
  updates for 3.6.6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

1 participant