Cannot Import PunktWordTokenizer in nltk 3.3

[ghost]

How do I use PunktWordTokenizer in nltk 3.3? Has it been deprecated or renamed?
>>> nltk.__version__ '3.3'
>>> from nltk.tokenize import PunktWordTokenizer Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: cannot import name 'PunktWordTokenizer'
Any help/suggestion is highly appreciated.

[alvations]

Punkt is a sentence tokenizer algorithm not word, for word tokenization, you can use functions in nltk.tokenize. Most commonly, people use the NLTK version of the Treebank word tokenizer with

>>> from nltk import word_tokenize
>>> word_tokenize("This is a sentence, where foo bar is present.")
['This', 'is', 'a', 'sentence', ',', 'where', 'foo', 'bar', 'is', 'present', '.']

Also, please do take a look at http://www.nltk.org/book/ch03.html

[ghost]

Yeah I usually use word_tokenize() . I might be wrong but wasn't PunktWordTokenizer present in the previous versions? Like in the nltk versions around 2014-15?

[alvations]

Yes, the PunktWordTokenizer was exposed previously but it wasn't a real word tokenizer, it's more like a pre-processor before Punkt decides on where to split the sentence. So now it's no longer exposed to users to avoid confusion.

If you're interested in improving Punkt, do take a look at #2008

[iamRVel]

Nope, It's available in 3.3 version but use the following code from nltk.tokenize.punkt import PunktSentenceTokenizer

[alvations]

Yes, it does seems like the PunktSentenceTokenizer has been re-exposured again https://github.com/nltk/nltk/blob/develop/nltk/tokenize/punkt.py#L1236

Closing this issue as resolved then =)

Please do reopen the issue if it's still relevant/unresolved.