python 3.7 compatibility PEP 479: ngrams

[weilu-mkt]

Hi,

Similar to #2148 I just found that using the command ngrams is also raising the RuntimeError: generator raised StopIteration. This happens when I try to loop over the ngrams generated, for example:

data = ngrams(sentence, 2) \\ for gram in data: \\ print(gram)

And the error happens at the second line.

Could you look into that issue as well? Thanks!

PS: I tried re-installing the package on Feb 1, 2019 and the problem seems to be there still

[purificant]

Hi @weilu-mkt,

What are the values of sentence and nltk.__version__?

[weilu-mkt]

Hi purificant,

Thanks for your help! I just checked and find my version of nltk is 3.3. An example of the variable sentence is as follows (a list containing words with their pos tags):
[('decis', 'N'), ('illustr', 'V'), ('scope', 'N'), ('stringenc', 'N'), ('rule', 'N'), ('refer', 'V')]

I just used pip3 install nltk==3.4 and it appears to be succesful: Successfully installed nltk-3.4 singledispatch-3.4.0.3 six-1.12.0, however my nltk.__version__ is still 3.3. And the issue still occurs

[purificant]

Please use nltk 3.4, it's the first version with Python 3.7 support, your issue should be resolved in this version.
If your nltk.__version__ still says 3.3 it means the code is still importing the older version.

[purificant]

I've attempted to reproduce this with nltk 3.3 as well, to confirm that it is indeed fixed in 3.4, however, I was not able to do it. My sample code:

$ python
Python 3.7.2 (default, Jan  6 2019, 09:19:32) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>> print(nltk.__version__)
3.3
>>> sentence = [('decis', 'N'), ('illustr', 'V'), ('scope', 'N'), ('stringenc', 'N'), ('rule', 'N'), ('refer', 'V')]
>>> data = nltk.ngrams(sentence, 2)
>>> for gram in data: print(gram)
... 
(('decis', 'N'), ('illustr', 'V'))
(('illustr', 'V'), ('scope', 'N'))
(('scope', 'N'), ('stringenc', 'N'))
(('stringenc', 'N'), ('rule', 'N'))
(('rule', 'N'), ('refer', 'V'))
>>

[weilu-mkt]

Ok, thanks a lot for your help. The problem is solved and I think it is because I didn't install 3.4 successfully.

[jalca014]

Hello all, I have still been having this exact issue while running ngrams within Google Colab today. I checked nltk.__version__ and it gave me 3.2.5, meaning Google Colab keeps importing a version of nltk from 2017 (at least, that's the copyright year given in the path /usr/local/lib/python3.7/dist-packages/nltk/util.py when I check it). Fortunately what I'm trying to do is still working, but I think resolving this would be convenient. I'm not sure how one could do that, however.

[purificant]

Try to install the latest version of nltk with

!pip install nltk

Along the lines of https://colab.research.google.com/notebooks/snippets/importing_libraries.ipynb