fixes tests in py3.8 #2432

srhrshr · 2019-10-19T01:12:42Z

@copper-head this is in response to your post on the nltk-dev google group to test out the repository against python version 3.8.0.

There were a few hiccups setting up the dev environment with the requirements compatible with the python 3.8 version as running pip install -r pip-req.txt didn't directly work on my machine. I ended up having to install the scipy and scikit-learn dependencies from source. You can find the full list of requirements on my virtualenv here.

In order to run the tests, I ran python nltk/test/runtests.py and fixed the import errors etc that I ran into along the way. This PR is a diff of changes I made for this.

Does this state of the logs show that tests ran successfully?

...................................................S..S.......SS....................SSSSSSSS............................................S....................SSSS...............................................nltk/nltk/test/unit/test_tokenize.py:60: DeprecationWarning: 
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
Snltk/nltk/test/unit/test_tokenize.py:86: DeprecationWarning: 
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
S................................
----------------------------------------------------------------------
Ran 660 tests in 194.011s

OK (SKIP=34)

Also, running tox -v py38 didn't directly work because the tox.ini file was not configured to work with python 3.8 (Duh!). But, I can adapt and run this too if this is how you want me to test. Let me know.

Thanks,
Harsha

iliakur · 2019-10-19T13:18:35Z

@alvations or @stevenbird know this better than me, but I would suggest starting with the tox environment and getting that to work first. If there's a mismatch between that and the local setups of those using 3.8 we should fix it separately.

Getting tox to work has the advantage that it'll also get tested by the github CI and we can officially say we support 3.8 until someone complains about their local setup (not a likely scenario).

srhrshr · 2019-10-19T16:13:32Z

The problem with setting up a tox environment is scipy isn't installable via pip and I had to build from source in my venv to get it to work. Looks like scipy is working on releasing a 3.8 compatible version: scipy/scipy#10927.

So, what do you suggest I do - is there a way to build from source while setting up a tox environment ?

iliakur · 2019-10-19T18:21:21Z

I'd go down the path of least resistance and set up an alert of some kind to notify me when scipy starts supporting 3.8 and then attempt stuff with NLTK.
The easiest would be to subscribe to that issue

This is the cost of having dependencies...

If, of course, you have time and interest, you could help adding 3.8 support to scipy, that's the more proactive approach.

srhrshr · 2019-10-19T20:12:52Z

scipy is close to a release and I've subscribed to their issue re this. Will take this up as soon as they've released the wheels.

stevenbird · 2019-11-18T10:24:38Z

[CI: retest]

stevenbird · 2019-11-20T13:25:03Z

[CI: retest]

iliakur · 2020-01-24T17:40:15Z

@srhrshr any luck getting this to work?

srhrshr · 2020-01-24T18:34:56Z

Will take this up this weekend. You can expect some chatter from me in about 12 hours.

alvations · 2020-01-28T03:06:02Z

Thank you @srhrshr for the help in testing NLTK for python 3.8. We sorely need this =)

Any idea why the python 3.8 isn't kicked in on the CI from the .travis.yml? I'll check the CI to see why that's happening too.

srhrshr · 2020-01-28T04:29:47Z

No worries @alvations - that must be because I didn't update the travis.yml - hopefully it should trigger now.

Btw, here is a detailed log of the local tox run.

tested w/ python 3.8.0:

-........................................................-...................
.........-.....-.................-........-..................................
…..-/…./nltk/nltk/tokenize/stanford_segmenter.py:305: DeprecationWarning:
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
-....................................................................-.
...-....-.....-.-..-.......-..-..............................................
.............................................................................
.............................................................................
.......................-..-.......--....................--------.............
.................................-....................----...................
………………………./…../nltk/nltk/test/unit/test_tokenize.py:60: DeprecationWarning:
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
-/…../nltk/nltk/test/unit/test_tokenize.py:86: DeprecationWarning:
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
-................................

-----------------------------------------------------------------------------
678 tests run in 165.804 seconds.
35 skipped (643 tests passed)
__________________________________________________________________________________ summary __________________________________________________________________________________
  py38: commands succeeded
  congratulations :)

——————————————————————————————————————

tested w/ python 3.7.4:

-........................................................-...................
.........-.....-.................-........-..................................
…..-….nltk/nltk/tokenize/stanford_segmenter.py:305: DeprecationWarning:
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
-....................................................................-.
...-....-.....-.-..-.......-..-..............................................
.............................................................................
.............................................................................
.......................-..-.......--....................--------.............
.................................-....................----...................
………………………./…/nltk/nltk/test/unit/test_tokenize.py:60: DeprecationWarning:
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
-…/nltk/nltk/test/unit/test_tokenize.py:86: DeprecationWarning:
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
-................................

-----------------------------------------------------------------------------
678 tests run in 174.890 seconds.
35 skipped (643 tests passed)
__________________________________________________________________________________ summary __________________________________________________________________________________
  py37: commands succeeded
  congratulations :)

——————————————————————————————————————

tested w/ python 3.6.9:

-........................................................-...................
.........-.....-.................-........-..................................
…..-…nltk/nltk/tokenize/stanford_segmenter.py:305: DeprecationWarning:
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
-....................................................................-.
...-....-.....-.-..-.......-..-..............................................
.............................................................................
.............................................................................
.......................-..-.......--....................--------.............
.................................-....................----...................
…………………………./nltk/nltk/test/unit/test_tokenize.py:60: DeprecationWarning:
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
-../nltk/nltk/test/unit/test_tokenize.py:86: DeprecationWarning:
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
-................................

-----------------------------------------------------------------------------
678 tests run in 167.563 seconds.
35 skipped (643 tests passed)
__________________________________________________________________________________ summary __________________________________________________________________________________
  py36: commands succeeded
  congratulations :)

——————————————————————————————————————

tested w/ python 3.5.4:

-........................................................-...................
.........-.....-.................-........-..................................
…..-../nltk/nltk/tokenize/stanford_segmenter.py:305: DeprecationWarning:
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
-....................................................................-.
...-....-.....-.-..-.......-..-..............................................
.............................................................................
.............................................................................
.......................-..-.......--....................--------.............
.................................-....................----...................
………………………./../nltk/nltk/test/unit/test_tokenize.py:60: DeprecationWarning:
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
-../nltk/nltk/test/unit/test_tokenize.py:86: DeprecationWarning:
The StanfordTokenizer will be deprecated in version 3.2.5.
Please use nltk.parse.corenlp.CoreNLPTokenizer instead.'
  seg = StanfordSegmenter()
-................................

-----------------------------------------------------------------------------
678 tests run in 187.496 seconds.
35 skipped (643 tests passed)
__________________________________________________________________________________ summary __________________________________________________________________________________
  py35: commands succeeded
  congratulations :)

alvations · 2020-01-28T06:09:03Z

@srhrshr Thanks for updating the travis yml. Now that looks great!!

@stevenbird @iliakur I think with scipy's Python 3.8 support upstream, nltk should build with 3.8 too.

From the CI side, it looks like it's LGTM for this.

stevenbird · 2020-02-05T23:15:08Z

Thanks @srhrshr and @alvations

fixes tests in py3.8

ef937a6

alvations self-assigned this Oct 27, 2019

alvations added CI python3.8 pythonic labels Nov 18, 2019

fixes tests in py3.8

230d07b

stevenbird mentioned this pull request Jan 24, 2020

Release new version #2490

Closed

srhrshr added 3 commits January 27, 2020 16:13

merges develop branch from upstream

9a58c20

updates tox.ini to support py38 after confirming that the tests pass.

74a283e

fixes upstream changes

a2d00dd

adds python 3.8 config item to travis yml

681809f

alvations self-requested a review January 28, 2020 06:09

alvations approved these changes Jan 28, 2020

View reviewed changes

alvations mentioned this pull request Jan 30, 2020

Dropping Python 2.7 Support #2296

Closed

stevenbird merged commit b0dcafc into nltk:develop Feb 5, 2020

srhrshr deleted the py38-compat branch February 5, 2020 23:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fixes tests in py3.8 #2432

fixes tests in py3.8 #2432

srhrshr commented Oct 19, 2019

iliakur commented Oct 19, 2019

srhrshr commented Oct 19, 2019

iliakur commented Oct 19, 2019 •

edited

srhrshr commented Oct 19, 2019

stevenbird commented Nov 18, 2019

stevenbird commented Nov 20, 2019

iliakur commented Jan 24, 2020

srhrshr commented Jan 24, 2020

alvations commented Jan 28, 2020 •

edited

srhrshr commented Jan 28, 2020

alvations commented Jan 28, 2020 •

edited

stevenbird commented Feb 5, 2020

fixes tests in py3.8 #2432

fixes tests in py3.8 #2432

Conversation

srhrshr commented Oct 19, 2019

iliakur commented Oct 19, 2019

srhrshr commented Oct 19, 2019

iliakur commented Oct 19, 2019 • edited

srhrshr commented Oct 19, 2019

stevenbird commented Nov 18, 2019

stevenbird commented Nov 20, 2019

iliakur commented Jan 24, 2020

srhrshr commented Jan 24, 2020

alvations commented Jan 28, 2020 • edited

srhrshr commented Jan 28, 2020

alvations commented Jan 28, 2020 • edited

stevenbird commented Feb 5, 2020

iliakur commented Oct 19, 2019 •

edited

alvations commented Jan 28, 2020 •

edited

alvations commented Jan 28, 2020 •

edited