Add Python 3.7 and Python 3.8 to CI test jobs #2450

munkm · 2020-02-10T22:16:55Z

PR Summary

This PR is a progression from the work happening in #2448. In that PR some of the CI test jobs on travis were changed as a result of updates our code related to numpy compatibility. The highest level of Python that we're testing against on both appveyor and travis right now is 3.6. This PR adds jobs to both appveyor and travis for Python 3.7 and 3.8. It also removes the python 2.7 testing from appveyor.

(this should wait to be merged until after 2448 is merged.)

…ts file

munkm · 2020-03-10T21:17:14Z

@yt-fido test this please

munkm · 2020-03-10T21:18:05Z

The environment issue for appveyor on python 3.8 is solved now, so this should be good to merge.

munkm · 2020-03-10T21:28:41Z

Nevermind. The build resolves but the tests aren't running.

neutrinoceros · 2020-03-16T09:35:59Z

I remember you mentioning matplotlib 3.2 failures on appveyor, but couldn't find your statement back. Since the present PR is challenging in itself, I'd suggest we freeze mpl to 3.1 here, for the time being.

munkm · 2020-03-16T22:02:12Z

So the test failures in this particular PR are not stemming from matplotlib 3.2 (yet. I'm sure they will appear if I push an update.). For some reason the Python 3.8 environment isn't running the tests at all and I need to figure out why.

munkm · 2020-03-19T04:01:23Z

@yt-fido test this please

munkm · 2020-03-20T14:32:15Z

Ok I did some digging on the failures on Jenkins. I think the issue is that for SciPy v 1.4 there are import errors with Python 3.5 (see here: scipy/scipy#11356) . When SciPy 1.5 is released we can use that (or if we update to Python 3.6 we can use Scipy 1.4). For now I'm bumping the SciPy version back down to 1.3.3.

For cartopy the failures are also resulting from SciPy. Because SciPy isn't importing, cartopy is trying to use pykdtree for the transform/projections and then an error is being thrown because that package isn't installed. Bumping the scipy version fixes this.

Xarthisius · 2020-03-20T23:17:12Z

I run into an issue with MPL-3.0.3 and py3.8. TriLinearInterpolation seems to take ages. I recall we had a similar problem when cython/c module responsible for it was compiled in debug mode upstream. I checked that mpl-3.1.3 works just fine. Can we use that version as min requirement for tests?

Xarthisius · 2020-03-21T01:43:46Z

@yt-fido test this please

munkm · 2020-03-21T05:09:34Z

Ok, done! I bumped the mpl version from 3.0.3 to 3.1.1 in thetest_requirements file, not the test_minimum_requirements file. Let me know if I should bump another place!

Xarthisius · 2020-03-27T19:29:17Z

.travis.yml

+    - stage: tests
+      name: "Python: 3.7 Unit Tests"
+      python: 3.7
+      script: coverage run $(which nosetests) -c nose_unit.cfg


I think it's sufficient to test only 3.6 and 3.8. Hopefully it's going to be enough and will save us a lot of time.

ok i've removed py 3.5 and 3.7 from appveyor and travis

matthewturk · 2020-04-01T01:51:14Z

The answer test difference results are here:

http://use.yt/upload/1f30ffb2

Looking these over, they all look quite reasonable, and likely are the result of matplotlib changes.

munkm · 2020-04-01T06:38:27Z

Ok all of the tests pass now, so this should be ready for review!

cphyc

I left some comments, mostly stylistic ones but overall this LGTM.

cphyc · 2020-04-01T08:18:15Z

tests/test_requirements.txt

@@ -16,12 +16,14 @@ pandas==0.23.4
 pytest~=5.2; python_version >= '3.0'
 pytest~=4.6; python_version < '3.0'
 requests==2.20.0
-scipy==1.1.0
+scipy==1.1.0; python_version < '3.0'


Since Python 2 is not supported, is there a reason to keep this?

We're still running tests for py2.7, just not on Travis. It hasn't been dropped yet.

cphyc · 2020-04-01T08:18:41Z

tests/test_requirements.txt

 sympy==1.5
 pyqt5==5.11.3; python_version >= '3.0'
 thingking==1.0.2; python_version < '3.0'
 pint==0.8.1
-netCDF4==1.4.2
+netCDF4==1.4.2; python_version < '3.0'
+netCDF4==1.5.3; python_version >= '3.0'


Same as above, I would advocate to remote the python_version < '3.0' line.

cphyc · 2020-04-01T08:22:58Z

.travis.yml

+    if [[ ${TRAVIS_BUILD_STAGE_NAME} == "lint" ]]; then
+      export TRAVIS_BUILD_STAGE_NAME="Lint"
+    fi
+    if [[ ${TRAVIS_BUILD_STAGE_NAME} != "Lint" ]]; then


Here and below, what is the reason to use Lint instead of lint?

I suppose it could be either, but basically what seemed to be happening is that while previously Travis had been capitalizing the stage names (i.e., lint became Lint) for some reason it no longer was in our jobs. And, because OS X runs bash 3.2, we couldn't use the ${VAR^^} or ${VAR,,} construction to change the case as a whole.

So, this is in there in case it switches back. It could use either one, but we figured that the least invasive one would be to assume the old behavior would return some day.

brittonsmith

Great job getting the test working again! I second all of the comments from @cphyc, but don't have any more of my own, so I'll approve.

neutrinoceros

Well done ! Same as @brittonsmith , I second @cphyc's review and have no further comment.
Thanks a lot for fixing this !

Porting #2450 (test job updates) to yt-4.0

munkm added 2 commits February 10, 2020 13:11

update appveyor test jobs

46fbf21

add py3.7 and py3.8 unit test jobs to travis

efe9ab6

munkm force-pushed the test-jobs-update branch from 7bdd77d to efe9ab6 Compare February 10, 2020 22:29

munkm added 5 commits February 11, 2020 13:26

bump scipy and netcdf4 versions for test requirements

85902e7

specify scipy and netcdf4 versions for both py2 and py3 in requiremen…

29e44b4

…ts file

Merge branch 'master' into test-jobs-update

969f512

fix issue with tearDown in geo_projections tests

39126ba

Merge branch 'master' into test-jobs-update

c6be810

munkm mentioned this pull request Feb 26, 2020

try to use pip instead of conda-build for develop install #2463

Closed

Merge branch 'master' into test-jobs-update

247c598

munkm added 2 commits March 20, 2020 12:09

add traverse-namespace to nose flags

75c6cb4

bump scipy testing version for py3 and higher

0284d5e

neutrinoceros mentioned this pull request Mar 20, 2020

[WIP] try fixing appveyor (conda develop -> pip install) ? #2493

Closed

bump mpl version for py3 and higher in testing

685589d

munkm added the yt-3.6 feature targeted for the yt-3.6 release label Mar 27, 2020

Xarthisius reviewed Mar 27, 2020

View reviewed changes

munkm mentioned this pull request Mar 31, 2020

Bump netcdf4 requirement #2507

Closed

munkm and others added 3 commits March 31, 2020 16:04

remove py 3.5 and 3.7 from CI test platforms

7ddc77c

Adding an echo to clarify the variable names

4e1fa82

Try to force uppercase LINT

8da3bc8

manually set the build stage name

dbc1aed

matthewturk and others added 2 commits March 31, 2020 20:52

Updating test results

7c2ed91

try using pip in appveyor install

d37c4e9

cphyc approved these changes Apr 1, 2020

View reviewed changes

brittonsmith approved these changes Apr 1, 2020

View reviewed changes

neutrinoceros approved these changes Apr 1, 2020

View reviewed changes

jzuhone approved these changes Apr 1, 2020

View reviewed changes

Xarthisius approved these changes Apr 1, 2020

View reviewed changes

Xarthisius merged commit fa959a6 into yt-project:master Apr 1, 2020

matthewturk added a commit to matthewturk/yt that referenced this pull request Apr 1, 2020

Porting yt-project#2450 to yt-4.0

ab97110

matthewturk mentioned this pull request Apr 1, 2020

Porting #2450 (test job updates) to yt-4.0 #2513

Merged

matthewturk mentioned this pull request Apr 13, 2020

[WIP] Use tox for managing environments #2537

Closed

Xarthisius added a commit that referenced this pull request Apr 17, 2020

Merge pull request #2513 from matthewturk/port-test-jobs-update

fecf079

Porting #2450 (test job updates) to yt-4.0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Python 3.7 and Python 3.8 to CI test jobs #2450

Add Python 3.7 and Python 3.8 to CI test jobs #2450

munkm commented Feb 10, 2020 •

edited

munkm commented Mar 10, 2020

munkm commented Mar 10, 2020

munkm commented Mar 10, 2020

neutrinoceros commented Mar 16, 2020

munkm commented Mar 16, 2020

munkm commented Mar 19, 2020

munkm commented Mar 20, 2020 •

edited

Xarthisius commented Mar 20, 2020

Xarthisius commented Mar 21, 2020

munkm commented Mar 21, 2020 •

edited

Xarthisius Mar 27, 2020

munkm Mar 31, 2020

matthewturk commented Apr 1, 2020

munkm commented Apr 1, 2020

cphyc left a comment

cphyc Apr 1, 2020

Xarthisius Apr 1, 2020

cphyc Apr 1, 2020

cphyc Apr 1, 2020

matthewturk Apr 1, 2020

brittonsmith left a comment

neutrinoceros left a comment

Add Python 3.7 and Python 3.8 to CI test jobs #2450

Add Python 3.7 and Python 3.8 to CI test jobs #2450

Conversation

munkm commented Feb 10, 2020 • edited

PR Summary

munkm commented Mar 10, 2020

munkm commented Mar 10, 2020

munkm commented Mar 10, 2020

neutrinoceros commented Mar 16, 2020

munkm commented Mar 16, 2020

munkm commented Mar 19, 2020

munkm commented Mar 20, 2020 • edited

Xarthisius commented Mar 20, 2020

Xarthisius commented Mar 21, 2020

munkm commented Mar 21, 2020 • edited

Xarthisius Mar 27, 2020

Choose a reason for hiding this comment

munkm Mar 31, 2020

Choose a reason for hiding this comment

matthewturk commented Apr 1, 2020

munkm commented Apr 1, 2020

cphyc left a comment

Choose a reason for hiding this comment

cphyc Apr 1, 2020

Choose a reason for hiding this comment

Xarthisius Apr 1, 2020

Choose a reason for hiding this comment

cphyc Apr 1, 2020

Choose a reason for hiding this comment

cphyc Apr 1, 2020

Choose a reason for hiding this comment

matthewturk Apr 1, 2020

Choose a reason for hiding this comment

brittonsmith left a comment

Choose a reason for hiding this comment

neutrinoceros left a comment

Choose a reason for hiding this comment

munkm commented Feb 10, 2020 •

edited

munkm commented Mar 20, 2020 •

edited

munkm commented Mar 21, 2020 •

edited