cov: add tests for ignite.contrib.engines #1808

ydcjeff · 2021-03-17T12:01:33Z

Part of #1790

Description: Add tests for ignite.contrib.engines

Check list:

New tests are added (if a new feature is added)
New doc strings: description and/or example code are in RST format
Documentation is updated (if required)

tests/ignite/contrib/engines/test_common.py

ignite/handlers/checkpoint.py

tests/ignite/contrib/engines/test_common.py

sdesrozis

@ydcjeff Thank you!

sdesrozis · 2021-03-18T07:55:13Z

@ydcjeff Windows tests are ko. Could you have a look ?

ydcjeff · 2021-03-18T07:58:25Z

@sdesrozis windows tests are failing on test_average_precision

=========================== short test summary info ===========================
FAILED tests/ignite/contrib/metrics/test_average_precision.py::test_binary_input_N
FAILED tests/ignite/contrib/metrics/test_average_precision.py::test_multilabel_input_N

Error is metric is computing nan

>       assert average_precision_score(np_y, np_y_pred) == pytest.approx(res)
E       assert nan == nan � ???
E         +nan
E         -nan � ???

vfdev-5 · 2021-03-19T17:17:14Z

@sdesrozis windows tests are failing on test_average_precision

=========================== short test summary info ===========================
FAILED tests/ignite/contrib/metrics/test_average_precision.py::test_binary_input_N
FAILED tests/ignite/contrib/metrics/test_average_precision.py::test_multilabel_input_N

Error is metric is computing nan

>       assert average_precision_score(np_y, np_y_pred) == pytest.approx(res)
E       assert nan == nan � ???
E         +nan
E         -nan � ???

@KickItLikeShika have you seen this ?

KickItLikeShika · 2021-03-19T18:04:58Z

@vfdev-5 This never happened before..

KickItLikeShika · 2021-03-19T18:15:43Z

@vfdev-5 i don't think it's something related to the metric computation itself, please restart the CI to see more details

vfdev-5 · 2021-03-19T18:30:00Z

@vfdev-5 i don't think it's something related for the metric computation itself, please restart the CI to see more details

At first, you can try to checkout this PR locally and try to repro the issue if you have a Win32 machine. I think it is related as with new tests we have altered random state and it generates a corner case samples...

KickItLikeShika · 2021-03-19T21:26:16Z

@vfdev-5 i'm using Linux, i will try to investigate more and see what's the problem exactly

sdesrozis · 2021-03-22T06:21:20Z

@fco-dv could you help to track this issue on windows ? Thanks !

vfdev-5

Thanks for the update @ydcjeff

tests/ignite/contrib/metrics/test_gpu_info.py

fco-dv · 2021-03-25T11:35:54Z

I've checkout the PR , those tests are PASSED on my side:

@fco-dv could you help to track this issue on windows ? Thanks !

(ignite-dev) D:\git\ignite>pytest  tests/ignite/contrib/metrics/test_average_precision.py -vvv
================================================= test session starts =================================================
platform win32 -- Python 3.7.7, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- d:\soft\miniconda3\envs\ignite-dev\python.exe
cachedir: .pytest_cache
rootdir: D:\git\ignite, inifile: setup.cfg
plugins: cov-2.8.1, forked-1.1.3, xdist-1.31.0
collected 15 items

tests/ignite/contrib/metrics/test_average_precision.py::test_no_sklearn PASSED                                   [  6%]
tests/ignite/contrib/metrics/test_average_precision.py::test_no_update PASSED                                    [ 13%]
tests/ignite/contrib/metrics/test_average_precision.py::test_input_types PASSED                                  [ 20%]
tests/ignite/contrib/metrics/test_average_precision.py::test_check_shape PASSED                                  [ 26%]
tests/ignite/contrib/metrics/test_average_precision.py::test_binary_input_N PASSED                               [ 33%]
tests/ignite/contrib/metrics/test_average_precision.py::test_multilabel_input_N PASSED                           [ 40%]
tests/ignite/contrib/metrics/test_average_precision.py::test_integration_binary_input_with_output_transform PASSED [ 46%]
tests/ignite/contrib/metrics/test_average_precision.py::test_integration_multilabel_input_with_output_transform PASSED [ 53%]
tests/ignite/contrib/metrics/test_average_precision.py::test_distrib_gpu SKIPPED                                 [ 60%]
tests/ignite/contrib/metrics/test_average_precision.py::test_distrib_cpu SKIPPED                                 [ 66%]
tests/ignite/contrib/metrics/test_average_precision.py::test_distrib_hvd SKIPPED                                 [ 73%]
tests/ignite/contrib/metrics/test_average_precision.py::test_multinode_distrib_cpu SKIPPED                       [ 80%]
tests/ignite/contrib/metrics/test_average_precision.py::test_multinode_distrib_gpu SKIPPED                       [ 86%]
tests/ignite/contrib/metrics/test_average_precision.py::test_distrib_single_device_xla SKIPPED                   [ 93%]
tests/ignite/contrib/metrics/test_average_precision.py::test_distrib_xla_nprocs SKIPPED                          [100%]

============================================ 8 passed, 7 skipped in 1.87s =============================================

vfdev-5 · 2021-03-25T13:12:51Z

@fco-dv try complete test suite, maybe random seed is not the same between reduced and total runs.

fco-dv · 2021-03-25T13:57:52Z

@vfdev-5 I've launched the tests with pytest tests -vvv and it passed , however I'm not sure to reproduce the command launched on the CI

tests/ignite/contrib/metrics/test_average_precision.py::test_no_update PASSED                                    [ 22%]
tests/ignite/contrib/metrics/test_average_precision.py::test_input_types PASSED                                  [ 22%]
tests/ignite/contrib/metrics/test_average_precision.py::test_check_shape PASSED                                  [ 22%]
tests/ignite/contrib/metrics/test_average_precision.py::test_binary_input_N PASSED                               [ 22%]
tests/ignite/contrib/metrics/test_average_precision.py::test_multilabel_input_N PASSED                           [ 22%]
tests/ignite/contrib/metrics/test_average_precision.py::test_integration_binary_input_with_output_transform PASSED [ 22%]
tests/ignite/contrib/metrics/test_average_precision.py::test_integration_multilabel_input_with_output_transform PASSED [ 22%]

sdesrozis · 2021-03-27T07:02:18Z

@fco-dv Sorry for that but could you test reproducing the CI, maybe in a docker ? IMO It would be useful to have this dockerfile in the repo to ease the tracking of CI errors on windows (I suppose).

vfdev-5 · 2021-04-11T08:50:33Z

@KickItLikeShika could you please inspect this failure on windows with test_average_precision.py. Thanks

ydcjeff · 2021-04-11T10:21:21Z

@vfdev-5 @KickItLikeShika Could this be failing case?

>>> y_pred = torch.randint(0, 2, size=(10,)).long()
>>> n_iters = 16
>>> idx
13
>>> y_pred[idx : idx + batch_size]
tensor([], dtype=torch.int64)

vfdev-5 · 2021-04-11T10:43:36Z

Thanks a lot @ydcjeff ! @KickItLikeShika can you make a pass over all contrib tests and fix all these issues we already talked about in some of your previous PRs.

KickItLikeShika · 2021-04-11T10:49:56Z

Thanks a lot @ydcjeff !

@KickItLikeShika can you make a pass over all contrib tests and fix all these issues we already talked about in some of your previous PRs.

@vfdev-5 I will do that

vfdev-5

Thanks a lot @ydcjeff !

ydcjeff added 4 commits March 17, 2021 18:30

cov: add test for ignite.contrib.engines

ec1ba33

Merge remote-tracking branch 'upstream/master' into common-cov

01fcefa

fix test

1cb62f7

use shutil.move

d0c617d

github-actions bot added the module: handlers Core Handlers module label Mar 17, 2021

sdesrozis reviewed Mar 17, 2021

View reviewed changes

tests/ignite/contrib/engines/test_common.py Outdated Show resolved Hide resolved

sdesrozis reviewed Mar 17, 2021

View reviewed changes

ignite/handlers/checkpoint.py Outdated Show resolved Hide resolved

ydcjeff commented Mar 18, 2021

View reviewed changes

tests/ignite/contrib/engines/test_common.py Outdated Show resolved Hide resolved

Jeff Yang added 2 commits March 18, 2021 09:26

Update tests/ignite/contrib/engines/test_common.py

1f7d613

Merge branch 'master' into common-cov

f750880

sdesrozis approved these changes Mar 18, 2021

View reviewed changes

Merge branch 'master' into common-cov

3afb9d7

sdesrozis and others added 4 commits March 18, 2021 10:49

Merge branch 'master' into common-cov

1f4e89a

Merge branch 'master' into common-cov

f36816f

Merge branch 'master' into common-cov

91de432

Merge branch 'master' into common-cov

2af8ede

Merge branch 'master' into common-cov

839cca4

Jeff Yang added 3 commits March 24, 2021 14:24

Merge branch 'master' into common-cov

5064ec0

fix: flake8 unused imports terminate_on_nan

5f33cae

fix: patch.dict for failing test_no_pynvml_package test

fe7f415

vfdev-5 reviewed Mar 24, 2021

View reviewed changes

tests/ignite/contrib/metrics/test_gpu_info.py Show resolved Hide resolved

tests/ignite/contrib/metrics/test_gpu_info.py Show resolved Hide resolved

Merge remote-tracking branch 'upstream/master' into common-cov

d3fb718

ydcjeff added 2 commits March 25, 2021 11:38

fix: remove py36 checks

11057fb

fix: skipif tests

68d11ad

Jeff Yang added 2 commits March 26, 2021 14:01

Merge branch 'master' into common-cov

fa02453

Merge branch 'master' into common-cov

415c4db

Jeff Yang and others added 3 commits April 1, 2021 13:18

Merge branch 'master' into common-cov

06d38be

Merge branch 'master' into common-cov

cb0fcfa

Merge branch 'master' into common-cov

20eb936

vfdev-5 mentioned this pull request Apr 8, 2021

Improve tests #1786

Closed

Merge remote-tracking branch 'upstream/master' into common-cov

bd6d236

ydcjeff and others added 3 commits April 11, 2021 15:28

test: debug failing test

3c50e0d

autopep8 fix

dc0ec68

test: debug failing test [skip netlify]

dbf1174

fix: change 10 -> 16

a661ad7

ydcjeff requested a review from vfdev-5 April 11, 2021 10:51

vfdev-5 approved these changes Apr 11, 2021

View reviewed changes

vfdev-5 merged commit 23d963c into pytorch:master Apr 11, 2021

ydcjeff deleted the common-cov branch April 11, 2021 11:18

Uh oh!

cov: add tests for ignite.contrib.engines #1808

cov: add tests for ignite.contrib.engines #1808

Uh oh!

Conversation

ydcjeff commented Mar 17, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sdesrozis left a comment

Choose a reason for hiding this comment

Uh oh!

sdesrozis commented Mar 18, 2021

Uh oh!

ydcjeff commented Mar 18, 2021

Uh oh!

vfdev-5 commented Mar 19, 2021

Uh oh!

KickItLikeShika commented Mar 19, 2021

Uh oh!

KickItLikeShika commented Mar 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vfdev-5 commented Mar 19, 2021

Uh oh!

KickItLikeShika commented Mar 19, 2021

Uh oh!

sdesrozis commented Mar 22, 2021

Uh oh!

vfdev-5 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

fco-dv commented Mar 25, 2021

Uh oh!

vfdev-5 commented Mar 25, 2021

Uh oh!

fco-dv commented Mar 25, 2021

Uh oh!

sdesrozis commented Mar 27, 2021

Uh oh!

vfdev-5 commented Apr 11, 2021

Uh oh!

ydcjeff commented Apr 11, 2021

Uh oh!

vfdev-5 commented Apr 11, 2021

Uh oh!

KickItLikeShika commented Apr 11, 2021

Uh oh!

vfdev-5 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

KickItLikeShika commented Mar 19, 2021 •

edited

Loading