Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check output in example tests #7280

Merged
merged 3 commits into from
Nov 27, 2019
Merged

Check output in example tests #7280

merged 3 commits into from
Nov 27, 2019

Conversation

niboshi
Copy link
Member

@niboshi niboshi commented May 25, 2019

Fixes #6094

Depends on #7277

@niboshi niboshi mentioned this pull request May 25, 2019
Closed
@niboshi niboshi force-pushed the example-test-check-output branch 8 times, most recently from a973930 to adce6d0 Compare May 25, 2019 14:55
@niboshi
Copy link
Member Author

niboshi commented May 27, 2019

I found pytest-repeat useful to investigate the acceptable fluctuations of the statistics, but it does not support unittest style tests. So I wrote the test in pytest-style.

@niboshi niboshi added the prio:high High priority. Urgent and needs to be worked on as soon as possible. label May 27, 2019
beam2d
beam2d reviewed May 28, 2019
View reviewed changes
examples/tests/test_mnist.py
c=(float, lambda x: 0.62 < x < 0.82),
d=(float, lambda x: 0.83 < x < 0.98),
e=(float, lambda x: 0. < x < 100.),
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to find information for choosing this new format method.

  1. Possible other solutions
  2. Pros and cons of these solutions (and thus why the above code is good)

Copy link
Member Author

@niboshi niboshi May 28, 2019 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I didn't come up with any other reasonable solutions.
I'll try to list possibilities any way.

  • Write in each test a logic to check the output line by line

    • Pros: What's being done in each step is clear from test implementation.
    • Cons: Prohibitively difficult to write/maintain test cases.

  • Use a regular expression to capture the numbers, and use lambdas to check the numerical ranges.

    • Pros: Being a standard notation
    • Cons: Difficult to express the output with a regular expression. It's also not easy to interpret for human. Possibilities to not be able to express with the syntax.

  • This method

    • Pros: Easy to interpret test cases. Easy to add a new test case (just copy-paste the output and replace some parts with {...}). Flexible (any checks can be added later).
    • Cons: Occasionally new logic might be needed in the tool behind. Being non-standard.

@niboshi niboshi mentioned this pull request Jun 11, 2019
Merged
@stale
Copy link

stale bot commented Aug 26, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale Not updated for a longer period of time. label Aug 26, 2019
@stale
Copy link

stale bot commented Sep 25, 2019

This issue is closed as announced. Feel free to re-open it if needed.

@stale stale bot closed this Sep 25, 2019
@kmaehashi kmaehashi added this to the Closed issues and PRs milestone Sep 26, 2019
@niboshi niboshi reopened this Oct 7, 2019
@stale stale bot removed the stale Not updated for a longer period of time. label Oct 7, 2019
@niboshi niboshi force-pushed the example-test-check-output branch 2 times, most recently from ecc7a2b to bb48251 Compare October 7, 2019 06:20
@niboshi
Copy link
Member Author

niboshi commented Oct 7, 2019

Rebased as #7475 has been merged.

@niboshi
Copy link
Member Author

niboshi commented Oct 7, 2019

PTAL

@kmaehashi kmaehashi removed this from the Closed issues and PRs milestone Oct 25, 2019
@emcastillo emcastillo self-assigned this Nov 20, 2019
@niboshi
Copy link
Member Author

niboshi commented Nov 20, 2019

Conflict resolved.

emcastillo
emcastillo approved these changes Nov 25, 2019
View reviewed changes
Copy link
Member

@emcastillo emcastillo left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Seems like we need to install matplotlib on Travis, or we could either remove the dependence from the example test?

@niboshi
Copy link
Member Author

niboshi commented Nov 25, 2019

matplotlib cannot be removed from requirements. I add a step to install matplotlib in Travis CI.

emcastillo
emcastillo reviewed Nov 25, 2019
View reviewed changes
scripts/ci/travis/run-tests.sh Outdated
@@ -174,6 +174,7 @@ case "${CHAINER_TRAVIS_TEST}" in
install)
run_prestep install_chainer_test_deps
run_prestep chainer_install_from_sdist
run_prestep chainer_example_tests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't be install_chainer_example_tests?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops... Force-pushed in f586e60.
Thanks!

@emcastillo
Copy link
Member

Jenkins, test this please

@emcastillo emcastillo added cat:test Test or CI related. st:test-and-merge State indicating that pull request is approved by a reviewer and can be merged after CI passes. labels Nov 25, 2019
@emcastillo emcastillo added this to the v7.0.0 milestone Nov 25, 2019
@chainer-ci
Copy link
Member

Jenkins CI test (for commit f586e60, target branch master) succeeded!

@emcastillo emcastillo merged commit abd7c43 into chainer:master Nov 27, 2019
@emcastillo emcastillo removed the st:test-and-merge State indicating that pull request is approved by a reviewer and can be merged after CI passes. label Nov 27, 2019
@niboshi niboshi deleted the example-test-check-output branch November 27, 2019 02:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@beam2d beam2d beam2d left review comments

@emcastillo emcastillo emcastillo approved these changes

Assignees

@beam2d beam2d

@emcastillo emcastillo

Labels
cat:test Test or CI related. prio:high High priority. Urgent and needs to be worked on as soon as possible.
Projects
None yet
Milestone
v7.0.0
Development

Successfully merging this pull request may close these issues.

Check output in example tests
5 participants
@niboshi @emcastillo @chainer-ci @beam2d @kmaehashi