Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERF: Loosen pytest-seed and add a file for collection broken seeds #80

Open
wants to merge 52 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
4c893a3
ERF: Loosen pytest-seed and add a file for collection broken seeds
NiMlr Apr 20, 2020
634eb55
ERF: more info for listing broken seeds
NiMlr May 14, 2020
4d15670
ERF: update broken seeds
NiMlr May 17, 2020
e352775
ERF: update broken seeds
NiMlr May 17, 2020
3985f0e
Update README
NiMlr Jun 16, 2020
b7c8de8
Rename README to README.md
NiMlr Jun 16, 2020
ae47980
Update README.md
NiMlr Jun 16, 2020
0a3cf4e
Update README.md
NiMlr Jun 16, 2020
30917d3
Update README.md
NiMlr Jun 16, 2020
a656531
Update README.md
NiMlr Jun 16, 2020
bd7831b
Update README.md
NiMlr Jun 16, 2020
780ec95
Update README.md
NiMlr Jun 16, 2020
adb7c11
Update README.md
NiMlr Jun 16, 2020
40c6765
Update README.md
NiMlr Jun 16, 2020
4e998b0
Update README.md
NiMlr Jun 16, 2020
c4e7fd0
Update README.md
NiMlr Jun 16, 2020
a0418db
Update README.md
NiMlr Jun 16, 2020
36e3a10
Update README.md
NiMlr Jun 16, 2020
e5de8ca
Update README.md
NiMlr Jun 16, 2020
0ec9cad
Update README.md
NiMlr Jun 16, 2020
66cbfff
Update README.md
NiMlr Jun 16, 2020
c2745d9
Update README.md
NiMlr Jun 16, 2020
921064e
Update README.md
NiMlr Jun 16, 2020
a947cef
Update README.md
NiMlr Jun 16, 2020
586b7c0
Update README.md
NiMlr Jun 16, 2020
642d1e9
Update README.md
NiMlr Jun 16, 2020
40183b5
Update README.md
NiMlr Jun 16, 2020
f4cab0d
Update README.md
NiMlr Jun 16, 2020
c6a4129
Update README.md
NiMlr Jun 16, 2020
49b5816
Update README.md
NiMlr Jun 16, 2020
06b0129
Update README.md
NiMlr Jun 16, 2020
a3c57a8
Update README.md
NiMlr Jun 16, 2020
1e37065
Update README.md
NiMlr Jun 16, 2020
ccc7f0a
Update README.md
NiMlr Jun 16, 2020
027d648
Update README.md
NiMlr Jun 16, 2020
44a4666
Update README.md
NiMlr Jun 16, 2020
bbe63ef
Update README.md
NiMlr Jun 16, 2020
b3ee9be
Update README.md
NiMlr Jun 16, 2020
78ec443
Update README.md
NiMlr Jun 16, 2020
24f18bf
Update README.md
NiMlr Jun 16, 2020
5f21186
Update README.md
NiMlr Jun 16, 2020
b1e01b1
Update README.md
NiMlr Jun 16, 2020
8f40edc
Update README.md
NiMlr Jun 16, 2020
f55de33
Update README.md
NiMlr Jun 16, 2020
1f8d476
Update README.md
NiMlr Jun 16, 2020
7a8f29a
Update README.md
NiMlr Jun 16, 2020
281490c
Update README.md
NiMlr Jun 16, 2020
d3afe60
Update README.md
NiMlr Jun 16, 2020
a72ac06
Update README.md
NiMlr Jun 17, 2020
873fa45
Update README.md
NiMlr Jun 17, 2020
8819c5e
Update README.md
NiMlr Jun 20, 2020
d494805
Update and rename README.md to README
NiMlr Jun 20, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions .travis.yml
Expand Up @@ -31,8 +31,8 @@ install:
- pip install $DEPS

script:
- pytest --cov-report= --cov-config=.coveragerc --cov=mdp --seed=725021957 mdp
- pytest --cov-report= --cov-config=.coveragerc --cov=bimdp --cov-append --seed=725021957 bimdp
- pytest --cov-report= --cov-config=.coveragerc --cov=mdp mdp
- pytest --cov-report= --cov-config=.coveragerc --cov=bimdp --cov-append bimdp

after_success:
- coveralls
12 changes: 12 additions & 0 deletions BROKEN_SEEDS
@@ -0,0 +1,12 @@
# This file is a collection of pytest-seeds that led to at least one of the
# tests failing, and it provides a way to spot faulty functionality.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add a clear specification of how to store the seeds. So, we should first find a broken seed so the we can add an example here. Anyway, we should explain that a failing test can be reproduced like this:

pytest --seed==N mdp/test/test_XXX.py -k test_YYY

The values for N, XXX and YYY can be read out of the pytest report after a failed test run.

Here an example of failing seed (from: 664d688)

pytest --seed=2133340156 mdp/test/test_GSFANode.py -k test_basic_GSFA_edge_dict

(note that to get a failure for this specific test you need to comment out the @pytest.mark.skip decorator in the file)
So maybe it is enough if we just add the above line to this file. When we discover other failing seed/tests, we can add the lines to reproduce here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think extending the section on testing in the docs is a better place for this or should at least be considered additionally?

#
# A description of how to gather the seeds is given at
# https://mdp-toolkit.github.io/development.html#how-to-handle-failed-tests.
#
# Please list them in the format:
# SEED, FUNCTIONALITY, TEST-NAME, PYTHON-VERSION, NUMPY-VERSION, SCIPY-VERSION, SKLEARN-VERSION

1766368184, SFANode, , 3.8.0.final.0, , 1.4.1, 0.23.0
400700733, VartimeCovariance, VartimeCovarianceMatrix1, 3.5.6.final.0, , 1.4.1, 0.22.2.post1
317438632, RBM, RBM_bv_learning, 3.6.7.final.0, , 1.4.1, 0.23.0
3 changes: 1 addition & 2 deletions README
@@ -1,3 +1,2 @@
Please refer to the online documentation at https://mdpdocs.readthedocs.io
The legacy documentation is still available at http://mdp-toolkit.sourceforge.net
Please refer to the online documentation at https://mdp-toolkit.github.io
The source code is available at https://github.com/mdp-toolkit/mdp-toolkit