New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Surprising interaction between file_pattern and expected_failing_examples #335

Closed
lesteve opened this Issue Jan 4, 2018 · 3 comments

Comments

Projects
None yet
3 participants
@lesteve
Contributor

lesteve commented Jan 4, 2018

If through filename_pattern you are not running an example that is expected to fail you will get an error that the example has not failed as expected.

I would expect that there is no error: in my use case I just want to run one example I could not care less about examples that are expected to fail.

To reproduce apply this diff (set filename_pattern to seaborn essentially):

diff --git a/doc/conf.py b/doc/conf.py
index ac53cf6..24c4cfe 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -339,6 +339,7 @@ sphinx_gallery_conf = {
                                        '../tutorials/seaborn']),
     'within_subsection_order': NumberOfCodeLinesSortKey,
     'find_mayavi_figures': find_mayavi_figures,
+    'filename_pattern': 'seaborn',
     'expected_failing_examples': ['../examples/no_output/plot_raise.py',
                                   '../examples/no_output/plot_syntaxerror.py'],
     'binder': {'org': 'sphinx-gallery',

Relevant part of the output:

Examples expected to fail, but not failling:
Please remove these examples from
sphinx_gallery_conf['expected_failing_examples']
/home/local/lesteve/dev/sphinx-gallery/examples/no_output/plot_syntaxerror.pyin your conf.py file
/home/local/lesteve/dev/sphinx-gallery/examples/no_output/plot_raise.py
@jnothman

This comment has been minimized.

Contributor

jnothman commented Mar 27, 2018

xpass should probably not be a fatal error in any case. Here we are using xfail because passing is unreliable.

@lesteve lesteve added this to the v0.1.14 milestone Mar 28, 2018

@lesteve

This comment has been minimized.

Contributor

lesteve commented Mar 28, 2018

I agree that a xpass could only be a warning. It definitely has some value, for example when mldata.org misbehaves and some examples may or may not fail almost randomly.

Having said that, it is slightly orthogonal to this issue which is about interaction between filename_pattern and expected_failing_examples.

I think we did it like this because we were worried that people would eventually fix their example and forget to remove it from the 'expected_failing_examples' configuration value.

Side-comment: interestingly pytest has a strict=True parameter for xfail, when you are sure that this test should fail. If it does not fail, then this is considered as an error.

@choldgraf

This comment has been minimized.

Contributor

choldgraf commented Apr 25, 2018

ping on this one - it's a minor blocker on the next release :-)

Titan-C added a commit to Titan-C/sphinx-gallery that referenced this issue May 23, 2018

Fix sphinx-gallery#335
Apply a filter to examples_not_expected_to_pass to validate that they where
actually run.

Put color to error messages header so that they can be easily found on
sphinx log.

lesteve added a commit to Titan-C/sphinx-gallery that referenced this issue Jun 4, 2018

Fix sphinx-gallery#335
Apply a filter to examples_not_expected_to_pass to validate that they where
actually run.

Put color to error messages header so that they can be easily found on
sphinx log.

@lesteve lesteve closed this in #379 Jun 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment