I can only guess that test_basic.py has so many expect_failure=True's out of a desire to be resilient if the testing environment doesn't have tools like SVN installed. Really, if there's a failure in [[http://bitbucket.org/ianb/pip/src/tip/tests/test_basic.py#cl-50|the SVN command]], though, it's unlikely that any of the following lines will pass. As it is, the instruction to skip looks like at best obfuscation and at worse something that could cause a real problem to slip by undetected. The right answer is probably to use a mechanism to skip tests with dependencies that aren't installed (and have a mode where everything is run anyway so we can find out what's needed)
I think I misinterpreted what's going on here. In most of these cases we
really should expect an error. In fact, it's stronger than that: we want
an error. The problem is that scripttest has no way to express that: itonly
has a way of ignoring errors but no way of insisting on them (i.e. asserting
if there's no error). As a result it just masks the cases where we expect an
error but in reality we don't get one, and there's nothing checking to make
sure we haven't just liberally sprinkled expect_error=True even where it
really doesn't belong.
Original Comment By: Dave Abrahams
Dave: was this cleaned up to your satisfaction in your branch, now merged? Or
is there reason to leave this open? Resolving unless you say otherwise.
Original Comment By: Carl Meyer
No, it wasn't cleaned up systematically. What should happen is that when
expect_error=True, the test framework raises an AssertionError unless an
error actually did occur. That would immediately reveal all the places where
it's been added erroneously.
Pretty sure this is long since no longer applicable.
we still have many cases of expect_failure=True. I'm guessing a number of those aren't need, but I'd have to look closer at them.
Oh, whoops. I totally misread this ticket, I thought it was talking about expected failures of the test suite not of script test, that's what I get for not reading closely enough.
Just to be clear, the flag is called expect_error, I do believe we still have many instances of them and most of them are invalid and should be removed.
Should we remove all test with "expect_error"? Is there a standard to identify it as invalid?
The easy fix part I had in mind was to check wheither the tests were in fact passing but I don't think we have any test unexpectedly passing. So nevermind the easy fix ^^
What's the appropriate action to take to resolve this then?
All over the place there are tests that call script.pip (or similar) that have an expect_error=True or expect_stderr=True that don't need it. The way to resolve this ticket is for someone to go through and remove any of those that are not needed for the tests to pass.