unXFAIL two tests in test_wester

[debugger22]

Closes #7157

[toolforger]

How about leaving in a reference to #7157 so if this regresses, people know where to look for more information?

[debugger22]

Done.

[debugger22]

@toolforger Are you +1 now?

[toolforger]

I'd have left in the reference to the issue report.
Aaron was suggesting it's related to randomness, not to Python versions; that aspect seems to be unresolved even here, so it's important to keep that hint accessible so people know what to check if this should regress.

[debugger22]

@skirpichev Was fad74a8 reverted?

[asmeurer]

The tests are run with random seeds so if it is still an issue it should show up.

[debugger22]

It passed once. Should I restart it?

[skirpichev]

@skirpichev Was fad74a8 reverted?

I don't see this commit in the history.

[debugger22]

Me too but then why it has sympy/sympy in the URL of the commit.

[skirpichev]

No idea. Perhaps, this is commit in my fork (now deleted).

[toolforger]

From the old messages, it seemed to me that the randomness has just selects one or the other of two equally valid results.
Without detail knowledge of what exactly is happening there, we don't know the probabilities. With a 50:50 probability, you'd have a confidence of 1:1023 that the fix works if the tests pass; if the probability is 99.999:0.001, you'd have to run many more tests to be reasonably confident, and you wouldn't know when to stop testing.
IOW more testing can only reveal a problem but it can't make you confident.

Since we haven't seen wrong results, we probably can still just run the test as it is and revisit the issue if we ever see a problem. A few more test runs (locally, I guess) should give you enough confidence that it won't happen too soon on Travis, or for anybody who runs the tests locally.
(Still, it would be nice to know what was causing the problem then, and whether it's fixed or not. I don't know how to properly deal with heuristics in testing though - how do you find out whether two results are equivalent if the equivalence algorithm is part of the stuff you're testing?)

[skirpichev]

With a 50:50 probability, you'd have a confidence of 1:1023 that the fix works if the tests pass

Looks as arbitrary numbers for me... (BTW, how many words and how little sense.)

@debugger22, try something like this (works fine for me):

$ ./bin/test --rerun=10 -k test_V12 sympy/core/tests/test_wester.py

Maybe some stuff in heurisch still depends on dict/set's sorting. Maybe not, i.e. after 2e06e1a (I have failure for above command before this commit.). I bet it was fixed by @smichr ;)

[debugger22]

It fails.

========================== test process starts ===========================
executable:         /usr/bin/python  (2.7.5-final-0) [CPython]
architecture:       64-bit
cache:              yes
ground types:       python 
random seed:        19341154
hash randomization: on (PYTHONHASHSEED=3289598534)

sympy/core/tests/test_wester.py[1] F                                [FAIL]

__________________________________________________________________________
________________ sympy/core/tests/test_wester.py:test_V12 ________________
  File "/home/mrsud/workspace/sympy/sympy/core/tests/test_wester.py", line 2345, in test_V12
    assert r1 == -1/(tan(x/2) + 2)
AssertionError

As @asmeurer already pointed out, it's independent of Python version too.

Perhaps it can be closed.

[skirpichev]

Oh, it seems I have found this "source of randomness".

[toolforger]

Am 12.04.2015 um 01:06 schrieb Sergey B Kirpichev:

With a 50:50 probability, you'd have a confidence of 1:1023 that the fix works if the tests pass

Looks as arbitrary numbers for me...

Yeah. Arbitrary numbers are often a stand-in for free variables. So:
If you have a k:(1-k) probability, the confidence is k**N after N tests.

(BTW, how many words and how little sense.)

Trolling taken to private mail.

[skirpichev]

BTW, there is no "wrong result". I get tan(x/2)/(2*tan(x/2) + 4), that's same antiderivative as expr below.

[debugger22]

Yes. The comment is misleading. It was already pointed out on the issue.