Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test failures: newton_cg and newton_cg_backtracking #2

Open
Dapid opened this issue Aug 28, 2014 · 10 comments
Open

Test failures: newton_cg and newton_cg_backtracking #2

Dapid opened this issue Aug 28, 2014 · 10 comments
Assignees

Comments

@Dapid
Copy link
Contributor

Dapid commented Aug 28, 2014

I have successfuly built and installed Optizelle, but make test reports two errors:

 171 - Solution_to_cpp_/home/david/Downloads/Optizelle-master/src/examples/simple_quadratic_cone/newton_cg.json (Failed)
    173 - Solution_to_cpp_/home/david/Downloads/Optizelle-master/src/examples/simple_quadratic_cone/newton_cg_backtracking.json (Failed)

I have activated Python, C++, and OpenMP bindings. I have only tested Release mode with flags O2 and O3, and march=native and default.

I am on a Fedora 20 box, with gcc 4.8.3, jsoncpp 0.6.0 release 0.11.rc2 from the repositories, and cmake 2.8.12.2.

@josyoun
Copy link
Member

josyoun commented Aug 28, 2014

Well, funny enough, that's normal and expected. Unless something new is going on, it's likely that these tests are failing because the number of iterations changed by 1 or 2. If you wouldn't mind verifying, run ctest -V -I 170,171,1 and ctest -V -I 172,173,1 and let know what is says.

As a longer discussion, most of these tests check two primary things. One, that we get the right answer. Two, that the number of iterations remains the same. Generally, if I make an algorithmic change and really mess things up, the number of iterations changes dramatically and I like to be alerted to that. Unfortunately, due to numerical differences between platforms, one or two problems have their iteration count change when running on a new machine. Really, this isnt an issue with the problem, but that my testing infrastructure needs to be improved to account for this. I've been meaning to add an issue about this, so I'm glad you reported it since that means I'll be forced to fix it now.

@Dapid
Copy link
Contributor Author

Dapid commented Aug 28, 2014

ctest -V -I 170,171,1

<passing test skipped>
    Start 171: Solution_to_cpp_/home/david/Downloads/Optizelle-master/src/examples/simple_quadratic_cone/newton_cg.json

171: Test command: /home/david/Downloads/Optizelle-master/build/src/unit/utility/diff_restart "/home/david/Downloads/Optizelle-master/src/examples/simple_quadratic_cone/newton_cg.json" "solution.json"
171: Test timeout computed to be: 9.99988e+06
171: Mismatch in Naturals.iter: 11 != 16.
2/2 Test #171: Solution_to_cpp_/home/david/Downloads/Optizelle-master/src/examples/simple_quadratic_cone/newton_cg.json ....***Failed    0.00 sec

50% tests passed, 1 tests failed out of 2

Total Test time (real) =   0.02 sec

The following tests FAILED:
    171 - Solution_to_cpp_/home/david/Downloads/Optizelle-master/src/examples/simple_quadratic_cone/newton_cg.json (Failed)
Errors while running CTest

ctest -V -I 172,173,1

1/2 Test #172: Execution_of_cpp_/home/david/Downloads/Optizelle-master/src/examples/simple_quadratic_cone/newton_cg_backtracking.json ...   Passed    0.01 sec
test 173
    Start 173: Solution_to_cpp_/home/david/Downloads/Optizelle-master/src/examples/simple_quadratic_cone/newton_cg_backtracking.json

173: Test command: /home/david/Downloads/Optizelle-master/build/src/unit/utility/diff_restart "/home/david/Downloads/Optizelle-master/src/examples/simple_quadratic_cone/newton_cg_backtracking.json" "solution.json"
173: Test timeout computed to be: 9.99988e+06
173: Mismatch in Naturals.iter: 59 != 55.
2/2 Test #173: Solution_to_cpp_/home/david/Downloads/Optizelle-master/src/examples/simple_quadratic_cone/newton_cg_backtracking.json ....***Failed    0.00 sec

50% tests passed, 1 tests failed out of 2

Total Test time (real) =   0.03 sec

The following tests FAILED:
    173 - Solution_to_cpp_/home/david/Downloads/Optizelle-master/src/examples/simple_quadratic_cone/newton_cg_backtracking.json (Failed)
Errors while running CTest

Maybe I can take a look, but I lack the expertise to know how much different is significant enough. The results here seem fair enough, though.

@josyoun josyoun self-assigned this Aug 28, 2014
@josyoun
Copy link
Member

josyoun commented Aug 28, 2014

Yup. Just what I thought. Nothing is broken except for my testing mechanisms. I'll try to fix this with something clever.

@Dapid
Copy link
Contributor Author

Dapid commented Sep 5, 2014

Just tried on a nother platform, a modern i7 with same OS, and I get an extra failure in test 45, for polak_ribiere. Again, the failure is:

45: Mismatch in Naturals.iter: 4 != 2.

Just as an extra datapoint.

@josyoun
Copy link
Member

josyoun commented Sep 11, 2014

I've been trying to figure out a good way to fix this and I'm at a bit of a loss. Really, I have the iteration check in the unit tests in order to figure out when I've screwed up the algorithms with a change. However, this does appear to be slightly platform dependent. In any case, I'm hesistant to put in an iteration range rather than a fixed iteration because I really need to see when I've changed any of the algorithms. Really, I'd like to have two tests modes: neurotic, that checks the exact iteration, and normal, which uses a range. Maybe there's a way to trick ctest into doing this.

@Dapid
Copy link
Contributor Author

Dapid commented Sep 11, 2014

Python's nose has an option to make certain tests as slow, so they are not run by default, but they do on 'full' mode. If ctest has something similar, you could have two tests: one with a range, and another "neurotic" with fixed number and marked as slow.

@Dapid
Copy link
Contributor Author

Dapid commented Sep 11, 2014

I found a possible option:

http://www.mantidproject.org/Writing_Performance_Tests

Set the neurotic ones as performance tests.

@josyoun
Copy link
Member

josyoun commented Sep 11, 2014

Good call. I'll have a look at this.

@Dapid
Copy link
Contributor Author

Dapid commented Dec 9, 2015

Some update, on develop, on a Fedora 23 laptop with an i5, GCC 5.1.1, compilation flags -march=core-avx-i -O2 -pipe -mtune=native -std=c++14, I get the following errors due to mismatch in the number of iterations.

The following tests FAILED:
    230 - Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/newton_cg.json (Failed)
    232 - Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/newton_cg_backtracking.json (Failed)
    234 - Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/sr1.json (Failed)
    236 - Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/tr_newton.json (Failed)
    238 - Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/tr_sd.json (Failed)
$ ctest -V -I 230,238,2
UpdateCTestConfiguration  from :/home/david/gits/Optizelle/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/david/gits/Optizelle/build/DartConfiguration.tcl
Test project /home/david/gits/Optizelle/build
Constructing a list of tests
Done constructing a list of tests
Checking test dependency graph...
Checking test dependency graph end
test 230
    Start 230: Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/newton_cg.json

230: Test command: /home/david/gits/Optizelle/build/src/unit/utility/diff_restart "/home/david/gits/Optizelle/src/examples/inequality_scaling/newton_cg.json" "solution.json"
230: Test timeout computed to be: 9.99988e+06
230: Mismatch in Naturals.iter: 17 != 22.
1/5 Test #230: Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/newton_cg.json ................***Failed    0.01 sec
test 232
    Start 232: Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/newton_cg_backtracking.json

232: Test command: /home/david/gits/Optizelle/build/src/unit/utility/diff_restart "/home/david/gits/Optizelle/src/examples/inequality_scaling/newton_cg_backtracking.json" "solution.json"
232: Test timeout computed to be: 9.99988e+06
232: Mismatch in Naturals.iter: 18 != 22.
2/5 Test #232: Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/newton_cg_backtracking.json ...***Failed    0.00 sec
test 234
    Start 234: Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/sr1.json

234: Test command: /home/david/gits/Optizelle/build/src/unit/utility/diff_restart "/home/david/gits/Optizelle/src/examples/inequality_scaling/sr1.json" "solution.json"
234: Test timeout computed to be: 9.99988e+06
234: Mismatch in Naturals.iter: 19 != 22.
3/5 Test #234: Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/sr1.json ......................***Failed    0.00 sec
test 236
    Start 236: Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/tr_newton.json

236: Test command: /home/david/gits/Optizelle/build/src/unit/utility/diff_restart "/home/david/gits/Optizelle/src/examples/inequality_scaling/tr_newton.json" "solution.json"
236: Test timeout computed to be: 9.99988e+06
236: Mismatch in Naturals.iter: 18 != 22.
4/5 Test #236: Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/tr_newton.json ................***Failed    0.00 sec
test 238
    Start 238: Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/tr_sd.json

238: Test command: /home/david/gits/Optizelle/build/src/unit/utility/diff_restart "/home/david/gits/Optizelle/src/examples/inequality_scaling/tr_sd.json" "solution.json"
238: Test timeout computed to be: 9.99988e+06
238: Mismatch in Naturals.iter: 18 != 22.
5/5 Test #238: Solution_to_python_/home/david/gits/Optizelle/src/examples/inequality_scaling/tr_sd.json ....................***Failed    0.00 sec

0% tests passed, 5 tests failed out of 5

Total Test time (real) =   0.04 sec

@josyoun
Copy link
Member

josyoun commented Dec 10, 2015

This is a manifestation of the same problem, which will likely require the same fix. Basically, the inequality_scaling problems were designed to be bad problems numerically in order to test the new interior point method. Specifically, in this problem we solve:

min 0.5 || x - c ||^2 st x >= lb

and set c=[-1, ..., -1] and lb=[1, ..., 1]. Here x \in R^10. Then, I set x_i = 1+10^(-i+1). Basically, we have x=[2, 1.1, 1.01, 1.001, ..., 1+1e-9] which is a way to generate a solution that exists pretty close to the boundary on some variables, but not on others. Anyway, the optimal solution is [1, ... 1] and the test is whether or not the interior point algorithm will get stuck on the boundary or move back toward the central path and then be guided into the optimal solution. Thankfully, it does the right thing, but it's flaky enough numerically where things works slightly different between C++, Python, and Octave/MATLAB. More specifically, they have a varying number of iterations to converge. In truth, it'd be nice if they were all the same, but I suppose we can't expect that. As long as they're close, it's fine. Anyway, I need to figure out a trick for the CTest stuff to allow some amount of variation. Also, due to the numerical sensitivity, these answers will almost certainty change depending on the platform, so we probably need a solution similar to what is required above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants