make test failed on tx2

[justwillim]

I'm trying to use ceres on nvidia TX2, it seems like test 10 failed on it.
I have tried it on TX1 and there's no issue.

Any thought about it?
Thanks

Test log are outlined below,

Test project /home/nvidia/ceres-solver/build
Start 1: array_utils_test
1/72 Test #1: array_utils_test ............................ Passed 0.01 sec
Start 2: autodiff_test
2/72 Test #2: autodiff_test ............................... Passed 0.01 sec
Start 3: autodiff_cost_function_test
3/72 Test #3: autodiff_cost_function_test ................. Passed 0.01 sec
Start 4: autodiff_local_parameterization_test
4/72 Test #4: autodiff_local_parameterization_test ........ Passed 0.01 sec
Start 5: block_jacobi_preconditioner_test
5/72 Test #5: block_jacobi_preconditioner_test ............ Passed 0.02 sec
Start 6: block_random_access_dense_matrix_test
6/72 Test #6: block_random_access_dense_matrix_test ....... Passed 0.01 sec
Start 7: block_random_access_diagonal_matrix_test
7/72 Test #7: block_random_access_diagonal_matrix_test .... Passed 0.01 sec
Start 8: block_random_access_sparse_matrix_test
8/72 Test #8: block_random_access_sparse_matrix_test ...... Passed 0.01 sec
Start 9: block_sparse_matrix_test
9/72 Test #9: block_sparse_matrix_test .................... Passed 0.01 sec
Start 10: bundle_adjustment_test
10/72 Test #10: bundle_adjustment_test ......................***Exception: Other1789.51 sec
Start 11: c_api_test
11/72 Test #11: c_api_test .................................. Passed 0.12 sec
Start 12: canonical_views_clustering_test
12/72 Test #12: canonical_views_clustering_test ............. Passed 0.01 sec
Start 13: compressed_row_sparse_matrix_test
13/72 Test #13: compressed_row_sparse_matrix_test ........... Passed 0.17 sec
Start 14: conditioned_cost_function_test
14/72 Test #14: conditioned_cost_function_test .............. Passed 0.01 sec
Start 15: corrector_test
15/72 Test #15: corrector_test .............................. Passed 9.46 sec
Start 16: cost_function_to_functor_test
16/72 Test #16: cost_function_to_functor_test ............... Passed 0.01 sec
Start 17: covariance_test
17/72 Test #17: covariance_test ............................. Passed 34.24 sec
Start 18: cubic_interpolation_test
18/72 Test #18: cubic_interpolation_test .................... Passed 0.21 sec
Start 19: detect_structure_test
19/72 Test #19: detect_structure_test ....................... Passed 0.01 sec
Start 20: dense_sparse_matrix_test
20/72 Test #20: dense_sparse_matrix_test .................... Passed 0.01 sec
Start 21: dynamic_autodiff_cost_function_test
21/72 Test #21: dynamic_autodiff_cost_function_test ......... Passed 0.01 sec
Start 22: dynamic_compressed_row_sparse_matrix_test
22/72 Test #22: dynamic_compressed_row_sparse_matrix_test ... Passed 0.01 sec
Start 23: dynamic_numeric_diff_cost_function_test
23/72 Test #23: dynamic_numeric_diff_cost_function_test ..... Passed 0.01 sec
Start 24: dynamic_sparsity_test
24/72 Test #24: dynamic_sparsity_test ....................... Passed 11.41 sec
Start 25: evaluator_test
25/72 Test #25: evaluator_test .............................. Passed 0.04 sec
Start 26: gradient_checker_test
26/72 Test #26: gradient_checker_test ....................... Passed 0.04 sec
Start 27: gradient_checking_cost_function_test
27/72 Test #27: gradient_checking_cost_function_test ........ Passed 0.02 sec
Start 28: gradient_problem_test
28/72 Test #28: gradient_problem_test ....................... Passed 0.01 sec
Start 29: gradient_problem_solver_test
29/72 Test #29: gradient_problem_solver_test ................ Passed 0.03 sec
Start 30: graph_test
30/72 Test #30: graph_test .................................. Passed 1.53 sec
Start 31: graph_algorithms_test
31/72 Test #31: graph_algorithms_test ....................... Passed 0.01 sec
Start 32: householder_vector_test
32/72 Test #32: householder_vector_test ..................... Passed 0.01 sec
Start 33: is_close_test
33/72 Test #33: is_close_test ............................... Passed 0.01 sec
Start 34: invert_psd_matrix_test
34/72 Test #34: invert_psd_matrix_test ...................... Passed 0.01 sec
Start 35: implicit_schur_complement_test
35/72 Test #35: implicit_schur_complement_test .............. Passed 0.02 sec
Start 36: iterative_schur_complement_solver_test
36/72 Test #36: iterative_schur_complement_solver_test ...... Passed 0.03 sec
Start 37: jet_test
37/72 Test #37: jet_test .................................... Passed 0.01 sec
Start 38: levenberg_marquardt_strategy_test
38/72 Test #38: levenberg_marquardt_strategy_test ........... Passed 0.03 sec
Start 39: dogleg_strategy_test
39/72 Test #39: dogleg_strategy_test ........................ Passed 0.03 sec
Start 40: line_search_minimizer_test
40/72 Test #40: line_search_minimizer_test .................. Passed 0.03 sec
Start 41: line_search_preprocessor_test
41/72 Test #41: line_search_preprocessor_test ............... Passed 0.03 sec
Start 42: local_parameterization_test
42/72 Test #42: local_parameterization_test ................. Passed 2.05 sec
Start 43: loss_function_test
43/72 Test #43: loss_function_test .......................... Passed 0.01 sec
Start 44: minimizer_test
44/72 Test #44: minimizer_test .............................. Passed 0.03 sec
Start 45: normal_prior_test
45/72 Test #45: normal_prior_test ........................... Passed 0.01 sec
Start 46: numeric_diff_cost_function_test
46/72 Test #46: numeric_diff_cost_function_test ............. Passed 0.01 sec
Start 47: ordered_groups_test
47/72 Test #47: ordered_groups_test ......................... Passed 0.52 sec
Start 48: parameter_block_test
48/72 Test #48: parameter_block_test ........................ Passed 2.59 sec
Start 49: parameter_block_ordering_test
49/72 Test #49: parameter_block_ordering_test ............... Passed 0.01 sec
Start 50: partitioned_matrix_view_test
50/72 Test #50: partitioned_matrix_view_test ................ Passed 0.01 sec
Start 51: polynomial_test
51/72 Test #51: polynomial_test ............................. Passed 0.01 sec
Start 52: problem_test
52/72 Test #52: problem_test ................................ Passed 14.29 sec
Start 53: program_test
53/72 Test #53: program_test ................................ Passed 0.02 sec
Start 54: reorder_program_test
54/72 Test #54: reorder_program_test ........................ Passed 0.02 sec
Start 55: residual_block_test
55/72 Test #55: residual_block_test ......................... Passed 0.02 sec
Start 56: residual_block_utils_test
56/72 Test #56: residual_block_utils_test ................... Passed 0.01 sec
Start 57: rotation_test
57/72 Test #57: rotation_test ............................... Passed 48.38 sec
Start 58: schur_complement_solver_test
58/72 Test #58: schur_complement_solver_test ................ Passed 0.03 sec
Start 59: schur_eliminator_test
59/72 Test #59: schur_eliminator_test ....................... Passed 0.02 sec
Start 60: single_linkage_clustering_test
60/72 Test #60: single_linkage_clustering_test .............. Passed 0.01 sec
Start 61: small_blas_test
61/72 Test #61: small_blas_test ............................. Passed 0.02 sec
Start 62: solver_test
62/72 Test #62: solver_test ................................. Passed 0.03 sec
Start 63: sparse_cholesky_test
63/72 Test #63: sparse_cholesky_test ........................ Passed 0.22 sec
Start 64: compressed_col_sparse_matrix_utils_test
64/72 Test #64: compressed_col_sparse_matrix_utils_test ..... Passed 0.02 sec
Start 65: symmetric_linear_solver_test
65/72 Test #65: symmetric_linear_solver_test ................ Passed 0.02 sec
Start 66: triplet_sparse_matrix_test
66/72 Test #66: triplet_sparse_matrix_test .................. Passed 0.53 sec
Start 67: trust_region_minimizer_test
67/72 Test #67: trust_region_minimizer_test ................. Passed 0.04 sec
Start 68: trust_region_preprocessor_test
68/72 Test #68: trust_region_preprocessor_test .............. Passed 0.03 sec
Start 69: unsymmetric_linear_solver_test
69/72 Test #69: unsymmetric_linear_solver_test .............. Passed 0.03 sec
Start 70: visibility_test
70/72 Test #70: visibility_test ............................. Passed 0.01 sec
Start 71: visibility_based_preconditioner_test
71/72 Test #71: visibility_based_preconditioner_test ........ Passed 0.01 sec
Start 72: system_test
72/72 Test #72: system_test ................................. Passed 0.04 sec

99% tests passed, 1 tests failed out of 72

Total Test time (real) = 1947.94 sec

[sandwichmaker]

@justwillim I do not have access to a tx2, can you share more detailed logs from the bundle adjustment test? did it just time out? can you run it from the command line.

something like

./bin/bundle_adjustment_test --test_srcdir=../data/ -logtostderr

[justwillim]

there too much print while I run this test independently, I clipped a part of them below:

final_residuals[i] evaluates to 0.95460845306081998,
expected_final_residuals_[i] evaluates to 1.1131732224770092, and
SystemTestProblem::kResidualTolerance evaluates to 0.0001.
Not close enough residual:37966
/home/nvidia/ceres/ceres-solver-1.12.0/internal/ceres/test_util.h:165: Failure
The difference between final_residuals[i] and expected_final_residuals_[i] is 0.11338762091053997, which exceeds SystemTestProblem::kResidualTolerance, where
final_residuals[i] evaluates to 0.011416297728878842,
expected_final_residuals_[i] evaluates to 0.12480391863941881, and
SystemTestProblem::kResidualTolerance evaluates to 0.0001.
Not close enough residual:37967
/home/nvidia/ceres/ceres-solver-1.12.0/internal/ceres/test_util.h:165: Failure
The difference between final_residuals[i] and expected_final_residuals_[i] is 0.085862200742258032, which exceeds SystemTestProblem::kResidualTolerance, where
final_residuals[i] evaluates to 0.42473861086730247,
expected_final_residuals_[i] evaluates to 0.33887641012504444, and
SystemTestProblem::kResidualTolerance evaluates to 0.0001.
Not close enough residual:37968
/home/nvidia/ceres/ceres-solver-1.12.0/internal/ceres/test_util.h:165: Failure
The difference between final_residuals[i] and expected_final_residuals_[i] is 0.078454742810264669, which exceeds SystemTestProblem::kResidualTolerance, where
final_residuals[i] evaluates to 0.092429569755572746,
expected_final_residuals_[i] evaluates to 0.17088431256583742, and
SystemTestProblem::kResidualTolerance evaluates to 0.0001.
Not close enough residual:37969
/home/nvidia/ceres/ceres-solver-1.12.0/internal/ceres/test_util.h:165: Failure
The difference between final_residuals[i] and expected_final_residuals_[i] is 0.028474610035872705, which exceeds SystemTestProblem::kResidualTolerance, where

[justwillim]

also, using the latest release version have the same issue

[sandwichmaker]

This seems like a convergence issue. I am not sure what math libraries you are using, and what is the underlying architecture that is being used. But my guess is that because of linear algebra libraries like eigen are doing something funny on this CPU? do you know if eigen for example passes all its tests on this platform?

[justwillim]

It passed all test after changed the eigen to latest release 3.3.3, although there are still 7 test in eigen. thank you for your light of this issue very much

[sandwichmaker]

@justwillim you are welcome, I am glad I was able to help.