Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation fault. Seems like error within ceres.solver #71

Closed
zeal-up opened this issue Sep 2, 2019 · 6 comments
Closed

Segmentation fault. Seems like error within ceres.solver #71

zeal-up opened this issue Sep 2, 2019 · 6 comments

Comments

@zeal-up
Copy link

zeal-up commented Sep 2, 2019 •
edited

Hi, there is a problem I can not solve, which I have tried two days.
The training process running as expected, but when eval the network, specifically, when calling the function evaluator.evaluate_uncertainty(), a 'Segmentation fault' error will arise. I use gdb to debug the code. It seems something error within the ceres solver. But I have try to recompile the ceres.
My gpu is RTX, so I use pytorch1.* as you recommend. And I can use all module in this repo except the uncertainty_pnp function.

I hope you can give me some advice.
The gbd information is as below:

 Thread 1 "python" received signal SIGSEGV, Segmentation fault.
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_set_length (__n=0, this=0x7fffffffd298)
    at /opt/conda/conda-bld/compilers_linux-64_1534514838838/work/.build/x86_64-conda_cos6-linux-gnu/build/build-cc-gcc-final/x86_64-conda_cos6-linux-gnu/libstdc++-v3/include/bits/basic_string.h:206
206	/opt/conda/conda-bld/compilers_linux-64_1534514838838/work/.build/x86_64-conda_cos6-linux-gnu/build/build-cc-gcc-final/x86_64-conda_cos6-linux-gnu/libstdc++-v3/include/bits/basic_string.h: No such file or directory.
(gdb)

The stack:

(gdb) bt
#0  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_set_length (__n=0, this=0x7fffffffd298)
    at /opt/conda/conda-bld/compilers_linux-64_1534514838838/work/.build/x86_64-conda_cos6-linux-gnu/build/build-cc-gcc-final/x86_64-conda_cos6-linux-gnu/libstdc++-v3/include/bits/basic_string.h:206
#1  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign (this=0x7fffffffd298, __str=...)
    at /opt/conda/conda-bld/compilers_linux-64_1534514838838/work/.build/x86_64-conda_cos6-linux-gnu/build/build-cc-gcc-final/x86_64-conda_cos6-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:274
#2  0x00007ffea05b014c in ceres::Solver::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) ()
   from /home/lab-lin.ziyao/work/my_pvnet/utils_c/extend_utils/lib/libceres.so.1
#3  0x00007ffea05b28b9 in ceres::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) ()
   from /home/lab-lin.ziyao/work/my_pvnet/utils_c/extend_utils/lib/libceres.so.1
#4  0x00007fffe1b963f6 in uncertainty_pnp () from /home/lab-lin.ziyao/work/my_pvnet/utils_c/extend_utils/_extend_utils.cpython-36m-x86_64-linux-gnu.so
#5  0x00007fffe1b93ef5 in _cffi_f_uncertainty_pnp (self=<optimized out>, args=<optimized out>) at _extend_utils.c:933
#6  0x0000555555665b91 in _PyCFunction_FastCallDict () at /tmp/build/80754af9/python_1564510748219/work/Objects/methodobject.c:234
#7  0x00005555556edabc in call_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4851
#8  0x000055555571075a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:3335
#9  0x00005555556e7c5b in _PyFunction_FastCall (globals=<optimized out>, nargs=4, args=<optimized out>, co=<optimized out>)
    at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4933
#10 fast_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4968
#11 0x00005555556edb95 in call_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4872
#12 0x000055555571075a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:3335
#13 0x00005555556e6e66 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4166
#14 0x00005555556e7ed6 in fast_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4992
#15 0x00005555556edb95 in call_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4872
#16 0x000055555571151c in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:3351
#17 0x00005555556e7c5b in _PyFunction_FastCall (globals=<optimized out>, nargs=3, args=<optimized out>, co=<optimized out>)
    at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4933
#18 fast_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4968
#19 0x00005555556edb95 in call_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4872
#20 0x000055555571075a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:3335
#21 0x00005555556e7c5b in _PyFunction_FastCall (globals=<optimized out>, nargs=0, args=<optimized out>, co=<optimized out>)
    at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4933
#22 fast_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4968
#23 0x00005555556edb95 in call_function () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4872
#24 0x000055555571075a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:3335
#25 0x00005555556e89b9 in _PyEval_EvalCodeWithName (qualname=0x0, name=0x0, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=<optimized out>, kwargs=0x0, kwnames=0x0, 
    argcount=0, args=0x0, locals=0x7ffff6e8e168, globals=0x7ffff6e8e168, _co=0x7ffff6d20e40) at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4166
#26 PyEval_EvalCodeEx () at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:4187
#27 0x00005555556e975c in PyEval_EvalCode (co=co@entry=0x7ffff6d20e40, globals=globals@entry=0x7ffff6e8e168, locals=locals@entry=0x7ffff6e8e168)
    at /tmp/build/80754af9/python_1564510748219/work/Python/ceval.c:731
#28 0x0000555555769744 in run_mod () at /tmp/build/80754af9/python_1564510748219/work/Python/pythonrun.c:1025
#29 0x0000555555769b41 in PyRun_FileExFlags () at /tmp/build/80754af9/python_1564510748219/work/Python/pythonrun.c:978
#30 0x0000555555769d43 in PyRun_SimpleFileExFlags () at /tmp/build/80754af9/python_1564510748219/work/Python/pythonrun.c:419
#31 0x0000555555769e4d in PyRun_AnyFileExFlags () at /tmp/build/80754af9/python_1564510748219/work/Python/pythonrun.c:81
#32 0x000055555576d833 in run_file (p_cf=0x7fffffffdfec, filename=0x5555558a82e0 L"main/train_linemod.py", fp=0x555555936030)
    at /tmp/build/80754af9/python_1564510748219/work/Modules/main.c:340
#33 Py_Main () at /tmp/build/80754af9/python_1564510748219/work/Modules/main.c:811
#34 0x000055555563788e in main () at /tmp/build/80754af9/python_1564510748219/work/Programs/python.c:69
#35 0x00007ffff7810830 in __libc_start_main (main=0x5555556377a0 <main>, argc=2, argv=0x7fffffffe1f8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7fffffffe1e8) at ../csu/libc-start.c:291
#36 0x00
@pengsida
Copy link
Member

pengsida commented Sep 2, 2019

What is your gcc version?
uncertainty-pnp is not related to pytorch but gcc.

@zeal-up
Copy link
Author

zeal-up commented Sep 2, 2019

My gcc version is 5.4.0.

@zeal-up
Copy link
Author

zeal-up commented Sep 2, 2019

I have another question. Why do you calculate scipy.linalg.sqrtm(covariances) as the weights. In the paper, the covariances are used to compute the uncertainty objective function directly.

@liuyuan-pal
Copy link

Hi, Zeal,
image

@zeal-up
Copy link
Author

zeal-up commented Sep 3, 2019

@liuyuan-pal Thanks a lot!

@zeal-up
Copy link
Author

zeal-up commented Sep 3, 2019

@pengsida I use scipy.optimize.leastsq to replace ceres. I think it will sacrifice some speed but the code will be simpler.

@zeal-up zeal-up closed this as completed Sep 3, 2019
@DeepLearningGPE DeepLearningGPE mentioned this issue Oct 16, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pengsida @liuyuan-pal @zeal-up