Ubuntu run time crash in malloc(): memory corruption #197
Comments
|
try to run it with "sudo": |
|
It crashes with a different stack trace: |
|
Added the #185 fix to OpenPose |
{{ message }}
Labels
help wanted/question
Extra attention is needed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue summary
When I run the examples, I get a malloc error shown below.
Executed command (if any)
./build/examples/openpose/openpose.bin --logging_level 0 --image_dir examples/media/
OpenPose output (if any)
...
Finished initialization on thread. In src/openpose/pose/poseRenderer.cpp:initializationOnThread():88
./include/openpose/thread/thread.hpp:threadFunction():185
./include/openpose/pose/wPoseExtractor.hpp:work():55
./include/openpose/producer/wDatumProducer.hpp:workProducer():53
./include/openpose/producer/wDatumProducer.hpp:workProducer():65
./include/openpose/core/wCvMatToOpInput.hpp:work():55
./include/openpose/core/wCvMatToOpInput.hpp:work():65
./include/openpose/core/wCvMatToOpOutput.hpp:work():55
./include/openpose/core/wCvMatToOpOutput.hpp:work():67
./include/openpose/pose/wPoseExtractor.hpp:work():70
./include/openpose/pose/wPoseRenderer.hpp:work():55
./include/openpose/pose/wPoseRenderer.hpp:work():65
./include/openpose/pose/wPoseExtractor.hpp:work():55
./include/openpose/producer/wDatumProducer.hpp:workProducer():53
./include/openpose/thread/wQueueOrderer.hpp:work():111
./include/openpose/core/wOpOutputToCvMat.hpp:work():54
./include/openpose/producer/wDatumProducer.hpp:workProducer():65
./include/openpose/core/wCvMatToOpInput.hpp:work():55
./include/openpose/core/wCvMatToOpInput.hpp:work():65
./include/openpose/core/wCvMatToOpOutput.hpp:work():55
./include/openpose/core/wCvMatToOpOutput.hpp:work():67
*** Error in `./build/examples/openpose/openpose.bin': malloc(): memory corruption (fast): 0x00007efd2c0009d0 ***
Stack trace in debugger:
#0 0x00007ffff6942428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff694402a in __GI_abort () at abort.c:89
#2 0x00007ffff69847ea in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7ffff6a9de98 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007ffff698d37a in malloc_printerr (ar_ptr=, ptr=,
str=0x7ffff6a9df10 "free(): invalid next size (fast)", action=3) at malloc.c:5006
#4 _int_free (av=, p=, have_lock=0) at malloc.c:3867
#5 0x00007ffff699153c in __GI___libc_free (mem=) at malloc.c:2968
#6 0x00007ffff78604eb in op::WKeypointScaler<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > >::work(std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >&) ()
from /home/user/nn/pose/openpose/.build_release/examples/openpose/../../lib/libopenpose.so.1.0.1
#7 0x00007ffff77abc0a in op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > >::checkAndWork(std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >&) ()
from /home/user/nn/pose/openpose/.build_release/examples/openpose/../../lib/libopenpose.so.1.0.1
#8 0x00007ffff77afaab in op::SubThread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >::workTWorkers(std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >&, bool) ()
from /home/user/nn/pose/openpose/.build_release/examples/openpose/../../lib/libopenpose.so.1.0.1
#9 0x00007ffff77b9087 in op::SubThreadQueueInOut<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > >, op::Queue<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::queue<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::deque<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::allocator<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > > > >::work() ()
from /home/user/nn/pose/openpose/.build_release/examples/openpose/../../lib/libopenpose.so.1.0.1
#10 0x00007ffff77b47eb in op::Thread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >::threadFunction() ()
from /home/user/nn/pose/openpose/.build_release/examples/openpose/../../lib/libopenpose.so.1.0.1
#11 0x00007ffff6fa5c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007fffeed5e6ba in start_thread (arg=0x7fffc32ef700) at pthread_create.c:333
#13 0x00007ffff6a143dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
I ran it using valgrind and got this crash. Seems like OpenCV has a memory error.
==8727== Thread 6:
==8727== Invalid write of size 1
==8727== at 0x517716E: op::floatPtrToUCharCvMat(cv::Mat&, float const*, op::Point const&, int) (openCv.cpp:45)
==8727== by 0x525C508: op::OpOutputToCvMat::formatToCvMat(op::Array const&) const (opOutputToCvMat.cpp:20)
==8727== by 0x52A634E: op::WOpOutputToCvMat<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > >::work(std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >&) (wOpOutputToCvMat.hpp:59)
==8727== by 0x51BD623: op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > >::checkAndWork(std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >&) (worker.hpp:68)
==8727== by 0x51B58B5: op::SubThread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >::workTWorkers(std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >&, bool) (subThread.hpp:82)
==8727== by 0x51B629A: op::SubThreadQueueInOut<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > >, op::Queue<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::queue<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::deque<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::allocator<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > > > >::work() (subThreadQueueInOut.hpp:66)
==8727== by 0x51B7B0D: op::Thread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >::threadFunction() (thread.hpp:190)
==8727== by 0x51D2828: void std::_Mem_fn_base<void (op::Thread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >::)(), true>::operator()<, void>(op::Thread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >) const (in /home/hmullings/nn/pose/openpose/.build_debug/lib/libopenpose.so.1.0.1)
==8727== by 0x51D27BC: void std::_Bind_simple<std::_Mem_fn<void (op::Thread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >::)()> (op::Thread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) (functional:1531)
==8727== by 0x51D256D: std::_Bind_simple<std::_Mem_fn<void (op::Thread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >::)()> (op::Thread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >)>::operator()() (functional:1520)
==8727== by 0x51D233D: std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (op::Thread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >::)()> (op::Thread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >)> >::_M_run() (thread:115)
==8727== by 0x6D91C7F: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==8727== Address 0x97eadc15 is 1 bytes after a block of size 84 alloc'd
==8727== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==8727== by 0x8BC9931: cv::fastMalloc(unsigned long) (in /usr/local/lib/libopencv_core.so.3.3.0)
==8727== by 0x8C66083: cv::Mat::create(int, int const*, int) (in /usr/local/lib/libopencv_core.so.3.3.0)
==8727== by 0x522F79E: cv::Mat::create(int, int, int) (mat.inl.hpp:784)
==8727== by 0x8C9212D: cv::_OutputArray::create(int, int, int, int, bool, int) const (in /usr/local/lib/libopencv_core.so.3.3.0)
==8727== by 0x8BBE8AB: cv::Mat::copyTo(cv::_OutputArray const&) const (in /usr/local/lib/libopencv_core.so.3.3.0)
==8727== by 0x51790CD: cv::Mat::Mat(std::initializer_list) (mat.inl.hpp:585)
==8727== by 0x5177017: op::floatPtrToUCharCvMat(cv::Mat&, float const*, op::Point const&, int) (openCv.cpp:34)
==8727== by 0x525C508: op::OpOutputToCvMat::formatToCvMat(op::Array const&) const (opOutputToCvMat.cpp:20)
==8727== by 0x52A634E: op::WOpOutputToCvMat<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > >::work(std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >&) (wOpOutputToCvMat.hpp:59)
==8727== by 0x51BD623: op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > >::checkAndWork(std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >&) (worker.hpp:68)
==8727== by 0x51B58B5: op::SubThread<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >, std::shared_ptr<op::Worker<std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > > > > >::workTWorkers(std::shared_ptr<std::vector<op::Datum, std::allocatorop::Datum > >&, bool) (subThread.hpp:82)
==8727==
--8727-- VALGRIND INTERNAL ERROR: Valgrind received a signal 11 (SIGSEGV) - exiting
--8727-- si_code=128; Faulting address: 0x0; sp: 0x813017e20
Type of issue
Your system configuration
Operating system (
lsb_release -ain Ubuntu):No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
CUDA version (
cat /usr/local/cuda/version.txtin most cases):CUDA Version 8.0.61
cuDNN version:
/usr/lib/libcudnn.so.5.1.10
GPU model (
nvidia-smiin Ubuntu):+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66 Driver Version: 375.66 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 0000:02:00.0 On | N/A |
| 48% 68C P2 64W / 250W | 3503MiB / 11156MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... Off | 0000:03:00.0 Off | N/A |
| 35% 51C P2 58W / 250W | 596MiB / 11172MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
Caffe version: Default from OpenPose or custom version.
The default included with the git repo.
OpenCV version: installed with
apt-get install libopencv-dev(Ubuntu) or default from OpenPose (Windows) or OpenCV 2.X or OpenCV 3.X.OpenCV 3.X
Generation mode (only for Ubuntu): Makefile + Makefile.config (default, Ubuntu) or CMake (Ubuntu, Windows) or Visual Studio (Windows).
cmake version 3.5.1
I ran ./ubuntu/install_caffe_and_openpose_if_cuda8.sh
diff --git a/ubuntu/Makefile.config.Ubuntu16_cuda8.example b/ubuntu/Makefile.config.Ubuntu16_cuda8.example
index c2940dc..352002a 100644
--- a/ubuntu/Makefile.config.Ubuntu16_cuda8.example
+++ b/ubuntu/Makefile.config.Ubuntu16_cuda8.example
@@ -18,7 +18,7 @@ USE_CUDNN := 1
# ALLOW_LMDB_NOLOCK := 1
# Uncomment if you're using OpenCV 3
-# OPENCV_VERSION := 3
+OPENCV_VERSION := 3
# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
@@ -92,7 +92,7 @@ BUILD_DIR := build
DISTRIBUTE_DIR := distribute
# Uncomment for debugging. Does not work on OSX due to BVLC/caffe#171
-# DEBUG := 1
+DEBUG := 1
# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0
Compiler (
gcc --versionin Ubuntu):gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
The text was updated successfully, but these errors were encountered: