crash after several seconds (malloc(): memory corruption)

[BRNKR]

Hey,

i am running this node on my Jetson TX2. After some seconds i get this crash error:

*** Error in `/home/nvidia/catkin_ws/devel/lib/darknet_ros/darknet_ros': malloc(): memory corruption: 0x00000001007389c0 ***

and sometimes this:

*** Error in `/home/nvidia/catkin_ws/devel/lib/darknet_ros/darknet_ros': free(): invalid next size (normal): 0x00000000006f7000 ***

any idea?

[pushkalkatara]

Hi, could you please set up AlexeyAB repo and run the same model without ROS code and check if the error still presists.

[pushkalkatara]

Also, are you running with Optical Flow enabled? Is the OpenCV3 package for ROS installed?

[pushkalkatara]

These errors basically occur in the case of memory overflow, writing to unallocated memory and reading it.
Also referencing this issue - AlexeyAB Issue - 1628#

[BRNKR]

@pushkalkatara thanks for your help. i will test it asap.

[DeepDuke]

Hi! Except for "malloc" and "free" problems, I also met

[darknet_ros-1] process has died [pid 13109], exit code -11

I am quite confused about this problem. I am running this package on tx2 with a Intel Realsense Camera D415. By the way, could you please tell me how can I run with Optical Flow enabled?

[pushkalkatara]

Hi @DeepDuke, exit code -11 basically occurs due to segmentation fault-invalid memory access.

• Did the node die immediately after roslaunch?
• Is the weight, config files pointed to the correct directory?

optical flow would work if you have opencv-contrib ros package installed.

[DeepDuke]

Hi@pushkalkatara The node died after running for about 3 minutes. I think I have put the weight and config files into the correct directory.

[pushkalkatara]

@DeepDuke if the node ran for 3 minutes with detection, then weights and config directory seems fine.
While running the node, use System Monitor to see Memory and Swap History. Please paste the results here if the memory usage increases with time or constant.
Try adding swap memory and re-running the node.

[DeepDuke]

@pushkalkatara I run the node on NVIDIA Jetson TX2. When i ran yolo3, the memory increased from 3.49G/7.67G to 4.50G/7.67G. It keeped at 4.50G/7.67G until the node died. The swap was always 0k/0k.
I also tried yolov3-tiny. When i run yolo3-tiny, the memory rised from 3.49G/7.67G to 3.72G/7.67G. When the ndoe died, the memory returned to 3.49G/7.67G. The swap still keepd unchanged at 0k/0k.

[DeepDuke]

@pushkalkatara BTW, yolo3-tiny could only run for less than a minute.

[pushkalkatara]

@DeepDuke how much time did it sustain 4.50G/7.67G until the node died?

[DeepDuke]

@pushkalkatara about 2-3 minutes for yolo3, less than a minute for yolo3-tiny

[NicolasBernard456]

Hello, not sure if related but i got a similar issue. As soon as Yolo detect an object and the tracking starts it crashes. GDB gave the following infos:

Thread 1 "darknet_ros" received signal SIGSEGV, Segmentation fault. darknet_ros::YoloROSTracker::trackThread (this=this@entry=0x7fffffffc130) at /home/nico/catkin_ws/src/darknet_ros/darknet_ros/src/YoloROSTracker.cpp:217 217 it->frames_counter = temp; (gdb) bt #0 darknet_ros::YoloROSTracker::trackThread (this=this@entry=0x7fffffffc130) at /home/nico/catkin_ws/src/darknet_ros/darknet_ros/src/YoloROSTracker.cpp:217 #1 0x00007ffff7822378 in darknet_ros::YoloROSTracker::YoloROSTracker (this=0x7fffffffc130, nh=...) at /home/nico/catkin_ws/src/darknet_ros/darknet_ros/src/YoloROSTracker.cpp:24 #2 0x00000000004058b7 in main (argc=1, argv=0x7fffffffc728) at /home/nico/catkin_ws/src/darknet_ros/darknet_ros/src/yolo_object_detector_node.cpp:15

[pushkalkatara]

Hi @NicolasBernard456 . Here are some steps for debugging -

• Using which ROS package are you reading the camera feed?
  Please use video_stream_opencv package for the same.
• Is the cfg file correct?

[Shame-fight]

@pushkalkatara
Thanks for the work you did, the code doesn't work on my computer and TX2. When I run the code "roslaunch darknet_ros darknet_ros.launch", the result is as follows. And there is no detection window, just the result under the terminal.
[ INFO] [1571276131.857562415]: Published
[ INFO] [1571276131.857801241]: Published
[ INFO] [1571276131.857998518]: Started darknet thread
[ INFO] [1571276131.858011827]: Reference Count > 0
[ INFO] [1571276131.858044404]: Published
[ INFO] [1571276131.858184366]: Inside tracking
[darknet_ros-1] process has died [pid 10361, exit code -11, cmd /home/jgx/catkin_ws/devel/lib/darknet_ros/darknet_ros __name:=darknet_ros __log:=/home/jgx/.ros/log/05036024-f07e-11e9-8718-983f9f190824/darknet_ros-1.log].
log file: /home/jgx/.ros/log/05036024-f07e-11e9-8718-983f9f190824/darknet_ros-1*.log
all processes on machine have died, roslaunch will exit
shutting down processing monitor...
... shutting down processing monitor complete
done

I found that the program will be killed immediately if the camera moves.I use the usb_cam package.I want to know how can solve this problems.thanks

[Shame-fight]

您好，不确定是否相关，但我遇到了类似的问题。一旦Yolo检测到物体并开始跟踪，它就会崩溃。GDB提供了以下信息：

Thread 1 "darknet_ros" received signal SIGSEGV, Segmentation fault. darknet_ros::YoloROSTracker::trackThread (this=this@entry=0x7fffffffc130) at /home/nico/catkin_ws/src/darknet_ros/darknet_ros/src/YoloROSTracker.cpp:217 217 it->frames_counter = temp; (gdb) bt #0 darknet_ros::YoloROSTracker::trackThread (this=this@entry=0x7fffffffc130) at /home/nico/catkin_ws/src/darknet_ros/darknet_ros/src/YoloROSTracker.cpp:217 #1 0x00007ffff7822378 in darknet_ros::YoloROSTracker::YoloROSTracker (this=0x7fffffffc130, nh=...) at /home/nico/catkin_ws/src/darknet_ros/darknet_ros/src/YoloROSTracker.cpp:24 #2 0x00000000004058b7 in main (argc=1, argv=0x7fffffffc728) at /home/nico/catkin_ws/src/darknet_ros/darknet_ros/src/yolo_object_detector_node.cpp:15

I have the same problems with you,did you have solve it? As soon as Yolo detect an object it crashes:[darknet_ros-1] process has died [pid 4547, exit code -11...