PoseNet Crashing

[nihalraman]

Description

Hi! We're interested in running PoseNet to track physical marker points continuously (or at least for long periods of time). When we run simple_pose.py, the coral device consistently crashes after running for 2-5 minutes. How can we fix this problem? Thanks for the help!

Click to expand!

Issue Type

Performance

Operating System

Mendel Linux

Coral Device

Dev Board

Other Devices

No response

Programming Language

No response

Relevant Log Output

No response

[hjonnala]

Hi @nihalraman I just ran the simple_pose.py about 30 minutes and it is working as expected.

• Are you using the same model in simple_pose.py?
• Can you please provide specific steps to reproduce the crashing scenario.
• Also, please share the log messages when posenet crashes.

Thanks

[nihalraman]

Hi @hjonnala,
Thanks for the response! Sorry, I meant pose_camera.py. We are just running the program ("python3 pose_camera.py"). I'm not sure about the log messages--the device just restarts (i.e. connected monitor goes black before we see it booting up).

[nihalraman]

Also, the crashes happened primarily while we ran a modified version of pose_camera.py that saves the marker points to a text file, but the device crashed when we ran pose_camera.py itself, too.

[hjonnala]

I am able to run the pose_camera.py more than 15 mins without crashing. Can you please share the modified version of pose_camera.py. Thanks!

[nihalraman]

Hi! I don't have access to the script right now, but it's basically just pose_camera.py moified to write landmark coordinates to a separate .txt file. Do you have any general troubleshooting tips?

[hjonnala]

I am not sure the following things fix the issue as I don't the exact reason why it is crashing. Please try the following things:

• Re flash the board using sd card
• Try with another dev board
• Connect the dev board to host machine with USB cable to avoid socket timeout (Socket timeout might happen if we connect the dev board over wifi without any USB OTG connection to the HOST Machine)

[nihalraman]

So pose_camera.py runs without issue. Here's our modified script. It runs for variable lengths of time (anywhere from 30 seconds to a few minutes with a person continuously within the camera frame) before crashing, so it doesn't seem like it's a memory issue. I'm not sure where we're going wrong....

Thanks!

save_pose_camera.txt

[hjonnala]

Hi @nihalraman save_pose_camera.py running fine for me without any issues for long time.

I am only able to produce the no memory left on the device scenario, even that happens the dev board is freezing without any detection on HDMI Display.

mendel@orange-horse:~/project-posenet$ python3 save_pose_camera.py 
/home/mendel/project-posenet/gstreamer.py:15: PyGIWarning: Gst was imported without specifying a version first. Use gi.require_version('Gst', '1.0') before import to ensure that the right version gets loaded.
  from gi.repository import GLib, GObject, Gst, GstBase, GstVideo, Gtk
/home/mendel/project-posenet/gstreamer.py:15: PyGIWarning: GstBase was imported without specifying a version first. Use gi.require_version('GstBase', '1.0') before import to ensure that the right version gets loaded.
  from gi.repository import GLib, GObject, Gst, GstBase, GstVideo, Gtk
/home/mendel/project-posenet/gstreamer.py:15: PyGIWarning: GstVideo was imported without specifying a version first. Use gi.require_version('GstVideo', '1.0') before import to ensure that the right version gets loaded.
  from gi.repository import GLib, GObject, Gst, GstBase, GstVideo, Gtk
/home/mendel/project-posenet/gstreamer.py:15: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version('Gtk', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import GLib, GObject, Gst, GstBase, GstVideo, Gtk
Loading model:  models/mobilenet/posenet_mobilenet_v1_075_481_641_quant_decoder_edgetpu.tflite
Gstreamer pipeline:  v4l2src device=/dev/video1 ! video/x-raw,width=640,height=480,framerate=30/1 ! decodebin ! videoflip video-direction=identity ! tee name=t
               t. ! queue max-size-buffers=1 leaky=downstream ! videoconvert ! freezer name=freezer ! rsvgoverlay name=overlay
                  ! videoconvert ! autovideosink
               t. ! queue max-size-buffers=1 leaky=downstream ! videoconvert ! videoscale ! video/x-raw,width=641,height=480 ! videobox name=box autocrop=true
                  ! video/x-raw,format=RGB,width=641,height=481 ! appsink name=appsink emit-signals=true max-buffers=1 drop=true
            
Exception in thread Thread-2:
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/home/mendel/project-posenet/gstreamer.py", line 191, in render_loop
    svg, freeze = self.render_callback(output, self.src_size, self.get_box())
  File "save_pose_camera.py", line 157, in render_overlay
    outfile.write(sv_pose_str+'\n')
OSError: [Errno 28] No space left on device

[hjonnala]

Closing this issue as it is not reproducible. Thanks!

[google-coral-bot]

Are you satisfied with the resolution of your issue?
Yes
No