Very long lag about camera frame display #166
Comments
|
I am also having this issue. |
|
try to resize: |
|
I already try that solution, but that do not solve the problem. |
|
Yeah I have tried resizing too. The issue is that the |
|
@Aldarme where is the robotmaster API? It might help us isolate the issue if we can compare how they handle the video, compared to this library. |
|
Have you tried using the library straight from the latest master branch? Don't use the one from pypi or release as they are not the updated one. |
|
Yes I am using the latest on master. I am not installing it from pypi. I am using the latest with the updates that use |
|
@Aldarme thanks, I will take a look at their implementation and see if I can figure out what they do to get rid of the lag. |
|
Okay so I have a non lagging version. Most of the work is based off of the RoboMaster API. However I removed large amounts and made it work with my implementation. For anyone interested how is how you can get it to work: You first need a import time
import queue
import socket
import threading
class VideoConnection(object):
def __init__(self):
self._sock = None
self._sock_queue = queue.Queue(32)
self._sock_recv = None
self._recv_count = 0
self._receiving = False
def __del__(self):
if self._sock:
self._sock.close()
def connect(self, camera_ip: str="0.0.0.0", camera_port: int=11111):
try:
self._sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self._sock.bind((camera_ip, camera_port))
print("OVER")
except Exception as e:
print("StreamConnection: connect addr {0}:{1}, exception {2}".format(camera_ip, camera_port, e))
return False
self._sock_recv = threading.Thread(target=self._recv_task)
self._sock_recv.start()
print("StreamConnection {0} successfully!".format(camera_ip))
return True
def disconnect(self):
self._receiving = False
self._sock_queue.put(None)
if self._sock_recv:
self._sock_recv.join()
self._sock.close()
self._sock_queue.queue.clear()
self._recv_count = 0
print("StreamConnection: disconnected")
def _recv_task(self):
self._receiving = True
print("StreamConnection: _recv_task, Start to receiving Data...")
while self._receiving:
try:
if self._sock is None:
break
data, addr = self._sock.recvfrom(4096)
if not self._receiving:
break
self._recv_count += 1
if self._sock_queue.full():
print("StreamConnection: _recv_task, sock_data_queue is full.")
self._sock_queue.get()
else:
self._sock_queue.put(data)
except socket.timeout:
print("StreamConnection: _recv_task， recv data timeout!")
continue
except Exception as e:
print("StreamConnection: recv, exceptions:{0}".format(e))
self._receiving = False
return
def read_buf(self, timeout=2):
try:
buf = self._sock_queue.get(timeout=timeout)
return buf
except Exception as e:
print("StreamConnection: read_buf, exception {0}".format(e))
return NoneThen you need a import queue
import threading
import libmedia_codec
import numpy as np
from video_connection import VideoConnection
class VideoHandler(object):
def __init__(self, camera_ip: str="0.0.0.0", camera_port: int=11111):
# Init variables
self._video_frame_count = 0
self._video_streaming = False
# Create the video queue
self._video_frame_queue = queue.Queue(64)
# Create the video decoder
self._video_decoder = libmedia_codec.H264Decoder()
# Turn on the video converter
self._video_stream_conn = VideoConnection()
self._video_stream_conn.connect(camera_ip, camera_port)
self._video_decoder_thread = threading.Thread(target=self._video_decoder_task)
self._video_decoder_thread.start()
self._x = 0
def _h264_decode(self, data):
res_frame_list = []
frames = self._video_decoder.decode(data)
for frame_data in frames:
(frame, width, height, ls) = frame_data
if frame:
frame = np.fromstring(frame, dtype=np.ubyte, count=len(frame), sep='')
frame = (frame.reshape((height, width, 3)))
res_frame_list.append(frame)
return res_frame_list
def _video_decoder_task(self):
self._video_streaming = True
print("_video_decoder_task, started!")
while self._video_streaming:
data = b''
buf = self._video_stream_conn.read_buf()
if not self._video_streaming:
break
if buf:
data += buf
frames = self._h264_decode(data)
for frame in frames:
try:
self._video_frame_count += 1
if self._video_frame_count % 30 == 1:
print("video_decoder_task, get frame {0}.".format(self._video_frame_count))
self._video_frame_queue.put(frame, timeout=2)
except Exception as e:
print("_video_decoder_task, decoder queue is full, e {}.".format(e))
continue
print("_video_decoder_task, quit.")
def read_video_frame(self, timeout=3, strategy="newest"):
if strategy == "pipeline":
return self._video_frame_queue.get(timeout=timeout)
elif strategy == "newest":
while self._video_frame_queue.qsize() > 1:
self._video_frame_queue.get(timeout=timeout)
return self._video_frame_queue.get(timeout=timeout)
else:
print("read_video_frame, unsupported strategy:{0}".format(strategy))
return NoneYou then can call the python3 -m pip install .Best of luck. |
|
@hildebrandt-carl |
|
Sure I have moved away from using this library and started writing my own. However it shouldn't take me too long to create a pull request for this library. I will try get it done before the end of the weekend. |
|
Could you create that PR? @hildebrandt-carl |
|
Hello, sadly i cant build libmedia_codec,i get this error: Any Help would be appreciated! |
|
@steiraa it first glance the error message @hildebrandt-carl I would also greatly appreaciate a pull request with your video decoding code. I can merge it after testing, but I have to search for a working Tello battery first, they all seem to die after some weeks of not using/charging. |
|
@hildebrandt-carl How is the progress on the PR? |
|
Hi, sorry just before the holidays I managed to get Covid, and I am not in the office this week. I will be back on Jan 2nd and then get something out. |
|
This seems to be related and might be a problem: dji-sdk/RoboMaster-SDK#49 |
|
You have to install it manually. Copy the whole |
|
I have created a working version on my fork. I have also added an example which displays the video in real time on your screen using
I am not sure what the etiquette is in this case. Do I submit a pull request even though I know about these issues? Or do we wait and resolve these first? If we wait would you mind taking a look at my fork and seeing if you are able to run it? My fork can be found here. To get it running clone it. Install everything using: Then go to the examples and run: Note: The drones motors will go into a low speed mode to cool the drone, but the drone wont take off. A lag free video feed should appear on your screen, and 60 seconds later everything should shutdown. |
|
Tested on jetson nano and I an getting more lag with this |
|
@IEEERoboticsWSU that's very strange. I was running it on X86. However I have some ARM Odroids. I will give it a try on an Odroid this weekend and let you know what I find. However I run my own DJI custom library on the Odriods (which use the same camera code) and they ran fine on that. Is there any chance you have an X86 machine you can try it on? Also @Aldarme or @M4GNV5 have either of you had a chance to try it? |
|
@IEEERoboticsWSU what was the lag when using the original? When I used the original I used to get roughly every 30th frame. |
|
I do have a x86 machine however this is for a robotics so getting this to work on the jetson is a priority for me. |
|
@IEEERoboticsWSU a thought occurred. What type of Tello are you using? Are you using the Tello or the Tello Edu? When I used the Tello I found there was significantly more lag than the Tello Edu. Right now my results are for a Tello Edu. This weekend I can try both running my solution on Arm, and my solution on both the Tello and Tello Edu. |
|
I have the EDU |
|
also on Ubuntu 22.04 x64 with Python 3.8.16 |
|
I am unable to get video on desktop ubuntu but can get it on the jetson nano with significant lag. |
@M4GNV5, libmedia_codec needs to be made available (by DJI) for your specific version of python in pypi, if its not available you can use the code in the PR to compile it yourself |
Hi,
I'm using tello TT drone and i'm using "djitellopy2" to perform flight control on it.
However, I'm facing a very low fps rate when trying to just display real time stream frame from the camera drone.
To perform the display I'm using openCV and the exemple code to display stream frames.
Using:
"
from djitellopy import Tello
import cv2
myDrone = Tello()
myDrone.connect()
myDrone.streamon()
frameObj = self.myDrone.get_frame_read()
if frameObj.grabbed:
cv2.imshow("ARuco augmented", frameObj.frame)
cv2.waitKey(1)
"
Is there a known issue about this or not ?
Best regards,
The text was updated successfully, but these errors were encountered: