D435i Low Framerate when using 2 or more USB-Cameras

[Starblazer75]

Required Info
Camera Model	D435i
Firmware Version	5.15.1
Operating System & Version	Ubuntu 22.04
Kernel Version (Linux Only)	5.15.122-tegra
Platform	Jetson Orin Nano
SDK Version	Source installed a week ago
Language	Python
Segment	Robot

Realsense Code:

import socket
import numpy as np
import pyrealsense2 as rs
from cv_bridge import CvBridge
import signal
import time

def initialize_camera():

	camera_pipeline = rs.pipeline()
	camera_config = rs.config()

	#camera_config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
	camera_config.enable_stream(rs.stream.color, 640, 480, rs.format.rgb8, 30)

	#camera_pipeline.start(camera_config)
	camera_pipeline.start()

	return camera_pipeline

def timeout_handler(signum, frame):
	raise TimeoutError()

def main():

	signal.signal(signal.SIGALRM, timeout_handler)

	camera_pipeline = initialize_camera()

	server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

	server_address = ('192.168.1.11', 12345)

	print("Starting UDP Server")

	server_socket.bind(server_address)

	bridge = CvBridge()

	while True:
		signal.alarm(0)

		frames = camera_pipeline.wait_for_frames()

		try:
			signal.alarm(1)

			color_frame = frames.get_color_frame()

			image_data = np.asarray(color_frame.get_data(), dtype=np.int32)

			compressed_data = bridge.cv2_to_compressed_imgmsg(image_data, 'jpg').data

			image_data = np.asarray(compressed_data, dtype=np.uint8)

			split_data = np.array_split(image_data, 3)

			data, address = server_socket.recvfrom(4096)

			server_socket.sendto(split_data[0].tobytes(), address)
			server_socket.recvfrom(4096)

			server_socket.sendto(split_data[1].tobytes(), address)
			server_socket.recvfrom(4096)

			server_socket.sendto(split_data[2].tobytes(), address)
			server_socket.recvfrom(4096)
			print("Sent")

		except TimeoutError:
			print("Timeout")
			server_socket.close()
			server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
			server_address = ('192.168.1.11', 12345)
			server_socket.bind(server_address)

			signal.alarm(0)
			continue


main()

USB-Camera Code:

import cv2
import socket
import numpy as np
import pyrealsense2 as rs
from cv_bridge import CvBridge
import signal
import time
import sys

import rclpy
from rclpy.node import Node

class CameraPublisher(Node):
	
	def __init__(self, video, port):
		super().__init__(f'camera_{video}')

		signal.signal(signal.SIGALRM, self.timeout_handler)

		self.server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
		server_address = ('192.168.1.11', port)
		print("Starting UDP Server")
		self.server_socket.bind(server_address)

		self.bridge = CvBridge()
		self.cap = cv2.VideoCapture(video)
		if not self.cap.isOpened():
			print("Could not open video device")
			sys.exit()
		else:
			self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
			self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)

		timer_period = 1/30
		self.timer = self.create_timer(timer_period, self.timer_callback)

	def timeout_handler(self, signum, frame):
		raise TimeoutError()
	
	def timer_callback(self):
		signal.alarm(1)

		try:
			ret, frame = self.cap.read()

			if not ret:
				print("Cant receive Frame")
				return

			compressed_data = self.bridge.cv2_to_compressed_imgmsg(frame, 'jpg').data

			image_data = np.asarray(compressed_data, dtype=np.uint8)

			split_data = np.array_split(image_data, 3)

			data, address = self.server_socket.recvfrom(4096)

			self.server_socket.sendto(split_data[0].tobytes(), address)
			self.server_socket.recvfrom(4096)

			self.server_socket.sendto(split_data[1].tobytes(), address)
			self.server_socket.recvfrom(4096)

			self.server_socket.sendto(split_data[2].tobytes(), address)
			self.server_socket.recvfrom(4096)
			print("Sent")

		except TimeoutError:
			print("Timeout")
			signal.alarm(0)
			time.sleep(0.75)
			return


def main(args=None):

	video = int(sys.argv[1])

	port = int(sys.argv[2])

	try:
		assert video != None and port != None
	except AssertionError:
		print("Usgae: python3 camera.py <video> <port>")
		sys.exit()

	rclpy.init(args=args)

	camera_publisher = CameraPublisher(video, port)

	try:
		rclpy.spin(camera_publisher)
	except KeyboardInterrupt:
		camera_publisher.cap.release()
		camera_publisher.destroy_node()

if __name__ == '__main__':
	main()

These are the two scripts that I am running to get camera feed. All of these work perfectly when going individually.

When I am running the Realsense code, it works until I run an instance of the USB-camera code. If I run 1 instance, the framerate is completely fine but when I run the second instance, the framerate is extremely inconsistent and never goes above 5Hz. If I start the pipeline to include config, then the code will hang until the 2nd instance of the USB-Camera code is interrupted. This causes it to throw the error of waiting 5000 milliseconds in wait_for_frames(), but if I change to poll_for_frames it will continue properly if the 2nd instance of USB-Camera code is stopped.

When putting the video address of the realsense camera in USB-Camera code, it also works perfectly fine until the 2nd instance of the other cameras stop. Then, the realsense code will hang.

An important note is that it doesn't matter which camera I stop for the realsense code to continue working. I have concluded that it is something going wrong with the realsense camera.

The USB-Camera code is limited to 30Hz, and the Realsense code is limited to whatever wait_for_frames() sets. 30Hz if working properly.

When trying to initialize it on the USB-Camera code after both the other cameras are working, it throws this error after doing a Keyboard Interrupt (It hangs until I interrupt or stop one of the processes):

---

Starting UDP Server
[ WARN:0] global ./modules/videoio/src/cap_gstreamer.cpp (1100) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
^C
(python3:26952): GStreamer-CRITICAL **: 19:00:17.083:
Trying to dispose element videoconvert0, but it is in PLAYING instead of the NULL state.
You need to explicitly set elements to the NULL state before
dropping the final reference, to allow them to clean up.
This problem may also be caused by a refcounting bug in the
application or some element.

(python3:26952): GStreamer-CRITICAL **: 19:00:17.083:
Trying to dispose element pipeline0, but it is in PAUSED instead of the NULL state.
You need to explicitly set elements to the NULL state before
dropping the final reference, to allow them to clean up.
This problem may also be caused by a refcounting bug in the
application or some element.

[ WARN:0] global ./modules/videoio/src/cap_gstreamer.cpp (651) startPipeline OpenCV | GStreamer warning: unable to start pipeline
Traceback (most recent call last):
File "/home/billee/billee_ws/src/sensors/sensors/camera.py", line 98, in
main()
File "/home/billee/billee_ws/src/sensors/sensors/camera.py", line 89, in main
camera_publisher = CameraPublisher(video, port)
File "/home/billee/billee_ws/src/sensors/sensors/camera.py", line 31, in init
self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
KeyboardInterrupt
[ WARN:0] global ./modules/videoio/src/cap_gstreamer.cpp (616) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

(python3:26952): GStreamer-CRITICAL **: 19:00:17.136:
Trying to dispose element appsink0, but it is in READY instead of the NULL state.
You need to explicitly set elements to the NULL state before
dropping the final reference, to allow them to clean up.
This problem may also be caused by a refcounting bug in the
application or some element.

---

The first warning happens for all of them, doesnt seem to change anything. I can't find the solution for this, any help is appreciated.

[MartyG-RealSense]

Hi @Starblazer75 The impression that I get from the code is that when there are two cameras attached the script cannot distinguish between them as separate RealSense and non-RealSense devices.

The RealSense SDK is able to access RealSense and non-RealSense cameras simultaneously if the non-RealSense camera is designated as a Platform Camera. You can test this in the RealSense Viewer tool when both cameras are attached by clicking the Add Source button at the top of the Viewer's options side-panel and selecting the USB camera from a menu drop-down.

If the USB camera is successfully recognized by Add Source then you can display the RGB streams from the RealSense camera and the non-RealSense camera side by side.

There are not many examples of implementing Platform Camera recognition in Python though unfortunately, other than some code from a pyrealsense2 device handling script called realsense_device_manager.py that determines whether or not the camera is a Platform Camera by reading the identification name of the camera to see whether it contains the words 'platform camera'.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/box_dimensioner_multicam/realsense_device_manager.py#L45

[Starblazer75]

When I try turning on any stream of the Platform Camera on realsense-viewer, it gives this error:

30/04 15:28:48,689 INFO [129723605150912] (synthetic-stream-gl.cpp:80) Initializing rendering, GLSL=0
30/04 15:28:48,689 INFO [129723605150912] (synthetic-stream-gl.cpp:89) 0 GPU objects initialized
30/04 15:28:48,729 INFO [129723605150912] (context.cpp:336) Found 2 RealSense devices (mask 0xff)
30/04 15:28:48,746 INFO [129723605150912] (rs.cpp:2697) Framebuffer size changed to 1850 x 1016
30/04 15:28:48,746 INFO [129723605150912] (rs.cpp:2697) Window size changed to 1850 x 1016
30/04 15:28:48,746 INFO [129723605150912] (rs.cpp:2697) Scale Factor is now 1
30/04 15:28:49,040 INFO [129723605150912] (context.cpp:336) Found 1 RealSense devices (mask 0xfe)
30/04 15:28:49,775 WARNING [129722531096128] (sensor.cpp:1291) Failed to add Backlight Compensation control for RGB Camera
30/04 15:28:49,783 WARNING [129722531096128] (sensor.cpp:1291) Failed to add Brightness control for RGB Camera
30/04 15:28:49,801 WARNING [129722531096128] (sensor.cpp:1291) Failed to add Contrast control for RGB Camera
30/04 15:28:49,821 WARNING [129722080020032] (messenger-libusb.cpp:42) control_transfer returned error, index: 100, error: Success, number: 0
30/04 15:28:49,829 WARNING [129722531096128] (sensor.cpp:1291) Failed to add Exposure control for RGB Camera
30/04 15:28:49,848 WARNING [129722531096128] (sensor.cpp:1291) Failed to add Gamma control for RGB Camera
30/04 15:28:49,866 WARNING [129722531096128] (sensor.cpp:1291) Failed to add Hue control for RGB Camera
30/04 15:28:49,885 WARNING [129722531096128] (sensor.cpp:1291) Failed to add Saturation control for RGB Camera
30/04 15:28:49,893 WARNING [129722531096128] (sensor.cpp:1291) Failed to add Sharpness control for RGB Camera
30/04 15:28:49,911 WARNING [129722531096128] (sensor.cpp:1291) Failed to add White Balance control for RGB Camera
30/04 15:28:49,912 WARNING [129722080020032] (messenger-libusb.cpp:42) control_transfer returned error, index: 100, error: Success, number: 0
30/04 15:28:49,930 WARNING [129722531096128] (sensor.cpp:1291) Failed to add Enable Auto Exposure control for RGB Camera
30/04 15:28:49,931 WARNING [129722080020032] (messenger-libusb.cpp:42) control_transfer returned error, index: 200, error: Success, number: 0
30/04 15:28:49,949 WARNING [129722531096128] (sensor.cpp:1291) Failed to add Enable Auto White Balance control for RGB Camera
30/04 15:28:53,377 WARNING [129723605150912] (device-model.cpp:437) Exception caught trying to detect presets: cannot parse device name from 'Platform Camera'
30/04 15:29:06,085 INFO [129723605150912] (sensor.cpp:1594) Request: YUYV Color,
Resolved to: YUYV Color,
Segmentation fault

A side note is that I am now using an x86 computer with Ubuntu 22.04, and when running realsense-viewer without sud o permissions, it gives an error that says it failed to set power state. This was a common problem for me, so I just gave the librealsense source file root permissions for it to keep working. Is this a potential problem?

I will keep working on a solution and let you know if I find anything

[Starblazer75]

Turns out to be a USB bandwidth problem. Not sure why some of the other errors were happening, but the specific one that I am worried about works fine when lowering the quality of the frames. Sorry about that, because it was only happening with the realsense camera I didn't think to modify the working cameras.

[MartyG-RealSense]

It's no problem at all. I'm pleased to hear that you diagnosed the problem. Thanks very much for the update!