NNstreamer stream detection video over udp

[Deepakpy]

i can currently use the below command for object detection and stream the results to the same device where camera and display are connected:
gst-launch-1.0 --no-position v4l2src device=/dev/video0 num-buffers=-1 ! video/x-raw,width=640,height=480,framerate=30/1 ! tee name=t t. ! queue max-size-buffers=2 leaky=2 ! videoconvert_pxp ! video/x-raw,width=300,height=300,format=BGR ! videoconvert ! video/x-raw,format=RGB ! tensor_converter ! tensor_filter framework=tensorflow-lite model=/home/root/tf/output/output/ssd_mobilenet_v2_coco.tflite ! tensor_decoder mode=bounding_boxes option1=tf-ssd option2=/home/root/tf/labels_coco.txt option3=0:1:2:3,50 option4=640:480 option5=300:300 ! videoconvert ! mix. t. ! queue max-size-buffers=2 ! compositor_pxp name=mix sink_0::zorder=2 sink_1::zorder=1 sink_0::alpha=0.3 ! autovideosink

Now i want to do the objectdetection part on the first device where camera is connected and stream the detection results video to a second device where display is connected. Any idea how this can be achieved.

I was able to use udp to send and receive only the video and not the detection part. Below is the plugins used:
Cam device: gst-launch-1.0 v4l2src ! video/x-h264,width=640,height=480,framerate=30/1 ! h264parse ! rtph264pay ! udpsink host=192.1xx.x.xxx port=4000

Display device: gst-launch-1.0 -v udpsrc port=4000 caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! rtph264depay ! avdec_h264 ! autovideosink sync=false

[gichan-jang]

If you do not need to use udp, how about using edgesrc/edgesink. (Supports TCP and MQTT protocols.)

Cam device pipeline (ximagesink is repaced with edgesink)

gst-launch-1.0     v4l2src name=cam_src ! videoconvert ! videoscale ! video/x-raw,width=640,height=480,format=RGB,framerate=30/1 ! tee name=t     t. ! queue leaky=2 max-size-buffers=2 ! videoscale ! video/x-raw,width=300,height=300,format=RGB ! tensor_converter !         tensor_transform mode=arithmetic option=typecast:float32,add:-127.5,div:127.5 !         tensor_filter framework=tensorflow-lite model=tflite_model/ssd_mobilenet_v2_coco.tflite !         tensor_decoder mode=bounding_boxes option1=mobilenet-ssd option2=tflite_model/coco_labels_list.txt option3=tflite_model/box_priors.txt option4=640:480 option5=300:300 !         compositor name=mix sink_0::zorder=2 sink_1::zorder=1 ! videoconvert ! edgesink port=3000     t. ! queue leaky=2 max-size-buffers=10 ! mix.

display device pipeline:

gst-launch-1.0  edgesrc dest-port=3000 ! video/x-raw,width=640,height=480,format=RGBA,framerate=0/1,pixel-aspect-ratio=1/1 ! videoconvert ! videoscale ! ximagesink

[Deepakpy]

Thanks for replying. If you don't mind can you elaborate how to use tcp with the edgesrc/edgesink. As in the udpsrc and sink allows to give the ip directly in thr pipeline. How am i supposed to use it with the edge src/sink

[gichan-jang]

If you installed nnstreamer, you can also use edgesrc/sink.
Check,

$ gst-inspect-1.0 edgesrc

To assign ip and port,

edgesink host=x.x.x.x port=xx connect-type=TCP (default)
edgesrc host=y.y.y.y port=yy dest-host=x.x.x.x.x (ip of edgesink device) dest-port=xx (port number of edgesink device) connect-type=TCP (default) ...

Simple pipeline to send video stream using edgesrc/sink

# Sender
$ gst-launch-1.0 v4l2src ! videoconvert ! videoscale !  video/x-raw,width=640,height=480,format=RGB,framerate=30/1 !  edgesink host=localhost port=3000

# Receiver 
$ gst-launch-1.0     edgesrc dest-port=3000 ! video/x-raw,width=640,height=480,format=RGB,framerate=0/1,pixel-aspect-ratio=1/1 ! videoconvert ! videoscale ! ximagesink

[Deepakpy]

Thanks that almost solved my issue

[nnbbensa]

Hello. This question is similar to mine so I decided to post here. Please let me know if I should open a new discussion and I will.

The edgesrc solution works for situations that allow for installation of nnstreamer to clients. However, in our case, backwards compatibility with other client solution requires staying with standard gstreamer streaming elements such as UDP element.

We are using the imx8mp board. We are using the following pipeline:
$GSTL -vvv v4l2src device=$USB_CAM_1 ! video/x-raw, width=640, height=480, framerate=30/1 \ ! tee name=t t. ! queue leaky=2 max-size-buffers=2 ! imxvideoconvert_g2d ! video/x-raw,width=300,height=300,format=RGBA ! \ videoconvert ! video/x-raw,format=RGB ! \ tensor_converter ! \ tensor_filter framework=tensorflow-lite model=${MODEL} \ custom=Delegate:External,ExtDelegateLib:libvx_delegate.so ! \ tensor_decoder mode=bounding_boxes option1=mobilenet-ssd-postprocess option2=${LABELS} \ option3=0:1:2:3,50 option4=640:480 option5=300:300 ! \ mix. t. ! queue max-size-buffers=2 ! \ imxcompositor_g2d name=mix latency=30000000 min-upstream-latency=30000000 \ sink_0::zorder=2 sink_1::zorder=1 ! \ imxvideoconvert_g2d ! "video/x-raw, width=640, height=480, framerate=30/1" \ ! vpuenc_h264 ! rtph264pay config-interval=1 pt=96 ! rtpstreampay ! udpsink host=$HOST_IP_1 port=$HOST_PORT_1 async=false sync=false
I do not see any errors and in fact the terminal msgs suggest that a live stream is running:
`
WARNING: Fallback unsupported op 32 to TfLite
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.

`
However, on the client side there is no video displayed. I have used wireshark to verify that there is no packets on HOST_PORT_1.

I can use similar pipeline (no object detection, overlay, and composite) just video encoding and streaming to the same port and client. Everything is working there no issues.
However, the pipeline above does not error out. It seems to run but streams nothing out from the edge board.

Please let me know if you can provide some suggestion as what could be wrong with the pipeline.
Thank you in advance.
NB

[myungjoo]

Your model appears to be not compatible with your tensorflow-lite binary:

WARNING: Fallback unsupported op 32 to TfLite
W [HandleLayoutInfer:281]Op 162: default layout inference pass.

Have you tested your tensorflow-lite binary that is linked to nnstreamer-tensorflow-lite with your model file and the same configuration (delegation). If you want to know why it doesn't work, make a shortest and simplest pipeline with the same error. And try to alter model, tflite options, and tflite binaries.

[nnbbensa]

Hello and Thank you for your prompt reply. I am not very versed with these new pipeline elements and don't fully understand them. But I will follow your suggestion and try a few options.

On the warning:
I think that means that the model attempted to run on NPU but was not fully compatible and hence ran on CPU partially. I have tested the model on tflite python code (interpreter.invoke()) and it provided object bounding box and correct label with high confidence. I get the same printed errors. I will have to address this issue but this is a performance concern. Clearly, despite NPU compatibility, the model runs. See below:

repeated static tests (1 image) in a loop. First detection was slower due to setup. Following detections much faster and acceptable performance.

python3 TF_Lite_Object_Detection.py 
Vx delegate: allowed_cache_mode set to 0.
Vx delegate: device num set to 0.
Vx delegate: allowed_builtin_code set to 0.
Vx delegate: error_during_init set to 0.
Vx delegate: error_during_prepare set to 0.
Vx delegate: error_during_invoke set to 0.
WARNING: Fallback unsupported op 32 to TfLite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
ie time (ms): 2.8258679994905833
ie time (ms): 0.27049900018027984
ie time (ms): 0.2191239982494153
ie time (ms): 0.21850000121048652
ie time (ms): 0.20987500101909973
ie time (ms): 0.2122489968314767
ie time (ms): 0.21299999934853986
ie time (ms): 0.22274999719229527
ie time (ms): 0.21312399985617958
ie time (ms): 0.2428740008326713
ie time (ms): 0.21099999867146835
ie time (ms): 0.21662400104105473
ie time (ms): 0.2118740012519993
ie time (ms): 0.20662500173784792
ie time (ms): 0.20462400061660446
ie time (ms): 0.20949900135747157
ie time (ms): 0.2010000025620684
ie time (ms): 0.20624899843824096
ie time (ms): 0.20950000180164352
ie time (ms): 0.20774999939021654
ie time (ms): 0.21124900013091974
ie time (ms): 0.2071250019071158
ie time (ms): 0.2039989994955249
ie time (ms): 0.2122490004694555
ie time (ms): 0.20274999769753776
ie time (ms): 0.23624900131835602
ie time (ms): 0.22049900144338608
ie time (ms): 0.212875002034707
ie time (ms): 0.2064990003418643
ie time (ms): 0.20724899877677672
ie time (ms): 0.21025000023655593
ie time (ms): 0.20474900156841613
ie time (ms): 0.20999899788876064
ie time (ms): 0.20975000006728806
ie time (ms): 0.20962399867130443
ie time (ms): 0.20724899877677672
ie time (ms): 0.20850000146310776
ie time (ms): 0.20674899860750884
ie time (ms): 0.20975000006728806
ie time (ms): 0.2087499997287523
ie time (ms): 0.2019989988184534
ie time (ms): 0.22162499953992665
ie time (ms): 0.2132500012521632
ie time (ms): 0.21574900165433064
ie time (ms): 0.21550000019487925
ie time (ms): 0.21212499996181577
ie time (ms): 0.20962399867130443
ie time (ms): 0.21475000175996684
ie time (ms): 0.2051250012300443
ie time (ms): 0.21274900063872337
ie time (ms): 0.21824999930686317
ie time (ms): 0.20862400197074749
ie time (ms): 0.20474900156841613
ie time (ms): 0.20737500017276034
ie time (ms): 0.20787399989785627
ie time (ms): 0.2089990011882037
ie time (ms): 0.21174999710638076
ie time (ms): 0.20187399786664173
ie time (ms): 0.2223739975306671
ie time (ms): 0.2132499976141844
ie time (ms): 0.20987400057492778
ie time (ms): 0.20849900101893581
ie time (ms): 0.20862499877694063
ie time (ms): 0.20249899898772128
ie time (ms): 0.2171239975723438
ie time (ms): 0.2074999974865932
ie time (ms): 0.2129989989043679
ie time (ms): 0.20874899928458035
ie time (ms): 0.21112500326125883
ie time (ms): 0.2044989996647928
ie time (ms): 0.20587399922078475
ie time (ms): 0.20450000010896474
ie time (ms): 0.2081240018014796
ie time (ms): 0.2074999974865932
ie time (ms): 0.22337500195135362
ie time (ms): 0.21474900131579489
ie time (ms): 0.21399999968707561
ie time (ms): 0.20612500156858005
ie time (ms): 0.2061240011244081
ie time (ms): 0.20487499932642095
ie time (ms): 0.2083750005112961
ie time (ms): 0.204873998882249
ie time (ms): 0.20662499809986912
ie time (ms): 0.20949900135747157
ie time (ms): 0.20787399989785627
ie time (ms): 0.20699999731732532
ie time (ms): 0.2129989989043679
ie time (ms): 0.20149900228716433
ie time (ms): 0.20812500224565156
ie time (ms): 0.20849900101893581
ie time (ms): 0.23524900097982027
ie time (ms): 0.21837500025867485
ie time (ms): 0.20849900101893581
ie time (ms): 0.20749900068040006
ie time (ms): 0.20987500101909973
ie time (ms): 0.2077489989460446
ie time (ms): 0.20824900275329128
ie time (ms): 0.21050000214017928
ie time (ms): 0.20862400197074749
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.

Question 1: If it runs on tflite it should run on nnstreamer? Or do I need to convert? If conversion is needed, then can you provide how? This is a retrained ssd v2 model on TF1.4.

Question 2: It seems like the gstreamer pipeline is running but I see no h.264 packets getting out. The pipeline suggests that the video packets should go out even if the bbox overlay does not? This maybe imxcompositor_g2d issue? Understand if that is not something you can answer but thought I would add it just in case this has been encountered. I have posted on NXP forum as well.

Thank you.

[myungjoo]

Question 1: If it runs on tflite it should run on nnstreamer? Or do I need to convert? If conversion is needed, then can you provide how? This is a retrained ssd v2 model on TF1.4.

Yes, as long as the same tf-lite binary is linked, it should run. There's no conversion required because nnstreamer treats a neural network as a black box, handled by the corresponding framework (tf-lite in this case). You really need to write a minimal pipeline that shows the same sympton.

Question 2: It seems like the gstreamer pipeline is running but I see no h.264 packets getting out. The pipeline suggests that the video packets should go out even if the bbox overlay does not? This maybe imxcompositor_g2d issue? Understand if that is not something you can answer but thought I would add it just in case this has been encountered. I have posted on NXP forum as well.

As above, I'd suggest to write a pipeline without tf-lite/tensor-filter, but with a media streams only. (you can create a stream from a static file, such as .png file). If I were you, I'd start with a minimal pipeline having imxcompositor and synthesized inputs (e.g., from .png/.bmp/.jpg or saved x-raw/video frames)