Updated Object Identification Sample

[varunjain3]

@nnshah1 hey!
Working on the Smart-City-Cloud, I realised that it would be beneficial if I could understand the making of platforms separately, I see that making the pipeline environment inside the smartcitysample doesn't take that much time to build, while this g-streamer container takes more than 5 hours to build, why is that and is there any way to avoid this?

Also for the re-identification models, I observe that we are using !gvaclassify , but can you make me understand that how should one make the .json file for the model? for eg, I see that in the emotion-recoginition model, you have used this input preproc https://github.com/intel/video-analytics-serving/blob/59fdcba3e7b631f391cf5654b30f78d56585411b/models/emotion_recognition/1/emotions-recognition-retail-0003.json#L3-L8
Can you tell me which layer_name are we referencing this to? ( I believe to the previous model)

Also, in the output_postpoc
https://github.com/intel/video-analytics-serving/blob/59fdcba3e7b631f391cf5654b30f78d56585411b/models/emotion_recognition/1/emotions-recognition-retail-0003.json#L9-L23
We have used some sort of converter, what is this.

My aim is to use the pipeline to run the person reidentification models. If you can help me understand how can I do that

[nnshah1]

@varunjain3 Thanks for the questions!

1. Docker build time. The default docker base image we use is based on the DL Streamer default base image which in turn currently follows a pattern similar to the Open Visual Cloud docker images repository. These docker files build most components by source to ensure that they are optimized for a given platform. Thus they take a long time. However Open Visual Cloud and OpenVINO have now released pre-built docker images on docker hub - and these can be used as the base image for va serving so that the build time is dramatically reduced. In VA Serving 0.3 you can issue the following command:

./docker/build.sh --base openvisualcloud/xeon-ubuntu1804-analytics-gst

2. Understanding the model-proc file. The layer names in the file refer to the neural network layer names as identified in the model.xml file. The model-proc file essentially describes the specific pre and post processing required to transform an incoming image into a tensor for passing to the model and for interpreting the tensor that the model produces. For classification one of the most common transformations is "tensor_to_label" which essentially transforms a vector of confidence values (one for each class identified by the model) into a single value based on the 'max' confidence.

3. Model proc for person-reidentification. You can use the following proc file:

   https://github.com/OpenVisualCloud/Smart-City-Sample/blob/51ffca882c843c81bd2b382131de27a507633677/analytics/entrance/models/person_reidentification_2020R2/1/person-reidentification-retail-0079.json

This transforms the output or rather retains the output as a feature embedding that can be used to identify persons against either a previously detected person.

You can find the corresponding pipeline description at:

https://github.com/OpenVisualCloud/Smart-City-Sample/blob/51ffca882c843c81bd2b382131de27a507633677/analytics/entrance/Xeon/gst/pipeline/2/pipeline.json

Note: this pipeline uses a custom python function via gvapython to do the matching. However you can also use a sample element gvaidentify in its place with a gallery of known persons.

 https://github.com/OpenVisualCloud/Ad-Insertion-Sample/blob/master/ad-insertion/analytics/Xeon/gst/pipelines/face_recognition/1/pipeline.json 

More information on gvaidentify, here:

  https://github.com/opencv/gst-video-analytics/tree/v1.0/samples/gst_launch/reidentification

4. Quick Note: We are also working on an incremental update with updated documentation on some of these topics in the next week (targeting release 0.3.1 next Friday). As we are also planning documentation updates for our 0.4 release - please let us know of topics you'd like to see covered.

[varunjain3]

Hey @nnshah1 ! Thanks for the answer, It really helped.
Working forward on this, I realized that I am still not able to understand how exactly does the gst-pipelines calls the python code and add it to the output message that is streamed through mqtt.

For eg, I tried the emotion recognition pipeline available on the video-analytics-sevices repository in the traffic scenerio in smart-city-sample and I was successfull printing the bounding box as well as the emotion using minor changes in the analysis.js.
https://github.com/intel/video-analytics-serving/tree/master/pipelines/gstreamer/emotion_recognition/1

I could see the .json output using the inspect element feature on the webpage, where the .json file stored the frame wise structured output of the model.

I tried running the entrance pipeline which contained the gvapython flag. And as I can see the output stream is changed using the gvapython code, as per what I understood, the pipline calls the process_frame method.
https://github.com/OpenVisualCloud/Smart-City-Sample/blob/5c3690eba2d7b664ab5a83d72177e6ec2aa0741f/analytics/entrance/custom_transforms/people_counting.py#L18-L28

I understood how all the counting was happening in the python code, yet I couldnt understand how the message stream was being changed by the python code.
Could you please explain me that.

Also I tried running the same person re identification models (face +reidentification) without the gvapython code hopping that I would get the .json file with frame wise output, but that was not the case, even though the analytics counted the number of people correctly but I wasn't satisfied with the output stream of data. and there were no bounding boxes as well.

Comming to the gvaidentify part, Thanks for sharing the related repos, I wanted to ask, will the pipeline that is present in ad-insertion-sample work without any changes directly in the traffic scenerio? I was confused because on this url you shared https://github.com/opencv/gst-video-analytics/tree/v1.0/samples/gst_launch/reidentification it says that I have to compile the custom gvaidentify.

For say, I want to make a custom library with face encodings, can I directly run the https://github.com/opencv/gst-video-analytics/tree/v1.0/samples/gst_launch/reidentification python code on my library with the correct models downloaded or do i need to compile some C code first in order to do that.

-please let us know of topics you'd like to see covered.
It took us a long time to understand the gstreamer-pipeline and all the various flags it contains. could there be a small brief about it in the code repo itself.
Also, I am still not sure on how can one edit the model-proc (json file) in order to accommodate various model requirements and flags can be used there.

[nnshah1]

I understood how all the counting was happening in the python code, yet I couldnt understand how the message stream was being changed by the python code.
Could you please explain me that.

The message is being updated on line 22 - 25. There the original message is loaded from the frame, modified, and then we remove the original message and add the modified one:

https://github.com/OpenVisualCloud/Smart-City-Sample/blob/5c3690eba2d7b664ab5a83d72177e6ec2aa0741f/analytics/entrance/custom_transforms/people_counting.py#L24

Comming to the gvaidentify part, Thanks for sharing the related repos, I wanted to ask, will the pipeline that is present in ad-insertion-sample work without any changes directly in the traffic scenerio?

No unfortunately some modifications are needed to support rtsp vs files -

For say, I want to make a custom library with face encodings, can I directly run the https://github.com/opencv/gst-video-analytics/tree/v1.0/samples/gst_launch/reidentification python code on my library with the correct models downloaded or do i need to compile some C code first in order to do that.

If you are using similar code / logic to that in the people counting.py you will not need any additional C code. The C element provides similar functionality. Depending on the underlying libraries and algorithms - the performance of cosine distance and matching in python may be need to be optimized / translated to C.

[nnshah1]

@varunjain3 How is your project going? Please let us know if there is anything else you need from us - we've also just released 0.3.1 with a set of improved documentation. It'd doesn't cover all the topics you've brought up here - but would like to get your feedback. If this topic is resolved, let us know and we can close the issue.

[divdaisymuffin]

Hi @nnshah1
Sorry for late response, I am from the same team working along varun. I saw all your suggestions, and tried to do the face recognition in smart-city cloud.
For doing so, I am using face-detection-adas, landmark-regression and face-reidentification-retail model.
I have created pipeline.json accordingly and given gvaidentify, from Ad-insertion pipeline I get to know that images should be converted to tensor and then given path to gvaidentify. So, I installed gst-video-analytics and then run gallery_generator.py, the pipeline is running but in output I am getting a gallery.json, which is half build and features directory empty.

Command that I run is
python gst-video-analytics/samples/gst_launch/reidentification/gallery_generator.py -s face_gallery_FP32/ - o face_gallery_FP32/ -d test/Smart-City-Sample/analytics/object/models/face_detection_2020R2/1/FP32/face-detection-adas-0001.xml -i test/Smart-City-Sample/analytics/ob ject/models/landmarks_detection_2020R2/1/FP32/landmarks-regression-retail-0009.xml -l test/Smart-City-Sample/analytics/object/models/reidentification_detection_2020R2/ 1/FP32/face-reidentification-retail-0095.xml --identification_modelproc gst-video-analytics/samples/gst_launch/reidentification/model_proc/face-reidentification-retail -0095.json --landmarks_regression_modelproc gst-video-analytics/samples/gst_launch/reidentification/model_proc/landmarks-regression-retail-0009.json

Please help me track the issue, I am also attaching some screenshots.

[divdaisymuffin]

@nnshah1 Apart from the above issue, when I am running the smart-city with the changed pipeline.json (screenshot in above comment) with face-detection-adas, landmark-regression and face-reidentification-retail model, I am getting error in model loading for landmark-regression and face-reidentification (screenshot attached), so I thought it could be model_proc issue, but when I am loading them individually, they are loading fine.
Please provide some suggestions in it.
Error msg shown is
{"levelname": "ERROR", "asctime": "2020-08-18 05:19:18,119", "message": "Error Loading Model reidentification_detection_2020R2 from: /home/models: invalid literal for int() with base 10: '.ipynb_checkpoints'", "module": "model_manager"}

Traceback (most recent call last): File "/home/runva.py", line 50, in loop 'max_running_pipelines': 1, File "/home/vaserving/vaserving.py", line 129, in start self.options.ignore_init_errors) File "/home/vaserving/model_manager.py", line 58, in init raise Exception("Error Initializing Models") Exception: Error Initializing Models Traceback (most recent call last): File "/home/detect-object.py", line 32, in connect raise Exception("VA exited. This should not happen.") Exception: VA exited. This should not happen.

[nnshah1]

@divdaisymuffin Can you provide a directory tree for the models directory - it looks like the model manager is loading a file with extension .ipync_checkpoints and getting confused. Probably this error should be ignored and not treated as fatal.

In the meantime you can also use the environment variable: IGNORE_INIT_ERRORS=True, to continue execution, and check that the models have been registered correctly using the models REST end point.

[nnshah1]

On the issue of gallery generation, can you increase the GST_DEBUG level to 3 and see if there are indeed faces detected within the images? It looks as if it is possible the faces are not being detected in the source files.

[divdaisymuffin]

Thank you @nnshah1 for your quick response.
The model loading issue is resolved, we have created certain changes in the pipeline.json, had to use the model-instance-id for each model.
I am posting corrections made to the pipeline.json.
"template":"rtspsrc udp-buffer-size=212992 name=source ! queue ! rtph264depay ! h264parse ! video/x-h264 ! tee name=t ! queue ! decodebin ! videoconvert name="videoconvert" ! video/x-raw,format=BGRx ! queue leaky=upstream ! gvadetect model-instance-id=det1 model="{models[face_detection_adas][1][network]}" model-proc="{models[face_detection_adas][1][proc]}" name="detection" ! queue ! gvaclassify model-instance-id=cls1 model="{models[landmarks_regression][1][network]}" model-proc="{models[landmarks_regression][1][proc]}" name="landmarksregression" ! queue ! gvaclassify model-instance-id=cls2 model="{models[face_reidentification][1][network]}" model-proc="{models[face_reidentification][1][proc]}" name="facereidentificationretail" ! queue ! gvaidentify gallery="/home/gallery/face_gallery_FP32/gallery.json" name="identify" ! queue ! gvametaconvert name="metaconvert" ! queue ! gvametapublish name="destination" ! appsink name=appsink t. ! queue ! splitmuxsink max-size-time=60500000000 name="splitmuxsink"",

[divdaisymuffin]

But now I am getting issue related to gvaidentify

{"levelname": "INFO", "asctime": "2020-08-18 12:34:12,820", "message": "Creating Instance of Pipeline object_detection/2", "module": "pipeline_manager"} {"levelname": "ERROR", "asctime": "2020-08-18 12:34:13,035", "message": "Error on Pipeline 1: gst-resource-error-quark: gallery file failed to open (3): /home/gst-video-analytics/samples/gst_launch/reidentification/gvaidentify/reid_gallery.cpp(73): EmbeddingsGallery (): /GstPipeline:pipeline17/GstGvaIdentify:identify:\nCannot open gallery file: /home/gallery/face_gallery_FP32/gallery.json.", "module": "gstreamer_pipeline"} PipelineStatus(avg_fps=0, avg_pipeline_latency=None, elapsed_time=3.457069396972656e-05, id=1, start_time=1597754053.039853, state=<State.ERROR: 4>) Pipeline object_detection Version 2 Instance 1 Ended with ERROR Traceback (most recent call last): File "/home/detect-object.py", line 32, in connect raise Exception("VA exited. This should not happen.") Exception: VA exited. This should not happen.

I am also attaching screenshot for the same, and I am also telling you the hierarchy of the gallery folder along with gallery.json (as I mentioned earlier I am not able to convert images to tensor, so I have used .jpgs in the gallery.json)

Please have a look, and help me with this glitch.

[nnshah1]

@divdaisymuffin The identify element will not support jpg and will require the tensors to be generated. Please increase the GST_DEBUG level before running the gallery generator. We will need to understand that issue before proceeding to the VA serving pipeline.

[divdaisymuffin]

@nnshah1 Thanks, I have increased GST_DEBUG level to 6, and got detailed error info. Pasting it here, please have a look.
command is
divya5@divya5:~$ python gst-video-analytics/samples/gst_launch/reidentification/gallery_generator.py -s face_gallery_FP32/ -o face_gallery_FP32/ -d test/Smart-City-Sample/analytics/object/models/face_detection_2020R2/1/FP32/face-detection-adas-0001.xml -i test/Smart-City-Sample/analytics/object/ models/landmarks_detection_2020R2/1/FP32/landmarks-regression-retail-0009.xml -l test/Smart-City-Sample/analytics/object/models/reidentification_det ection_2020R2/1/FP32/face-reidentification-retail-0095.xml --identification_modelproc gst-video-analytics/samples/gst_launch/reidentification/model_ proc/face-reidentification-retail-0095.json --landmarks_regression_modelproc gst-video-analytics/samples/gst_launch/reidentification/model_proc/lan dmarks-regression-retail-0009.json

ERROR is here:
Setting pipeline to PAUSED ... ERROR: Pipeline doesn't want to pause. ERROR: from element /GstPipeline:pipeline0/GstGvaPython:gvapython0: Error creating Python callback Additional debug info: /root/gst-video-analytics/gst/elements/gvapython/gstgvapython.c(205): gst_gva_python_start (): /GstPipeline:pipeline0/GstGvaPython:gvapython0: Module: /home/divya5/gst-video-analytics/samples/gst_launch/reidentification/gallery_generator.py Class: Processor Function: process_frame Arg: ["face_gallery_FP32/gallery/features/vicky_0.tensor"] Keyword Arg: {} Setting pipeline to NULL ... Freeing pipeline ... Error while running pipeline

[nnshah1]

Just to confirm, which version of gst-video-analytics is this?

[divdaisymuffin]

@nnshah1 I have used this git link to install gst-vide0-analytics
https://github.com/opencv/gst-video-analytics/tree/v1.0
so, its v1.0

[divdaisymuffin]

hi @nnshah1 I changed to version 2020.2, but still getting same error
Please provide some insight into it

[nnshah1]

@divdaisymuffin Thank you for your patience. I'm in the process of lining up the right resources to look into your issue. Can you give me more details about the project you and @varunjain3 are part of? If it helps facilitate we can set up a time to discuss in more detail.

[dhavalsvora]

@nnshah1 - Divya & Varun are team members of company AIVIDTECHVISION LLP. We are building SaaS based Video Analytics Platform. We are excited to use Open Visual Cloud, OpenVINO code base in our offering and also contribute in enhancing it further. we all are currently facing technical issues as mentioned by @varunjain3, @divdaisymuffin. Your help on this will accelerate our understanding about Open Visual cloud and will also help us contribute better. I am open to do conference call with you for any queries on this regard. thanks, Dhaval Vora - Co-Founder & CEO, AIVIDTECHVISION.

[nnshah1]

Please find the updated sample here demonstrating how to enable object identification via a public face recognition model and the latest video analytics serving.

object_identification_sample.zip
README.pdf

[varunjain3]

Thanks @nnshah1 ! Do give us some time to understand this and get back to you shortly.

[divdaisymuffin]

hi @nnshah1, Thank you so much for the help.
I am working with sample provided by you. While following the process I got stuck with certain errors.

1. few warnings with
   git apply ../object_identification_sample/object-identification-sample.patch

git apply ../object_identification_sample/object-identification-sample.patch ../object_identification_sample/object-identification-sample.patch:101: trailing whitespace. ../object_identification_sample/object-identification-sample.patch:104: trailing whitespace. ../object_identification_sample/object-identification-sample.patch:107: trailing whitespace. ../object_identification_sample/object-identification-sample.patch:116: trailing whitespace. ../object_identification_sample/object-identification-sample.patch:122: trailing whitespace. warning: squelched 79 whitespace errors warning: 84 lines add whitespace errors.

and rerunning the same give me detailed errors
git apply ../object-identification-sample.patch ../object-identification-sample.patch:101: trailing whitespace. ../object-identification-sample.patch:104: trailing whitespace. ../object-identification-sample.patch:107: trailing whitespace. ../object-identification-sample.patch:116: trailing whitespace. ../object-identification-sample.patch:122: trailing whitespace. error: patch failed: docker/Dockerfile:42 error: docker/Dockerfile: patch does not apply error: extensions/vas_identify.py: already exists in working directory error: models/face_recognition/1/face-recognition-resnet100-arcface.json: already exists in working directory error: pipelines/gstreamer/object_identification/1/pipeline.json: already exists in working directory error: pipelines/gstreamer/object_identification/2/pipeline.json: already exists in working directory error: samples/object_identification/8bdbdf3c-f11c-11ea-8ccf-1c697a06fd65.1.jpeg: already exists in working directory error: samples/object_identification/README.md: already exists in working directory error: samples/object_identification/download_recognition_model.sh: already exists in working directory error: samples/object_identification/object_identification.jpg: already exists in working directory error: samples/object_identification/object_identification.py: already exists in working directory error: samples/object_identification/object_identification.svg: already exists in working directory error: samples/object_identification/pipeline_diagram_version_1.mmd: already exists in working directory error: samples/object_identification/pipeline_diagram_version_1.svg: already exists in working directory error: samples/object_identification/pipeline_diagram_version_2.mmd: already exists in working directory error: samples/object_identification/pipeline_diagram_version_2.svg: already exists in working directory error: patch failed: vaserving/gstreamer_pipeline.py:41 error: vaserving/gstreamer_pipeline.py: patch does not apply error: patch failed: vaserving/vaserving.py:135 error: vaserving/vaserving.py: patch does not apply

Please help me with this.

[varunjain3]

@divdaisymuffin I think you will need to correctly pull the side-branch of the va-serving as specified by @nnshah1 in the readme, it works fine for me.

[divdaisymuffin]

@nnshah1 Following instructions further I got another issue while downloading the model.

sudo ./samples/object_identification/download_recognition_model.sh

I got following issue

fc0bf831bab1: Pull complete Digest: sha256:0b581106c80a9d8d2ce79cd8686fdc94fab20ce9c6738c04bf22cf3885a17e22 Status: Downloaded newer image for openvino/ubuntu18_data_dev:2020.4 ################|| Downloading models ||################ Traceback (most recent call last): File "/usr/lib/python3.6/pathlib.py", line 1248, in mkdir self._accessor.mkdir(self, mode) File "/usr/lib/python3.6/pathlib.py", line 387, in wrapped return strfunc(str(pathobj), *args) FileNotFoundError: [Errno 2] No such file or directory: '/home/openvino/tmp/models/public/face-recognition-resnet100-arcface' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/intel/openvino/deployment_tools/tools/model_downloader/downloader.py", line 352, in <module> main() File "/opt/intel/openvino/deployment_tools/tools/model_downloader/downloader.py", line 323, in main for model in models] File "/opt/intel/openvino/deployment_tools/tools/model_downloader/downloader.py", line 323, in <listcomp> for model in models] File "/opt/intel/openvino/deployment_tools/tools/model_downloader/downloader.py", line 212, in download_model output.mkdir(parents=True, exist_ok=True) File "/usr/lib/python3.6/pathlib.py", line 1252, in mkdir self.parent.mkdir(parents=True, exist_ok=True) File "/usr/lib/python3.6/pathlib.py", line 1248, in mkdir self._accessor.mkdir(self, mode) File "/usr/lib/python3.6/pathlib.py", line 387, in wrapped return strfunc(str(pathobj), *args) PermissionError: [Errno 13] Permission denied: '/home/openvino/tmp/models/public'

I got permission denied even I am running with root privileges.

[divdaisymuffin]

@varunjain3 I am following Readme.pdf only. Please suggest me what I am doing wrong.

[nnshah1]

@divdaisymuffin The warnings on git apply are to be expected - apologies for the confusion there. The patch can only be applied once so rerunning it will generate the errors (the first time it will display warnings).

Please start with a fresh va serving clone, and run the download script without sudo.

I believe the issue has to do with the permissions of the openvino user

If you continue to run into issues, please modify the download shell script to pass a user flag to the docker commands:

docker run -it --rm $ENVIRONMENT --user ${USER} -v$TEMP_MODELS:/home/openvino/tmp/models $OPENVINO_IMAGE $DOWNLOAD

[divdaisymuffin]

Thanks @nnshah1 that solved my issue 👍
Now proceeding further.

[divdaisymuffin]

@nnshah1
The tensor generation for classroom.mp4 is working fine with the below script and I am getting expected output also.
python3 ./samples/object_identification/object_identification.py --enroll --max-enrolled 4 --dump-frames --stop-on-max

But I am getting one error while running object_identification.py with head-pose-face-detection-male.mp4
Script that I run:
python3 ./samples/object_identification/object_identification.py --source https://github.com/intel-iot-devkit/sample-videos/b

Error:
{"levelname": "ERROR", "asctime": "2020-09-08 19:46:30,547", "message": "Error on Pipeline 1: gst-resource-error-quark: Not Found (3): gstsouphttpsrc.c(1428): gst_soup_http_src_parse_status (): /GstPipeline:pipeline5/GstURISourceBin:source/GstSoupHTTPSrc:source:\nNot Found (404), URL: https://github.com/intel-iot-devkit/samplevideos/blob/master/head-pose-face-detection-female-and-male.mp4?raw=true, Redirect to: (NULL)", "module": "gstreamer_pipeline"}

and I am not getting expected result for head-pose-face-detection-male.mp4

[nnshah1]

https://github.com/intel-iot-devkit/sample-videos/blob/master/head-pose-face-detection-female-and-male.mp4?raw=true

Looks like a typo in the instructions - apologies - the correct url has: sample-videos instead of samplevideos

[divdaisymuffin]

It worked, Thanks @nnshah1

[divdaisymuffin]

hi @nnshah1, I got a error at step 2 of Deploying Object Identification as a Microservice.

command
./docker/run.sh -e IGNORE_INIT_ERRORS=true -v /tmp:/tm

Error:
docker: Error response from daemon: error gathering device information while adding custom device "/dev/dri": no such file or directory.

I am running it on azure vm with ubuntu 18.04.

[nnshah1]

it looks like you are running on a system without an graphics devices exposed, in the docker run script you can comment out the like which adds this device to the container - will add detection for this scenario in the future.

Please comment out this line:

https://github.com/intel/video-analytics-serving/blob/e735a1728fb4f19ddce51b5e6d42ef2bf4456ac7/docker/run.sh#L241

[divdaisymuffin]

Thanks @nnshah1 for providing this object-reidentification patch in VAserving and also for your guidance. I am able to get the output in the file object_identification_results.txt which is stored in /tmp location.

I have also checked "/home/user/video-analytics-serving/samples/object_identification/results/" directory and it contains video specific frames and results.jsonl file.

But I haven't found the post-processed video as output as a file or a stream, Please suggest How I can see that. Do I need to make changes to pipeline.json at /video-analytics-serving/pipelines/gstreamer/object_identification/1/ specially to gvametapublish ?

[nnshah1]

@divdaisymuffin That's great to hear!

To save the watermarked results as a video file or as a stream you would need to modify

object_identification/2/

to use splitmuxsink or an rtspsink instead of jpegenc

To integrate the results into smart cities however, you can modify

object_identification/1/

to add recording based on splitmuxsink without watermarks.

[divdaisymuffin]

Thanks @nnshah1 I tried with gvawatermark option for now, so I made changes to the object_identification/2/pipeline.json, replacing "jpegenc" with "rtspsink" and "splitmuxsink" both, and also provided a location to save the output.mp4.

But I am getting same error in both cases "rtspsink" and "splitmuxsink" (Errors shown in attached screenshot). Please suggest how to implement it to get the output as video.

Pipeline.json

"urisourcebin name=source ! decodebin ! videoconvert name=videoconvert ! video/x-raw,format=BGRx", " ! gvadetect model-instance-id=detection model={models[face_detection_adas][1][network]} name=detection", " ! gvaclassify model={models[landmarks_regression][1][network]} model-proc={models[landmarks_regression][1][proc]} name=landmarks model-instance-id=landmarks", " ! gvaclassify model={models[face_recognition][1][network]} model-proc={models[face_recognition][1][proc]} name=recognition model-instance-id=recognition", " ! gvapython name=identify module=/home/video-analytics-serving/extensions/vas_identify.py class=Identify", " ! gvametaconvert name=metaconvert ! queue ! gvametapublish name=destination", " ! tee name = t ! queue ! gvawatermark ! videoconvert ! splitmuxsink", " ! gvapython name=capture module=/home/video-analytics-serving/extensions/vas_identify.py class=Capture", " ! multifilesink name=filesink location=/home/divya10/video-analytics-serving/out.mp4 t. ! queue", " ! appsink name=appsink"

Please see attached screenshots for details

[nnshah1]

Please try something like:

"urisourcebin name=source ! decodebin ! videoconvert name=videoconvert ! video/x-raw,format=BGRx", 
" ! gvadetect model-instance-id=detection model={models[face_detection_adas][1][network]} name=detection", 
" ! gvaclassify model={models[landmarks_regression][1][network]} model-proc={models[landmarks_regression][1][proc]} name=landmarks model-instance-id=landmarks", 
" ! gvaclassify model={models[face_recognition][1][network]} model-proc={models[face_recognition][1][proc]} name=recognition model-instance-id=recognition", 
" ! gvapython name=identify module=/home/video-analytics-serving/extensions/vas_identify.py class=Identify", 
" ! gvametaconvert name=metaconvert ! queue ! gvametapublish name=destination", 
" ! tee name = t ! queue ! gvawatermark ! videoconvert ! x264enc ! h264parse ",
" ! splitmuxsink location=/home/divya10/video-analytics-serving/out_%02d.mp4 t. ! queue", 
" ! appsink name=appsink"

This site has some nice instructions:

https://github.com/DamZiobro/gstreamerCheatsheet

Including: https://github.com/DamZiobro/gstreamerCheatsheet#splitmuxsink-test-overwritting-max-2-files

[divdaisymuffin]

Thank you @nnshah1 , I tried gvawatermark one, error resolved but I am not able to find output video file.

Smart city one I didn't tried yet, I will try the smart city implementation and update you on that.

[nnshah1]

Looking at this path: /home/divya10/video-analytics-serving/out_%02d.mp4

Looks like it is a path from your host machine - it won't be accessible directly in the container.

In dev mode we mount /tmp. Try switching this path to /tmp/out_%02d.mp4

[divdaisymuffin]

Hey @nnshah1 I need your help again, I was trying to integrate the results to smart-city following template worked for me to get the out put
"template": "rtspsrc udp-buffer-size=212992 name=\"source\" ! queue ! rtph264depay ! h264parse ! video/x-h264 ! tee name=t ! queue ! decodebin ! videoconvert name=\"videoconvert\" ! video/x-raw,format=BGRx ! queue leaky=upstream ! gvadetect ie-config=CPU_BIND_THREAD=NO model-instance-id=detection model=\"{models[face_detection_adas][1][network]}\" name=\"detection\" ! queue ! gvaclassify model=\"{models[landmarks_regression][1][network]}\" model-proc=\"{models[landmarks_regression][1][proc]}\" name=\"landmarks\" model-instance-id=landmarks ! queue ! gvaclassify model=\"{models[face_recognition][1][network]}\" model-proc=\"{models[face_recognition][1][proc]}\" name=\"recognition\" model-instance-id=recognition ! queue ! gvapython name=\"identify\" module=\"extensions/vas_identify\" class=\"Identify\" ! gvametaconvert name=\"metaconvert\" ! queue ! gvametapublish name=\"destination\" ! appsink name=appsink t. ! queue ! splitmuxsink max-size-time=60500000000 name=\"splitmuxsink\""

So I am getting the streaming and everything but have 2 issues:

1. The gallery.json and gallery folder is not at the path defined in the vas_identify.py
2. I am not getting bounding boxes around the faces ( already changed analytics.js in the smartcity)

For details I am attaching screenshots.

1. The gallery.json is not at location defined in the vas_identify.py
   

2. the output doesn't contain bounding box
   

3. added face label in the analytics.jsbut still no bounding box
   

Please help me to solve this

[nnshah1]

@divdaisymuffin

Two things I think may be at play:

1. The gallery either has to be pre-built and added to the container, or the enroll parameter needs to be set to true (can be done via the request, or change the default)

2. in the sample we left out the model-proc file for the detection model. The reason was that gvawatermark will use the detection label instead of the identification label if both are present. In the smart cities case, since gvawatermark is not used, please add the model-proc file back to the template.

"template": "rtspsrc udp-buffer-size=212992 name=\"source\" ! queue ! rtph264depay ! h264parse ! video/x-h264 ! tee name=t ! queue ! decodebin ! videoconvert name=\"videoconvert\" ! video/x-raw,format=BGRx ! queue leaky=upstream ! gvadetect ie-config=CPU_BIND_THREAD=NO model-instance-id=detection model=\"{models[face_detection_adas][1][network]}\" name=\"detection\" model-proc=\"{models[face_detection_adas][1][proc]}\" ! queue ! gvaclassify model=\"{models[landmarks_regression][1][network]}\" model-proc=\"{models[landmarks_regression][1][proc]}\" name=\"landmarks\" model-instance-id=landmarks ! queue ! gvaclassify model=\"{models[face_recognition][1][network]}\" model-proc=\"{models[face_recognition][1][proc]}\" name=\"recognition\" model-instance-id=recognition ! queue ! gvapython name=\"identify\" module=\"extensions/vas_identify\" class=\"Identify\" ! gvametaconvert name=\"metaconvert\" ! queue ! gvametapublish name=\"destination\" ! appsink name=appsink t. ! queue ! splitmuxsink max-size-time=60500000000 name=\"splitmuxsink\""

That will then add the "face" label to the bounding box - and I believe will be rendered by smart cities

[nnshah1]

@divdaisymuffin Does this unblock your progress?

[divdaisymuffin]

Hi @nnshah1 , I tried your suggestions to get gallery directory, but that doesn't solved my issue :(

Although your suggestion to get bounding box worked very well.

I think to get gallery directory created in the smartcity docker swarm environment ,changes need to be added to the docker file of the analytics container of the Smartcity, So I am trying that right now. I will let you know, if it works.

[nnshah1]

@divdaisymuffin Is there a gallery directory created in the analytics container? If the gallery.json is what is missing - we need to call the "save gallery" method from the sample code either on startup or shutdown. If the"enroll" parameter is set to true - you should see the gallery directory with tensor files created.

[divdaisymuffin]

@nnshah1 , No gallery directory is not getting created by the pipeline. yeah enroll parameter is True, but still no gallery directory is created.

[divdaisymuffin]

hi @nnshah1 , I am able to enroll faces now, what worked for me is to put the vas_identify.py at /home/Smart-city-sample/analytics/object/ , earlier I put it at /home/Smart-city-sample/analytics/object/extensions.
But @nnshah I need your help still to locate the created gallery, if its getting created, because I tried to locate it, it does not show up in the system. Additionally I noticed that even now system is doing enrolling, but after stats not coming up, for details please see the attached screenshot.

[nnshah1]

@divdaisymuffin

The simplest thing will be to add additional print statements to vas_identify to print where the tensors are getting stored.

The gallery.json file itself - needs to be created via the code in the sample (vas_identify does not create the gallery.json itself, only the tensors).

The stats are also printed by the sample app and not vas_identify - but you can add print statements into vas_identify to turn on additional logging when objects are matched or not matched - or to keep summary stats.

[Shirish-Ranoji]

Hi @nnshah1

I m in a bit of trouble in extending this video analytics serving module to a new set of problem statements.

We are extending to our use case and accordingly modifying the changes required.

We replaced the face detection model with the person detection model and reidentification with the person reidentification. We were able to enroll in the tensors and dump the frames.

Now we find that there are many false/incorrect detections that have to be suppressed.

In many of the libraries, we have a confidence/detection threshold which can effectively eliminate the incorrect and extra bounding boxes generated at the time of inference.

Is there any parameter as such to target and tweak them when using it in GStreamer pipelines? I have attached the images as to why I am in need of these parameters.

[nnshah1]

1. For enrolling / identifying two objects as the same you can modify the threshold value in vas_identify.py. Currently this is set to 0.6 by default - but can be modified either in the extension itself or in the request.

2. For detection threshold you can modify the threshold property of the detect element in pipeline.json.

   " ! gvadetect model-instance-id=detection model={models[face_detection_adas][1][network]} name=detection threshold=0.7"

3. the matching algorithm in the sample is nearest neighbor and doesn't ensure that for example the same person is identified twice in the same frame.

   https://github.com/OpenVisualCloud/Smart-City-Sample/blob/master/analytics/entrance/custom_transforms/people_counting.py

   Shows person reidentification (rolling) using a Munkres matching algorithm (more sophisticated but more computationally intensive).

4. depending on the video / feed and the assumptions of distance to the objects - you can also decide in the vas_identify to exclude detections of a specific size (i.e. too large or too little) - before matching.

[nnshah1]

@Shirish-Ranoji , @divdaisymuffin

Could we close this issue as the basics have been demonstrated? We can open a new issues to discuss further enhancements. Does that seem reasonable?

[Shirish-Ranoji]

Hi, @nnshah1 Thankyou so much for your quick response.

I would look into custom_transforms from the smart city project and try the pipeline with the detection threshold.

Also, it makes sense to close this issue and start a new one as it has grown too long.