Timelapse Sample. #352

AdityaKBhadragond14 · 2024-05-15T11:33:56Z

IMPORTANT: All PRs must be linked to an issue (except for extremely trivial and straightforward changes).

Description
This PR contains the Surveillance summary timelapse based on motion detection sample number 7 from above linked issue.
This sample utilizes the following modules:

Mp4ReaderSource
MotionVectorExtractor
ColorConversionXForm
H264EncoderNVCodec
Mp4WriterSink

The Mp4ReaderSource reads a mp4 file. Mp4ReaderSource next is MotionVectorExtractor which gets the frames which match the given threshold and which are in motion. After MotionVectorExtractor the frames are passed to ColorConversionFormX to convert from BGR tor RGB and then from RGB to YUV420Planar because the H264EncoderNVCodec module only accepts YUV frames. After color conversion next is H264EncoderNVCodec which encodes the frames. After H264EncoderNVCodec next is Mp4WriterSink which writes the frames to disk.
Also in the MotionVectorExtractor I have introduced a flag sendOverlayFrame which when enabled sends the overlay frame else does not send the overlay frames. Since we dont need the overlay frames in this sample setting it to false.
This PR also has the fix for the issue Raw yuv420 output from motion extractor module is having horizontally layered bands on the frame.

Alternative(s) considered

Have you considered any alternatives? And if so, why have you chosen the approach in this PR?

Type

Feature and Bug Fix.

Screenshots (if applicable)

Checklist

I have read the Contribution Guidelines
I have written Unit Tests
I have discussed my proposed solution with code owners in the linked issue(s) and we have agreed upon the general approach

…Queues and seek feature

mraduldubey

formatting and other comments, restructure the format of tests, new impl for tests is required

mraduldubey · 2024-05-20T08:11:27Z

base/include/MotionVectorExtractor.h

@@ -24,6 +24,7 @@ class MotionVectorExtractorProps : public ModuleProps
 		return ModuleProps::getSerializeSize() + sizeof(sendDecodedFrame) + sizeof(motionVectorThreshold);
 	}
 	bool sendDecodedFrame = false;
+	bool sendOverlayFrame = false;


should be added in serialization part below

mraduldubey · 2024-05-20T08:15:07Z

base/src/MotionVectorExtractor.cpp

@@ -202,6 +208,19 @@ int DetailFfmpeg::decodeAndGetMotionVectors(AVPacket* pkt, frame_container& fram
 			{
 				outFrame = makeFrameWithPinId(0, motionVectorPinId);
 			}
+			if (sendOverlayFrame == false) {


formatting

put a comment on what is this change about

mraduldubey · 2024-05-20T08:15:37Z

base/src/MotionVectorExtractor.cpp

@@ -244,8 +264,15 @@ void DetailOpenH264::getMotionVectors(frame_container& frames, frame_sp& outFram
 	int32_t mMotionVectorSize = mWidth * mHeight * 8;
 	int16_t* mMotionVectorData = nullptr;
 	memset(&pDstInfo, 0, sizeof(SBufferInfo));
-	outFrame = makeFrameWithPinId(mMotionVectorSize, motionVectorPinId);
-	mMotionVectorData = static_cast<int16_t*>(outFrame->data());
+	if (sendOverlayFrame) 


formatting, in vs code use Ctrl+K, F to auto format

mraduldubey · 2024-05-20T08:16:18Z

base/src/MotionVectorExtractor.cpp

@@ -290,8 +318,19 @@ void DetailOpenH264::getMotionVectors(frame_container& frames, frame_sp& outFram
 			frames.insert(make_pair(motionVectorPinId, outFrame));
 		}
 	}
-
-	if ((!sDecParam.bParseOnly) && (pDstInfo.pDst[0] != nullptr) && (mMotionVectorSize != mWidth * mHeight * 8))
+	if(sendOverlayFrame == false){


what is this change about ? why is this reqd in the sample PR

This is required to determine whether the frames are in the motion or not when we set the sendOverlayFrame to false.

mraduldubey · 2024-05-20T08:18:45Z

samples/CMakeLists.txt

@@ -0,0 +1,10 @@
+#dependencies
+
+find_package(Threads REQUIRED)


Threads is required here, why ?

Not required forgot to remove this from the CMakeLists was just testing something. Removing it.

mraduldubey · 2024-05-20T08:27:39Z

samples/timelapse-sample/CMakeLists.txt

+  ${LIBRE_INC_DIR}
+  ${NVCODEC_INCLUDE_DIR}
+)
+target_link_libraries(


If I remove this then how do I link the aprapipes lib with the samples?

remove the libraries that are not required by the sample boss

mraduldubey · 2024-05-20T08:27:49Z

samples/timelapse-sample/CMakeLists.txt

+find_library(APRA_PIPES_LIB NAMES aprapipes)
+add_library(example_aprapipes STATIC ${SOURCE})
+add_executable(runner_test ${UT_FILE})
+target_include_directories ( example_aprapipes PRIVATE


call it samples

Renaming it to timelapsesample would be better as there will be multiple samples with this name.

mraduldubey · 2024-05-20T08:28:49Z

samples/timelapse-sample/timelapse_summary.cpp

+int main() {
+  LoggerProps loggerProps;
+  loggerProps.logLevel = boost::log::trivial::severity_level::info;
+  Logger::setLogLevel(boost::log::trivial::severity_level::info);


both 79 & 80 are not required. one is enough

mraduldubey · 2024-05-20T08:29:43Z

samples/timelapse-sample/timelapse_summary.h

+  bool stopPipeline();
+
+private:
+  PipeLine pipeline;


add a method called test, and call from test added in timelapse-sample/ - it should use step - not the pipeline

mraduldubey · 2024-05-20T08:30:25Z

samples/timelapse-sample/timelapse_summary_test.cpp

+
+	timelapsePipeline -> startPipeline();
+
+	timelapsePipeline -> stopPipeline();


this is not a test. it has no checks. also, need to use step() instead

Done added an assertion to check the frame size.

…ad setting the RAW_IMAGE output pin to MotionVectorExtractor. 2. Enhanced the threshold logic to get the frames based on average of the vectors. 3. Removed not required tests.

mohammedzakikochargi · 2024-06-03T15:48:31Z

base/src/MotionVectorExtractor.cpp

-void DetailFfmpeg::getMotionVectors(frame_container& frames, frame_sp& outFrame, frame_sp& decodedFrame)
-{
+
+bool discardNoisyFrames(int black_pixel_percent, cv::Mat image) {


We should also check if the pixels are all white

mohammedzakikochargi · 2024-06-03T15:49:55Z

base/src/MotionVectorExtractor.cpp

-	{
+	if ((!sDecParam.bParseOnly) && (pDstInfo.pDst[0] != nullptr) &&
+		(mMotionVectorSize != mWidth * mHeight * 8) &&
+		(abs(avgX) > threshold || abs(avgY) > threshold)) {


Add a new bool prop and enable avg based thresholding only then

I discussed this with Mradul he suggested this should not be sample specific. The improvement should be for the module and not sample specific.

mohammedzakikochargi · 2024-06-03T15:50:33Z

base/src/MotionVectorExtractor.cpp

-
-		cv::cvtColor(yuvImgCV, bgrImg, cv::COLOR_YUV2BGR_I420);
-		frames.insert(make_pair(rawFramePinId, decodedFrame));
+		if (!discardNoisyFrames(97, bgrImg)) {


This should be also checked in the ffmpeg strategy

AdityaKBhadragond14 · 2024-06-05T08:58:23Z

I have addressed the comments please have a look @mohammedzakikochargi .

…ed comments.

mraduldubey

make some minor changes

mraduldubey · 2024-06-18T08:33:50Z

base/CMakeLists.txt

+  ${Boost_LIBRARIES}
+  ${FFMPEG_LIBRARIES}
+  ${OpenCV_LIBRARIES}
+  ${JETSON_LIBS}


do you need all libs in the aprapipessampleut ? I dont think so.

mraduldubey · 2024-06-18T08:47:11Z

samples/timelapse-sample/CMakeLists.txt

+  ${LIBRE_INC_DIR}
+  ${NVCODEC_INCLUDE_DIR}
+)
+target_link_libraries(


remove the libraries that are not required by the sample boss

mraduldubey · 2024-06-18T08:49:43Z

samples/timelapse-sample/timelapse_summary.cpp

+  mColorchange1 = boost::shared_ptr<ColorConversion>(new ColorConversion(
+      ColorConversionProps(ColorConversionProps::BGR_TO_RGB)));
+  //convert frames from RGB to YUV420PLANAR
+  //the two step color change is done because H264Encoder takes YUV data and we don't have direct BGR to YUV.


delete after YUV data

mraduldubey · 2024-06-18T08:50:47Z

samples/timelapse-sample/timelapse_summary.cpp

+  return true;
+}
+
+bool TimelapsePipeline::stopPipeline() {


format this file with Ctrl+k,f

mraduldubey · 2024-06-18T08:51:01Z

samples/timelapse-sample/timelapse_summary.cpp

+  //the two step color change is done because H264Encoder takes YUV data and we don't have direct BGR to YUV.
+  mColorchange2 = boost::shared_ptr<ColorConversion>(new ColorConversion(
+      ColorConversionProps(ColorConversionProps::RGB_TO_YUV420PLANAR)));
+  mSync = boost::shared_ptr<Module>(


why is sync required

sync is required by the encoder.

sample example for valve module as well as seek and flush queue features

…raPipes into ns/samples_timelapse

Sample: Face Detection

AdityaKBhadragond14 and others added 14 commits May 8, 2024 12:55

timelapse sample

e7b8c0b

timelaspe file changes

0b02ac2

wip sample

55235e4

samples WIP.

c116aad

added main to sample cpp file.

213bf2c

removed sample test file from tests.

5cb1025

added sample example to create thumbnail from mp4 video

9c70041

Improved motion vector data code.

2f3adb6

reduced thread sleep seconds

738763b

using absolute path.

5315e62

Reverting unwanted changes.

b89e3a6

added new sample which will start video from beginning using flushAll…

ab3f182

…Queues and seek feature

refactored CMake list and renamed files

f1d8cc0

using resolved time stamp when seeking

da51b44

mraduldubey reviewed May 20, 2024

View reviewed changes

SatwikSShanbhag and others added 15 commits May 20, 2024 15:20

Merge branch 'ab/timelapse_sample' into ss/sample_example

ad28405

Fixed the ghost channel issue in MotionVectorExtractor.

ad0752f

revert vcpkg.

005cd59

changes.

e9ccde5

taking video paths from terminal and modified cmake.

a5df32a

vcpkg baseline revert.

cce5ba2

reset vcpkg head.

87c26d2

vcpkg revert

4627b36

vcpkg revert.

31d8bf5

update submodule

518a75e

update vcpkg final

b45c423

Merge branch 'Apra-Labs:main' into ab/timelapse_sample

95bdf12

Corrected cmake for samples.

e0f0e91

Corrected base cmake for samples.

dc1c0a6

Running pipeline for face detection

29287b1

SatwikSShanbhag and others added 9 commits May 27, 2024 19:24

refactored test and CMake file

744d40d

Corrected cmake and test for timelapse sample.

574fc11

1. Removed the sendOverlayFrames prop to remove overlay frames, inste…

773a376

…ad setting the RAW_IMAGE output pin to MotionVectorExtractor. 2. Enhanced the threshold logic to get the frames based on average of the vectors. 3. Removed not required tests.

Algo to remove noisy black frames.

3c26a39

files format.

6dae9cd

refactored CMake and file structure for thumbnail samples

f499d5f

refactored CMake and file structure for play mp4 from beginning sample

53a3c59

base CMake changes

b26a871

Merge branch 'ab/timelapse_sample' into ss/sample_example

48f2556

mohammedzakikochargi reviewed Jun 3, 2024

View reviewed changes

Addressed comments.

f07bc4e

AdityaKBhadragond14 requested a review from mohammedzakikochargi June 5, 2024 08:57

nsabale7 and others added 2 commits June 14, 2024 14:27

changes in cmake

a2102f1

Added readme file for samples. code refactoring and renaming. Address…

06f679e

…ed comments.

AdityaKBhadragond14 requested a review from mraduldubey June 17, 2024 10:29

mraduldubey requested changes Jun 18, 2024

View reviewed changes

removed not required libs and file formatting.

133d93b

AdityaKBhadragond14 requested a review from mraduldubey June 18, 2024 10:45

SatwikSShanbhag and others added 11 commits June 18, 2024 19:37

Merge branch 'ab/timelapse_sample' into ss/sample_example

5888659

restored files

fab05b3

Merge branch 'main' into ab/timelapse_sample

d920279

Merge branch 'ab/timelapse_sample' into ss/sample_example

7478f05

Merge pull request #2 from SatwikSShanbhag/ss/sample_example

7f79097

sample example for valve module as well as seek and flush queue features

Merge branch 'ab/timelapse_sample' into ns/samples_timelapse

1c2fa9d

changes in constructor

064cce5

Merge branch 'ns/samples_timelapse' of https://github.com/nsabale7/Ap…

197b75d

…raPipes into ns/samples_timelapse

Merge pull request #3 from nsabale7/ns/samples_timelapse

1031260

Sample: Face Detection

relay sample.

8cbb00b

base cmake changes.

61690e5


		timelapsePipeline -> startPipeline();

		timelapsePipeline -> stopPipeline();

Timelapse Sample. #352

Are you sure you want to change the base?

Timelapse Sample. #352

Conversation

AdityaKBhadragond14 commented May 15, 2024 • edited Loading

mraduldubey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mohammedzakikochargi Jun 3, 2024 • edited Loading

Choose a reason for hiding this comment

AdityaKBhadragond14 commented Jun 5, 2024

mraduldubey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AdityaKBhadragond14 commented May 15, 2024 •

edited

Loading

mohammedzakikochargi Jun 3, 2024 •

edited

Loading