Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/docs/CPP-Streaming-Component-API.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ Process a single video frame for the current segment.

Must return true when the component begins generating the first track for the current segment. After it returns true, the Component Executable will ignore the return value until the component begins processing the next segment.

If the `job_properties` map contained in the `MPFStreamingVideoJob` struct passed to the component constructor contains a `QUALITY_SELECTION_THRESHOLD` entry, then this function should only return true for a detection with a quality value that meets or exceeds that threshold. Refer to the [Quality Selection Guide](Quality-Selection-Guide/index.html). After the Component Executable invokes `EndSegment()` to retrieve the segment tracks, it will discard detections that are below the threshold. If all the detections in a track are below the threshold, then the entire track will be discarded.
If the `job_properties` map contained in the `MPFStreamingVideoJob` struct passed to the component constructor contains a `CONFIDENCE_THRESHOLD` entry, then this function should only return true for a detection with a quality value that meets or exceeds that threshold. After the Component Executable invokes `EndSegment()` to retrieve the segment tracks, it will discard detections that are below the threshold. If all the detections in a track are below the threshold, then the entire track will be discarded. [NOTE: In the future the C++ Streaming Component API may be updated to support `QUALITY_SELECTION_THRESHOLD` instead of `CONFIDENCE_THRESHOLD`.]

Note that this function may not be invoked for every frame in the current segment. For example, if `FRAME_INTERVAL = 2`, then this function will only be invoked for every other frame since those are the only ones that need to be processed.

Expand Down
39 changes: 16 additions & 23 deletions docs/docs/Quality-Selection-Guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,8 @@ example, a face detection component may generate detections with a `DESCRIPTOR_M
quality of the face embedding and how useful it is for reidentification. The Workflow Manager will search the
`detection_properties` map in each detection and track for that key and use the corresponding value as the detection
quality. The value associated with this property must be an integer or floating point value, where higher values
indicate higher quality.

One exception is when this property is set to `CONFIDENCE` and no `CONFIDENCE` property exists in the
`detection_properties` map. Then the `confidence` member of each detection and track is used instead.
indicate higher quality. The one exception is that if this property is set to `CONFIDENCE`, then the `confidence` member
of each detection and track is used to determine quality.

The primary way in which OpenMPF uses detection quality is to determine the track "exemplar", which is the highest
quality detection in the track. For components that do not compute a quality value, or where all detections have
Expand All @@ -48,23 +46,18 @@ the `QUALITY_SELECTION_THRESHOLD`, then only that many artifacts will be extract
# Hybrid Quality Selection

In some cases, there may be a detection property that a component would like to use as a measure of quality but it
doesn't lend itself to simple thresholding. For example, a face detector might be able to calculate the face pose, and
would like to select faces that are in the most frontal pose as the highest quality detections. The yaw of the face pose
may be used to indicate this, but if it's values are between, say, -90 degrees and +90 degrees, then the highest quality
detection would be the one with a value of yaw closest to 0. This violates the need for the quality selection property
to take on a range of values where the highest value indicates the highest quality.

Another use case might be where the component would like to choose detections based on a set of quality values, or
properties. Continuing with the face pose example, the component might like to designate the detection with pose closest
to frontal as the highest quality, but would also like to assign high quality to detections where the pose is closest to
profile, meaning values of yaw closest to -90 or +90 degrees.

In both of these cases, the component can create a custom detection property that is used to rank these detections as it
sees fit. It could use a detection property called `RANK`, and assign values to that property to rank the detections
from lowest to highest quality. In the example of the face detector wanting to use the yaw of the face pose, the
detection with a value of yaw closest to 0 would be assigned a `RANK` property with the highest value, then the
detections with values of yaw closest to +/-90 degrees would be assigned the second and third highest values of `RANK`.
Detections without the `RANK` property would be treated as having the lowest possible quality value. Thus, the track
exemplar would be the face with the frontal pose, and the `ARTIFACT_EXTRACTION_POLICY_TOP_QUALITY_COUNT` property could
be set to 3, so that the frontal and two profile pose detections would be kept as track artifacts.
doesn't lend itself to simple thresholding, perhaps because its value is not linearly increasing, or it is not numeric. The
component can in this case create a custom property that represents the quality of detections using a numerical value that
corresponds to the ordering of the detections from low to high quality.

As a simple example, a face detector might be able to calculate the face pose and would like to select for artifact
extraction the face that is closest to frontal pose, and the two that are closest to left and right profile pose. If the face
detector computes the yaw with values between -90 degrees and +90 degrees, then the numerical order of those values would
not produce the desired result. In this case, the component could create a custom detection property called `RANK`, and
assign values to that property that orders the detections from highest to lowest quality. The face detection component would
assign the highest value of `RANK` to the detection with a value of yaw closest to 0, and the detections with values of yaw
closest to +/-90 degrees would be assigned the second and third highest values of `RANK`. Detections without the `RANK`
property would be treated as having the lowest possible quality value. Thus, the track exemplar would be the face with the
frontal pose, and the `ARTIFACT_EXTRACTION_POLICY_TOP_QUALITY_COUNT` property would be set to 3, so that the frontal and
two profile pose detections would be kept as track artifacts in addition to the exemplar.

2 changes: 1 addition & 1 deletion docs/site/CPP-Streaming-Component-API/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -449,7 +449,7 @@ <h3 id="beginsegmentvideosegmentinfo">BeginSegment(VideoSegmentInfo)</h3>
<h3 id="processframemat">ProcessFrame(Mat ...)</h3>
<p>Process a single video frame for the current segment.</p>
<p>Must return true when the component begins generating the first track for the current segment. After it returns true, the Component Executable will ignore the return value until the component begins processing the next segment.</p>
<p>If the <code>job_properties</code> map contained in the <code>MPFStreamingVideoJob</code> struct passed to the component constructor contains a <code>QUALITY_SELECTION_THRESHOLD</code> entry, then this function should only return true for a detection with a quality value that meets or exceeds that threshold. Refer to the <a href="../Quality-Selection-Guide/index.html">Quality Selection Guide</a>. After the Component Executable invokes <code>EndSegment()</code> to retrieve the segment tracks, it will discard detections that are below the threshold. If all the detections in a track are below the threshold, then the entire track will be discarded.</p>
<p>If the <code>job_properties</code> map contained in the <code>MPFStreamingVideoJob</code> struct passed to the component constructor contains a <code>CONFIDENCE_THRESHOLD</code> entry, then this function should only return true for a detection with a quality value that meets or exceeds that threshold. After the Component Executable invokes <code>EndSegment()</code> to retrieve the segment tracks, it will discard detections that are below the threshold. If all the detections in a track are below the threshold, then the entire track will be discarded. [NOTE: In the future the C++ Streaming Component API may be updated to support <code>QUALITY_SELECTION_THRESHOLD</code> instead of <code>CONFIDENCE_THRESHOLD</code>.]</p>
<p>Note that this function may not be invoked for every frame in the current segment. For example, if <code>FRAME_INTERVAL = 2</code>, then this function will only be invoked for every other frame since those are the only ones that need to be processed.</p>
<p>Also, it may not be invoked for the first nor last frame in the segment. For example, if <code>FRAME_INTERVAL = 3</code> and the segment size is 10, then it will be invoked for frames {0, 3, 6, 9} for the first segment, and frames {12, 15, 18} for the second segment.</p>
<ul>
Expand Down
35 changes: 15 additions & 20 deletions docs/site/Quality-Selection-Guide/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -259,9 +259,8 @@ <h1 id="quality-selection-properties">Quality Selection Properties</h1>
quality of the face embedding and how useful it is for reidentification. The Workflow Manager will search the
<code>detection_properties</code> map in each detection and track for that key and use the corresponding value as the detection
quality. The value associated with this property must be an integer or floating point value, where higher values
indicate higher quality.</p>
<p>One exception is when this property is set to <code>CONFIDENCE</code> and no <code>CONFIDENCE</code> property exists in the
<code>detection_properties</code> map. Then the <code>confidence</code> member of each detection and track is used instead.</p>
indicate higher quality. The one exception is that if this property is set to <code>CONFIDENCE</code>, then the <code>confidence</code> member
of each detection and track is used to determine quality.</p>
<p>The primary way in which OpenMPF uses detection quality is to determine the track "exemplar", which is the highest
quality detection in the track. For components that do not compute a quality value, or where all detections have
identical quality, the Workflow Manager will choose the first detection in the track as the exemplar.</p>
Expand All @@ -281,23 +280,19 @@ <h1 id="quality-selection-properties">Quality Selection Properties</h1>
the <code>QUALITY_SELECTION_THRESHOLD</code>, then only that many artifacts will be extracted.</p>
<h1 id="hybrid-quality-selection">Hybrid Quality Selection</h1>
<p>In some cases, there may be a detection property that a component would like to use as a measure of quality but it
doesn't lend itself to simple thresholding. For example, a face detector might be able to calculate the face pose, and
would like to select faces that are in the most frontal pose as the highest quality detections. The yaw of the face pose
may be used to indicate this, but if it's values are between, say, -90 degrees and +90 degrees, then the highest quality
detection would be the one with a value of yaw closest to 0. This violates the need for the quality selection property
to take on a range of values where the highest value indicates the highest quality.</p>
<p>Another use case might be where the component would like to choose detections based on a set of quality values, or
properties. Continuing with the face pose example, the component might like to designate the detection with pose closest
to frontal as the highest quality, but would also like to assign high quality to detections where the pose is closest to
profile, meaning values of yaw closest to -90 or +90 degrees.</p>
<p>In both of these cases, the component can create a custom detection property that is used to rank these detections as it
sees fit. It could use a detection property called <code>RANK</code>, and assign values to that property to rank the detections
from lowest to highest quality. In the example of the face detector wanting to use the yaw of the face pose, the
detection with a value of yaw closest to 0 would be assigned a <code>RANK</code> property with the highest value, then the
detections with values of yaw closest to +/-90 degrees would be assigned the second and third highest values of <code>RANK</code>.
Detections without the <code>RANK</code> property would be treated as having the lowest possible quality value. Thus, the track
exemplar would be the face with the frontal pose, and the <code>ARTIFACT_EXTRACTION_POLICY_TOP_QUALITY_COUNT</code> property could
be set to 3, so that the frontal and two profile pose detections would be kept as track artifacts.</p>
doesn't lend itself to simple thresholding, perhaps because its value is not linearly increasing, or it is not numeric. The
component can in this case create a custom property that represents the quality of detections using a numerical value that
corresponds to the ordering of the detections from low to high quality.</p>
<p>As a simple example, a face detector might be able to calculate the face pose and would like to select for artifact
extraction the face that is closest to frontal pose, and the two that are closest to left and right profile pose. If the face
detector computes the yaw with values between -90 degrees and +90 degrees, then the numerical order of those values would
not produce the desired result. In this case, the component could create a custom detection property called <code>RANK</code>, and
assign values to that property that orders the detections from highest to lowest quality. The face detection component would
assign the highest value of <code>RANK</code> to the detection with a value of yaw closest to 0, and the detections with values of yaw
closest to +/-90 degrees would be assigned the second and third highest values of <code>RANK</code>. Detections without the <code>RANK</code>
property would be treated as having the lowest possible quality value. Thus, the track exemplar would be the face with the
frontal pose, and the <code>ARTIFACT_EXTRACTION_POLICY_TOP_QUALITY_COUNT</code> property would be set to 3, so that the frontal and
two profile pose detections would be kept as track artifacts in addition to the exemplar.</p>

</div>
</div>
Expand Down
2 changes: 1 addition & 1 deletion docs/site/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -400,5 +400,5 @@ <h1 id="overview">Overview</h1>

<!--
MkDocs version : 0.17.5
Build Date UTC : 2024-03-25 21:18:57
Build Date UTC : 2024-03-29 13:21:30
-->
Loading