Updated tracker and eval to compute timely accuracy. #160
Conversation
There was a problem hiding this comment.
Tracking implementation looks good besides minor fixing. It would be worth adding a docstring comment explaining that the tracker implementation only covers the case when the tracker runtime < frame gap.
There's also the issue about anchoring (not all frames' ground truths are evaluated on), which I think we should address
| self._csv_logger.info('{},{},{},{},{}'.format( | ||
| time_epoch_ms(), sim_time, self.config.name, 'runtime', | ||
| runtime)) | ||
| # Write metrics to csv log file | ||
| self.__write_metrics_to_csv(metrics_summary_df, sim_time) |
There was a problem hiding this comment.
Regardless of the anchoring (start or finish) we should avoid using sim_time because it's neither: if the queue gets backed up multiple (start, end) inferences will be processed and logged with the same sim_time.
| @@ -75,26 +79,53 @@ def on_watermark(self, timestamp, obstacle_tracking_stream): | |||
| frame_msg = self._frame_msgs.popleft() | |||
There was a problem hiding this comment.
If all streams send a watermark.is_top at the very end (after a watermark for every message is sent), when the watermark.is_top is sent this function will get invoked, and pop_left will raise an error because the buffer is empty.
To get around this we should first explicitly agree on a universal procedure for sending watermark.is_top (either set the watermark coming after the last message to be is_top or sending a separate watermark.is_top message afterwards). If we do the latter then we can add an if statement before the frame_msg.popleft that forwards the top watermark and returns.
There was a problem hiding this comment.
Let's assume that there's a special top watermark, that comes last. I'll add an if statement to the function.
| metrics_summary_df = self.__get_tracker_metrics( | ||
| tracker_obstacles, ground_obstacles, | ||
| self._end_anchroned_accumulator) | ||
| self.__write_metrics_to_csv(metrics_summary_df, ground_time) |
There was a problem hiding this comment.
| self.__write_metrics_to_csv(metrics_summary_df, ground_time) | |
| self.__write_metrics_to_csv(metrics_summary_df, "end", ground_time) |
No description provided.