-
Notifications
You must be signed in to change notification settings - Fork 437
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add F1 metric computation during detection tasks #3539
Add F1 metric computation during detection tasks #3539
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that we still need to work more than this PR:
- If we introduce
MetricCollection
, the metric key structure will be changed.
This will affect to the checkpoint callback and the lr scheduler behaviors:
monitor: val/map_50 |
callback_monitor: val/map_50 |
- We still need a dedicated class for this mentioned in Add a new metric for computing both f-measure and mean AP for detection training #3531 (comment) This is because we have to fix
best_confidence_threshold
during test time. To fix this value, we've been passingbest_confidence_threshold
viameter.compute(best_confidence_threshold)
.
According to both problem above, I recommend to add a dedicated metric class.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, if this new metric is just for train, not test, what about checking and overriding it to FMeasure
when used on testing at configure_metric
if possible?
training_extensions/src/otx/core/model/base.py
Lines 300 to 311 in 772e6c5
def configure_metric(self) -> None: | |
"""Configure the metric.""" | |
if not callable(self.metric_callable): | |
raise TypeError(self.metric_callable) | |
metric = self.metric_callable(self.label_info) | |
if not isinstance(metric, (Metric, MetricCollection)): | |
msg = "Metric should be the instance of `torchmetrics.Metric` or `torchmetrics.MetricCollection`." | |
raise TypeError(msg, metric) | |
self._metric = metric.to(self.device) |
Co-authored-by: Vinnam Kim <vinnam.kim@gmail.com>
* Remove LITMODULE_PER_TASK * Disable Resnext101_ATSS model on XPU (#3514) * diable resnext101_atss on XPU * give detailed xpu device info to perf tag * revert debug code --------- Co-authored-by: kirill prokofiev <kirill.prokofiev@intel.com> * Remove invalid perf benchmark reference history for v2.0.0 (#3517) * Fix a bug that dino_v2 model can't be run w/ HPO (#3518) * add reduce function to dino backbone * add unit test * update integration test * change name * Fix detection export performance degradation (#3520) * Fix to use right index * Align forward by adding missed parts * Revert metrics threshold (#3528) Revert threshold Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com> * Tile with full img optional (#3530) * make tile dataset with full image optional * Add a feature to adapt max value of HPO batch size search space (#3532) * implement adaptive max value of bs search space * implement unit test * Remove duplicates in get_idx_list_per_classes (#3537) * Remove duplicates in get_idx_list_per_classes * Reduce time complex * Revisit Docker image build script (#3536) * Revisit Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> * Fix test Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> --------- Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> * Bump datumaro version to 1.6.1 (#3535) * Add F1 metric computation during detection tasks (#3539) * add f1 metric during training for detection * add f1 metric during training for detection * add a new metric * remove configure_metric in det model * Update doctstring for MeanAveragePrecisionFMeasure Co-authored-by: Vinnam Kim <vinnam.kim@gmail.com> * Trigger Build * resolve precommit error --------- Co-authored-by: Vinnam Kim <vinnam.kim@gmail.com> * Fix yolox export perf degradation (#3534) * Add `Focus` export pipeline * Update `update_ov_subset_pipeline` to update `image_color_channel` * Fix MaskRCNN IR Accuracy Drop (#3540) fix ir maskrcnn accuracy drop * Hotfix/geti integration (#3543) * Fix Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> * Fix Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> --------- Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> * Fix Anomaly OV export flag (#3558) swap_rgb to False Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com> * Fix accuracy drop in MaskRCNN (#3562) * fix maskrcnn accuracy drop * pad to square * remove unnecessary changes * remove unnecessary changes * Update codeql workflow to generate a report (#3559) * Fix & Refine HPO (#3565) * refine engine/hpo * implement draft test code * change replace to shutil.copy * use same initial weight during HPO * implement unit test * search_space support Path * move Path out of TYPE_CHECKING * Fix Optimize in Anomaly Task (#3561) * use val_dataloader Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com> * Apply suggestions from code review Co-authored-by: Harim Kang <harim.kang@intel.com> --------- Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com> Co-authored-by: Harim Kang <harim.kang@intel.com> * Update dependencies (#3570) * Bump anomalib from 1.0.1 to 1.1.0 (#3572) Bump anomalib version to 1.1.0 * Fix F1 instance seg accuracy drop (#3578) * add MaskRLEMeanAPFMeasureCallable * update recipes * format * Update for using locally stored pretrained weights in the CI for torch hub models (#3541) * apply changed arg of OTXModel.export() to YOLOX model * update dinov2 pretrained weights loading * skip yolox-tiny-tile xai test * make semi-sl dino support pickle --------- Signed-off-by: Ashwin Vaidya <ashwinnitinvaidya@gmail.com> Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> Co-authored-by: Kang, Harim <harim.kang@intel.com> Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com> Co-authored-by: kirill prokofiev <kirill.prokofiev@intel.com> Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com> Co-authored-by: Ashwin Vaidya <ashwin.vaidya@intel.com> Co-authored-by: Eugene Liu <eugene.liu@intel.com> Co-authored-by: Vinnam Kim <vinnam.kim@intel.com> Co-authored-by: Wonju Lee <wonju.lee@intel.com> Co-authored-by: Vinnam Kim <vinnam.kim@gmail.com>
Summary
How to test
Checklist
License
Feel free to contact the maintainers if that's a concern.