Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors & improvements in Metrics descriptions #1923

Closed
robmarkcole opened this issue Mar 3, 2024 · 2 comments · Fixed by #1924
Closed

Errors & improvements in Metrics descriptions #1923

robmarkcole opened this issue Mar 3, 2024 · 2 comments · Fixed by #1924
Labels
documentation Improvements or additions to documentation trainers PyTorch Lightning trainers

Comments

@robmarkcole
Copy link
Contributor

robmarkcole commented Mar 3, 2024

Description

  • In segmentation trainer, configure_metrics, Multiclass Jaccard Index (IoU) is described as using macro but it is using micro
  • In classification, MultiLabelClassificationTask, OA mentions pixels, but it should refer to image level metrics such as true and false positive
  • Regression metrics cannot take macro/micro as args, perhaps we remove the Note to avoid confusion
  • Detection, MeanAveragePrecision computes many metrics. Run the example at https://github.com/Lightning-AI/torchmetrics/blob/master/examples/detection_map.py returns
0s
result
{'map': tensor(0.6000),
 'map_50': tensor(1.),
 'map_75': tensor(1.),
 'map_small': tensor(-1.),
 'map_medium': tensor(-1.),
 'map_large': tensor(0.6000),
 'mar_1': tensor(0.6000),
 'mar_10': tensor(0.6000),
 'mar_100': tensor(0.6000),
 'mar_small': tensor(-1.),
 'mar_medium': tensor(-1.),
 'mar_large': tensor(0.6000),
 'map_per_class': tensor(-1.),
 'mar_100_per_class': tensor(-1.),
 'classes': tensor(0, dtype=torch.int32)}
  • Note, outside the scope of this issue, but we do not pass dist_sync_on_step - we could and default to False, but allow setting true for distributed training with Lightning

CC @jdilger

Steps to reproduce

NA

Version

dev

@robmarkcole robmarkcole changed the title Error in Metrics Errors & improvements in Metrics descriptions Mar 3, 2024
@adamjstewart
Copy link
Collaborator

We had to delay the release until today so we actually still have time to include this if someone quickly opens a PR.

@robmarkcole
Copy link
Contributor Author

I would love to but once my kids are up, my plate is full for the day!

@adamjstewart adamjstewart added documentation Improvements or additions to documentation trainers PyTorch Lightning trainers labels Mar 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation trainers PyTorch Lightning trainers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants