Skip to content
This repository was archived by the owner on Nov 21, 2022. It is now read-only.
This repository was archived by the owner on Nov 21, 2022. It is now read-only.

TextClassificationTransformer should log torchmetrics object instead of computed Tensors #276

@stefan-schroedl

Description

@stefan-schroedl

🐛 Bug

In line 58 of the TextClassificationTransformer.common_step() method (https://github.com/Lightning-AI/lightning-transformers/blob/master/lightning_transformers/task/nlp/text_classification/model.py#L58), Logging is called with a dictionary of metric values computed for the current batch. I am new to this package but I believe the logger has to be called with the torchmetrics.Metric subclass for cases where the computed value is different from the average of the per-batch values, such as for class Precision - otherwise the aggregation gives wrong results. Similar code exists in TokenClassificationTransformer and ImageClassificationTransformer as well.

To Reproduce

Using TextClassificationTransformer with Precision and Recall metrics (as configured in the default) will result in inaccurate per-epoch values for validation and testing.

Environment

  • PyTorch Version: 1.12.1
  • OS: Linux
  • How you installed PyTorch (conda, pip, source): pip
  • Build command you used (if compiling from source):
  • Python version: 3.7
  • CUDA/cuDNN version:

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is neededquestionFurther information is requested

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions