Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] Fix correct definition of torchmetrics inside pytorch lightning module #1365

Merged
merged 14 commits into from Aug 14, 2023

Conversation

leoniewgnr
Copy link
Collaborator

🔬 Background

See #1361

🔮 Key changes

Move metrics definition inside pytorch lightning module as described in https://torchmetrics.readthedocs.io/en/latest/pages/lightning.html

@leoniewgnr leoniewgnr self-assigned this Jul 4, 2023
@leoniewgnr leoniewgnr added the status: in development Pull requests which are in development label Jul 4, 2023
@github-actions
Copy link

github-actions bot commented Jul 4, 2023

Model Benchmark

Benchmark Metric main current diff
YosemiteTemps MAE_val 1.3442 1.3442 -0.0%
YosemiteTemps RMSE_val 2.00245 2.00245 -0.0%
YosemiteTemps Loss_val 0.00077 0.00077 -0.0%
YosemiteTemps MAE 1.3192 1.3192 0.0%
YosemiteTemps RMSE 2.13518 2.13518 -0.0%
YosemiteTemps Loss 0.00064 0.00064 -0.0%
YosemiteTemps time 98.4182 76.36 -22.41% 🎉
PeytonManning MAE_val 0.58159 0.58159 -0.0%
PeytonManning RMSE_val 0.72216 0.72216 0.0%
PeytonManning Loss_val 0.01239 0.01239 -0.0%
PeytonManning MAE 0.41671 0.41671 -0.0%
PeytonManning RMSE 0.55961 0.55961 0.0%
PeytonManning Loss 0.00612 0.00612 0.0%
PeytonManning time 20.9439 15.34 -26.76% 🎉
AirPassengers MAE_val 13.0626 13.0626 0.0%
AirPassengers RMSE_val 15.9452 15.9453 0.0%
AirPassengers Loss_val 0.00131 0.00131 0.0%
AirPassengers MAE 9.88156 9.88156 0.0%
AirPassengers RMSE 11.7354 11.7354 0.0%
AirPassengers Loss 0.00052 0.00052 0.0%
AirPassengers time 8.61348 6.61 -23.26% 🎉
Model training plots

Model Training

PeytonManning

YosemiteTemps

AirPassengers

@codecov
Copy link

codecov bot commented Jul 5, 2023

Codecov Report

Merging #1365 (b1c50ee) into main (90f6e6e) will increase coverage by 0.02%.
Report is 1 commits behind head on main.
The diff coverage is 90.90%.

@@            Coverage Diff             @@
##             main    #1365      +/-   ##
==========================================
+ Coverage   89.81%   89.84%   +0.02%     
==========================================
  Files          38       38              
  Lines        5058     5061       +3     
==========================================
+ Hits         4543     4547       +4     
+ Misses        515      514       -1     
Files Changed Coverage Δ
neuralprophet/forecaster.py 87.40% <ø> (ø)
neuralprophet/utils_metrics.py 80.76% <87.50%> (+15.55%) ⬆️
neuralprophet/np_types.py 92.30% <100.00%> (-0.55%) ⬇️
neuralprophet/time_net.py 90.63% <100.00%> (-0.45%) ⬇️

@leoniewgnr leoniewgnr marked this pull request as ready for review July 5, 2023 23:11
@leoniewgnr leoniewgnr added status: needs review PR needs to be reviewed by Reviewer(s) and removed status: in development Pull requests which are in development labels Jul 5, 2023
Copy link
Owner

@ourownstory ourownstory left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good mostly!
We just need make sure to only have one source for the list of METRICS.
I also added some other open Qs

neuralprophet/time_net.py Outdated Show resolved Hide resolved
neuralprophet/utils_metrics.py Outdated Show resolved Hide resolved
neuralprophet/utils_metrics.py Show resolved Hide resolved
neuralprophet/time_net.py Outdated Show resolved Hide resolved
@hxyue1 hxyue1 self-requested a review August 14, 2023 20:07
@leoniewgnr leoniewgnr merged commit 4c31824 into main Aug 14, 2023
14 checks passed
@leoniewgnr leoniewgnr deleted the correct-definition-torchmetrics branch August 14, 2023 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: needs review PR needs to be reviewed by Reviewer(s)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants