overall MAPE for STNorm on PEMS07 #51
Comments
|
After checking the training log, ST-Norm's overall MAPE is indeed problematic. 2022-11-03 12:55:25,075 - easytorch-training - INFO - Epoch 95 / 100
2022-11-03 12:56:18,227 - easytorch-training - INFO - Result <train>: [train_time: 53.15 (s), lr: 5.00e-04, train_MAE: 18.2307, train_RMSE: 30.4419, train_MAPE: 0.0808 ]
2022-11-03 12:56:18,228 - easytorch-training - INFO - Start validation.
2022-11-03 12:56:22,276 - easytorch-training - INFO - Result <val>: [val_time: 4.05 (s), val_MAE: 19.8844, val_RMSE: 32.5213, val_MAPE: 0.0858]
2022-11-03 12:56:22,307 - easytorch-training - INFO - Checkpoint checkpoints/STNorm_100/8cb47c273d8e73528d8c8058998a0860/STNorm_best_val_MAE.pt saved
2022-11-03 12:56:26,161 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 17.2057, Test RMSE: 27.4493, Test MAPE: 0.0720
2022-11-03 12:56:26,164 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 18.3410, Test RMSE: 29.7707, Test MAPE: 0.0766
2022-11-03 12:56:26,166 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 19.1426, Test RMSE: 31.5075, Test MAPE: 0.0798
2022-11-03 12:56:26,169 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 19.7441, Test RMSE: 32.8585, Test MAPE: 0.0830
2022-11-03 12:56:26,172 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 20.2191, Test RMSE: 34.0451, Test MAPE: 0.0849
2022-11-03 12:56:26,174 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 20.6346, Test RMSE: 34.9905, Test MAPE: 0.0861
2022-11-03 12:56:26,177 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 21.0152, Test RMSE: 35.8100, Test MAPE: 0.0873
2022-11-03 12:56:26,180 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 21.3745, Test RMSE: 36.5240, Test MAPE: 0.0892
2022-11-03 12:56:26,182 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 21.6551, Test RMSE: 37.0576, Test MAPE: 0.0901
2022-11-03 12:56:26,185 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 21.9236, Test RMSE: 37.5292, Test MAPE: 0.0913
2022-11-03 12:56:26,187 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 22.1922, Test RMSE: 38.0034, Test MAPE: 0.0926
2022-11-03 12:56:26,190 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 22.5857, Test RMSE: 38.5200, Test MAPE: 0.0942
2022-11-03 12:56:26,218 - easytorch-training - INFO - Result <test>: [test_time: 3.91 (s), test_MAE: 20.5028, test_RMSE: 34.6671, test_MAPE: 0.0856]
2022-11-03 12:56:26,243 - easytorch-training - INFO - Checkpoint checkpoints/STNorm_100/8cb47c273d8e73528d8c8058998a0860/STNorm_095.pt savedThanks very much for reporting this issue, and I will update the results. |
|
The results have been updated in the commit 08997c2. |
Thanks for your quick response! I also found Graph WaveNet's MAPE at Horizon 6 (8.24%) is extremely low(even smaller than which at Horizon 3 for 8.63%). There might be similar problem for DCRNN's MAPE at Horizon 3 (6.86%), which is better than MTGNN's performance(6.89%). However, DCRNN's performance is much worse than MTGNN under other Horizons. I can't reproduce it either. |
|
Thanks very much for your further suggestions. 2022-11-01 17:27:27,131 - easytorch-training - INFO - Epoch 180 / 200
2022-11-01 17:27:37,069 - easytorch-training - INFO - Result <train>: [train_time: 9.94 (s), lr: 2.50e-04, train_MAE: 14.0349, train_RMSE: 23.2042, train_MAPE: 0.0936]
2022-11-01 17:27:37,070 - easytorch-training - INFO - Start validation.
2022-11-01 17:27:38,466 - easytorch-training - INFO - Result <val>: [val_time: 1.40 (s), val_MAE: 14.6842, val_RMSE: 23.8004, val_MAPE: 0.1044]
2022-11-01 17:27:38,491 - easytorch-training - INFO - Checkpoint checkpoints/GraphWaveNet_200/983887ce40ff7d9ff3cf94dddf1cd557/GraphWaveNet_best_val_MAE.pt saved
2022-11-01 17:27:39,932 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 12.4887, Test RMSE: 19.5742, Test MAPE: 0.0803
2022-11-01 17:27:39,933 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 13.0537, Test RMSE: 20.7378, Test MAPE: 0.0835
2022-11-01 17:27:39,934 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 13.5009, Test RMSE: 21.6306, Test MAPE: 0.0863
2022-11-01 17:27:39,935 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 13.8442, Test RMSE: 22.3552, Test MAPE: 0.0882
2022-11-01 17:27:39,936 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 14.1508, Test RMSE: 22.9266, Test MAPE: 0.0902
2022-11-01 17:27:39,937 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 14.4118, Test RMSE: 23.4517, Test MAPE: 0.0924
2022-11-01 17:27:39,937 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 14.6821, Test RMSE: 23.9416, Test MAPE: 0.0937
2022-11-01 17:27:39,938 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 14.9098, Test RMSE: 24.3487, Test MAPE: 0.0953
2022-11-01 17:27:39,939 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 15.1343, Test RMSE: 24.6971, Test MAPE: 0.0969
2022-11-01 17:27:39,940 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 15.3347, Test RMSE: 25.0472, Test MAPE: 0.0983
2022-11-01 17:27:39,941 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 15.5267, Test RMSE: 25.3649, Test MAPE: 0.0992
2022-11-01 17:27:39,942 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 15.8140, Test RMSE: 25.7723, Test MAPE: 0.1011
2022-11-01 17:27:39,946 - easytorch-training - INFO - Result <test>: [test_time: 1.45 (s), test_MAE: 14.4043, test_RMSE: 23.3941, test_MAPE: 0.0921]
2022-11-01 17:27:39,970 - easytorch-training - INFO - Checkpoint checkpoints/GraphWaveNet_200/983887ce40ff7d9ff3cf94dddf1cd557/GraphWaveNet_180.pt saved
2022-11-01 17:27:39,977 - easytorch-training - INFO - The estimated training finish time is 2022-11-01 17:31:55The performance of DCRNN on METR-LA is correct. DCRNN outperforms Graph WaveNet and MTGNN at short-term horizons. Here is the training log: 2022-11-02 12:24:40,516 - easytorch-training - INFO - Epoch 94 / 100
2022-11-02 12:28:57,768 - easytorch-training - INFO - Result <train>: [train_time: 257.25 (s), lr: 1.00e-06, train_MAE: 2.7032, train_RMSE: 5.5099, train_MAPE: 0.0711]
2022-11-02 12:28:57,768 - easytorch-training - INFO - Start validation.
2022-11-02 12:29:06,105 - easytorch-training - INFO - Result <val>: [val_time: 8.34 (s), val_MAE: 2.7504, val_RMSE: 5.4149, val_MAPE: 0.0765]
2022-11-02 12:29:06,115 - easytorch-training - INFO - Checkpoint checkpoints/DCRNN_100/90729dd9a08d58d63e6c9cea877b0df1/DCRNN_best_val_MAE.pt saved
2022-11-02 12:29:21,572 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 2.1762, Test RMSE: 3.7696, Test MAPE: 0.0519
2022-11-02 12:29:21,574 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 2.4718, Test RMSE: 4.5959, Test MAPE: 0.0615
2022-11-02 12:29:21,575 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 2.6695, Test RMSE: 5.1683, Test MAPE: 0.0686
2022-11-02 12:29:21,576 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 2.8237, Test RMSE: 5.6119, Test MAPE: 0.0745
2022-11-02 12:29:21,577 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 2.9533, Test RMSE: 5.9699, Test MAPE: 0.0797
2022-11-02 12:29:21,578 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 3.0657, Test RMSE: 6.2738, Test MAPE: 0.0842
2022-11-02 12:29:21,579 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 3.1666, Test RMSE: 6.5409, Test MAPE: 0.0882
2022-11-02 12:29:21,580 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 3.2560, Test RMSE: 6.7684, Test MAPE: 0.0917
2022-11-02 12:29:21,581 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 3.3370, Test RMSE: 6.9717, Test MAPE: 0.0950
2022-11-02 12:29:21,582 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 3.4080, Test RMSE: 7.1505, Test MAPE: 0.0978
2022-11-02 12:29:21,583 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 3.4761, Test RMSE: 7.3172, Test MAPE: 0.1006
2022-11-02 12:29:21,584 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 3.5426, Test RMSE: 7.4758, Test MAPE: 0.1032
2022-11-02 12:29:21,592 - easytorch-training - INFO - Result <test>: [test_time: 15.48 (s), test_MAE: 3.0289, test_RMSE: 6.2336, test_MAPE: 0.0831]
2022-11-02 12:29:21,607 - easytorch-training - INFO - Checkpoint checkpoints/DCRNN_100/90729dd9a08d58d63e6c9cea877b0df1/DCRNN_094.pt saved |
|
I will update the results later. |
|
The results have been updated, and I have fixed a typo in DCRNN configs on metr-la and pems03 datasets. Now you should can reproduce the results. |
|
Thanks for that! That would solved my problem. Looking forward you checking the metrics at a proper time. |
|
You are welcome. I have tested DCRNN on the mete-la dataset, and the results are the same as mentioned above. |
|
I have no more questions for now. Thanks again for your excellent work! |
|
You are welcome! If you have any other questions, please feel free to create a new issue, and I will reply quickly. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanks for sharing your excellent work!
When I estimate the performance of my work with BasicTS, I found the overall MAPE of STNorm on PEMS07 is extremely low to 8.00%, which is almost the MAPE of third horizon(7.98%). Also, I cannot reproduce the performance but get overall 8.85% for MAPE.
Is it possible that it happened to be a little digital mistake here?
Thanks again for your nice work!
The text was updated successfully, but these errors were encountered: