Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question about optimal mask ratio #43

Open
Jimmy-7664 opened this issue May 16, 2023 · 5 comments
Open

Question about optimal mask ratio #43

Jimmy-7664 opened this issue May 16, 2023 · 5 comments

Comments

@Jimmy-7664
Copy link

Hi, Thank you for your awesome work. I am trying to reproduce the results of your Hyper-parameter Study experiment, but when I changed the mask ratio, I got different results from the paper on all three datasets.
In order to reproduce the problem, I clone STEP locally, use the dataset you provided, and then I only change the mask ratio to 0.25 and GPU num to 4 and then re-train the TSFormer and STEP models. Here are the results.
Result for PEMS04 dataset,

"2023-04-06 11:44:21,101 - easytorch-training - INFO - Epoch 89 / 100
2023-04-06 11:48:05,343 - easytorch-training - INFO - Result : [train_time: 224.24 (s), lr: 6.25e-05, train_MAE: 17.7495, train_RMSE: 28.6461, train_MAPE: 0.1334]
2023-04-06 11:48:05,345 - easytorch-training - INFO - Start validation.
2023-04-06 11:50:55,722 - easytorch-training - INFO - Result : [val_time: 170.38 (s), val_MAE: 18.1383, val_RMSE: 28.1314, val_MAPE: 0.1222]
2023-04-06 11:50:57,130 - easytorch-training - INFO - Checkpoint checkpoints/STEP_100/e87a2127b73d6ba8cda30085f7beb2b3/STEP_best_val_MAE.pt saved
2023-04-06 11:53:47,204 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 16.3986, Test RMSE: 26.5760, Test MAPE: 0.1097
2023-04-06 11:53:47,209 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 16.8978, Test RMSE: 27.5110, Test MAPE: 0.1138
2023-04-06 11:53:47,213 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 17.3182, Test RMSE: 28.1838, Test MAPE: 0.1168
2023-04-06 11:53:47,216 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 17.6034, Test RMSE: 28.6846, Test MAPE: 0.1193
2023-04-06 11:53:47,219 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 17.8579, Test RMSE: 29.0966, Test MAPE: 0.1208
2023-04-06 11:53:47,222 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 18.0855, Test RMSE: 29.4533, Test MAPE: 0.1219
2023-04-06 11:53:47,226 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 18.2933, Test RMSE: 29.7824, Test MAPE: 0.1234
2023-04-06 11:53:47,229 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 18.4903, Test RMSE: 30.0777, Test MAPE: 0.1249
2023-04-06 11:53:47,232 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 18.6646, Test RMSE: 30.3532, Test MAPE: 0.1263
2023-04-06 11:53:47,235 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 18.8327, Test RMSE: 30.6055, Test MAPE: 0.1280
2023-04-06 11:53:47,238 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 19.0170, Test RMSE: 30.8598, Test MAPE: 0.1294
2023-04-06 11:53:47,241 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 19.2633, Test RMSE: 31.1607, Test MAPE: 0.1311
2023-04-06 11:53:47,286 - easytorch-training - INFO - Result : [test_time: 170.16 (s), test_MAE: 18.0602, test_RMSE: 29.3930, test_MAPE: 0.1221]"

Result for METR-LA dataset,

2023-04-10 06:42:39,345 - easytorch-training - INFO - Epoch 96 / 100
2023-04-10 06:45:14,351 - easytorch-training - INFO - Result : [train_time: 155.01 (s), lr: 1.56e-04, train_MAE: 2.7075, train_RMSE: 5.3461, train_MAPE: 0.0707]
2023-04-10 06:45:14,354 - easytorch-training - INFO - Start validation.
2023-04-10 06:45:56,978 - easytorch-training - INFO - Result : [val_time: 42.62 (s), val_MAE: 2.6742, val_RMSE: 5.1144, val_MAPE: 0.0727]
2023-04-10 06:45:59,152 - easytorch-training - INFO - Checkpoint checkpoints/STEP_100/aa28a6aab40136a6af5885691c378f94/STEP_best_val_MAE.pt saved
2023-04-10 06:47:22,991 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 2.1564, Test RMSE: 3.7474, Test MAPE: 0.0513
2023-04-10 06:47:22,994 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 2.4244, Test RMSE: 4.4858, Test MAPE: 0.0598
2023-04-10 06:47:22,998 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 2.6010, Test RMSE: 4.9821, Test MAPE: 0.0661
2023-04-10 06:47:23,002 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 2.7384, Test RMSE: 5.3681, Test MAPE: 0.0713
2023-04-10 06:47:23,005 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 2.8518, Test RMSE: 5.6969, Test MAPE: 0.0758
2023-04-10 06:47:23,009 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 2.9472, Test RMSE: 5.9612, Test MAPE: 0.0793
2023-04-10 06:47:23,012 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 3.0293, Test RMSE: 6.1882, Test MAPE: 0.0824
2023-04-10 06:47:23,015 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 3.1039, Test RMSE: 6.3804, Test MAPE: 0.0852
2023-04-10 06:47:23,019 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 3.1690, Test RMSE: 6.5411, Test MAPE: 0.0877
2023-04-10 06:47:23,023 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 3.2300, Test RMSE: 6.6843, Test MAPE: 0.0900
2023-04-10 06:47:23,026 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 3.2900, Test RMSE: 6.8177, Test MAPE: 0.0923
2023-04-10 06:47:23,030 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 3.3564, Test RMSE: 6.9536, Test MAPE: 0.0946
2023-04-10 06:47:23,075 - easytorch-training - INFO - Result : [test_time: 83.92 (s), test_MAE: 2.9081, test_RMSE: 5.8956, test_MAPE: 0.0780]

Result for PEMS-BAY dataset,

2023-04-12 21:09:24,701 - easytorch-training - INFO - Epoch 99 / 100
2023-04-12 21:17:14,073 - easytorch-training - INFO - Result : [train_time: 469.37 (s), lr: 3.13e-05, train_MAE: 1.4102, train_RMSE: 3.1304, train_MAPE: 0.0307]
2023-04-12 21:17:14,076 - easytorch-training - INFO - Start validation.
2023-04-12 21:19:04,273 - easytorch-training - INFO - Result : [val_time: 110.20 (s), val_MAE: 1.4673, val_RMSE: 3.0659, val_MAPE: 0.0333]
2023-04-12 21:19:07,612 - easytorch-training - INFO - Checkpoint checkpoints/STEP_100/88e31eeaa70eda996a9b04849238f61f/STEP_best_val_MAE.pt saved
2023-04-12 21:22:46,004 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 0.8249, Test RMSE: 1.5080, Test MAPE: 0.0159
2023-04-12 21:22:46,013 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 1.0714, Test RMSE: 2.1557, Test MAPE: 0.0215
2023-04-12 21:22:46,023 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 1.2304, Test RMSE: 2.6389, Test MAPE: 0.0256
2023-04-12 21:22:46,031 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 1.3444, Test RMSE: 2.9996, Test MAPE: 0.0288
2023-04-12 21:22:46,038 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 1.4280, Test RMSE: 3.2660, Test MAPE: 0.0313
2023-04-12 21:22:46,046 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 1.4926, Test RMSE: 3.4677, Test MAPE: 0.0333
2023-04-12 21:22:46,054 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 1.5432, Test RMSE: 3.6218, Test MAPE: 0.0349
2023-04-12 21:22:46,062 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 1.5850, Test RMSE: 3.7426, Test MAPE: 0.0363
2023-04-12 21:22:46,070 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 1.6201, Test RMSE: 3.8362, Test MAPE: 0.0374
2023-04-12 21:22:46,079 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 1.6513, Test RMSE: 3.9144, Test MAPE: 0.0385
2023-04-12 21:22:46,088 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 1.6804, Test RMSE: 3.9816, Test MAPE: 0.0393
2023-04-12 21:22:46,096 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 1.7121, Test RMSE: 4.0462, Test MAPE: 0.0402
2023-04-12 21:22:46,249 - easytorch-training - INFO - Result : [test_time: 218.64 (s), test_MAE: 1.4320, test_RMSE: 3.3536, test_MAPE: 0.0319]
2023-04-12 21:22:47,739 - easytorch-training - INFO - Checkpoint checkpoints/STEP_100/88e31eeaa70eda996a9b04849238f61f/STEP_099.pt saved

Here is a comparison of the two results, STEP's results are from https://arxiv.org/pdf/2206.09113.pdf, and https://github.com/zezhishao/BasicTS.

I would like to ask you to help me figure it out.
Looking forward to your reply.
Best regards

@zezhishao
Copy link
Owner

That's strange, have you tested other masking ratios? E.g., the ratios in the paper.
Since the code has be significantly refactored, I now need to redo these experiments, which may take some time.

@Jimmy-7664
Copy link
Author

Thank you for your reply. I also run with mask ratio=0.75. The results and the performance in PEMS-BAY and METR-LA are not very different from the paper, but I can not reproduce the results of PEMS04 in the paper. My result of PEMS04 is similar with https://github.com/zezhishao/STEP/blob/github/training_logs/STEP_PEMS04.log.

@Jimmy-7664
Copy link
Author

@zezhishao Have you tested this problem yet? Looking forward to your reply.

@zezhishao
Copy link
Owner

zezhishao commented Jun 14, 2023

Hi @Jimmy-7664 , I am very sorry for the late reply. I have been very busy looking for a job recently.
At present, I haven't finished the test and come to an insightful conclusion.
I tested a few masking ratios, and the results under different masking ratios seem to have no specific rules.
In addition, I have not tested the masking ratios in the paper ,and STEP has been refactored, so I'm not sure if something went wrong.

@zezhishao
Copy link
Owner

I tested a few more masking ratios again.
From preliminary tests, masking ratio seems to only affect efficiency, and the impact on performance is not significant.
This is not consistent with the results of the ablation study in the paper. I will continue to test the previous version of STEP to see if I can reproduce the results at that time. If it is not consistent, I will immediately revise the results of the paper.
In addition, if you have any further findings or insights, please let me know. Maybe we can figure out why and write a paper together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants