-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reproducing the results for ETTh2 #44
Comments
Hi, ETTh2 has different characteristics compared to ETTh1, so you have to change hyper parameters.
We performed many different combinations of By the way, our current code has updated the data scaling way compared with the paper on arxiv, so we will update the experiment results later. Thanks for your attention to our work. |
Suppose there is no more discussion, and I will close this issue now. |
Hi Again, Thanks a lot for the quick response. Would it be possible for you to provide the hyper parameters for all the datasets used in the univariate and multivariate setting (for each sequence length) so that I could do a comparison. Sorry for the delay in response. Additionally, I am getting much better results with the ETTm1 dataset while using the hyper parameters from ETTh1 dataset.
Could you please confirm if these values makes sense? or am I missing something? Thanks again. |
Hi, It seems that we named ETTm1 and ETTm2 files incorrectly in ETDataset , and we will fix this error immediately. And we will update two new experimental results tables (due to changes of data scaling) at March 20th, and we will also provide the hyperparameter settings for each experiment at that time. Please look forward to it. Thanks! |
@cookieminions Hi, Thank you for your answer. For Etth2, the pred and true are still not matching well after I changed all the parameter according to yours, is there anything else I can change to improve the results. I also couldn't find the true figure in the Etth2 data set, has the result data been scaled? Is the first prediction point start at (8569(train)+2857(val)+500(i)+48(input_len)+24(label_len))th points? Thank you very much. |
Hi @cookieminions I am unsure about the testing procedure, if I understand testing part correctly, each 168(input_len)+48(label len)+48(pred_len)=264 points is one section and then repeat for the next section (264 points) for prediction. So the 48 prediction points are using model trained from previous training and validation data and then depend on 168(input len) and 48(label len) to make prediction (24 points). Could you please also check the code whether there are any code need to be changed? |
Hi, I make a graph to help explain how to get test samples from test series, hope it will be helpful: For example, the first sample is As for the figure, And hope Informer can be helpful to your work in the use of other specific tasks. |
Thank you for your kindness. I really appreciate your help! |
You are welcome. Thanks for your attention to our work. |
Hi @cookieminions, When I try to find where the true points are in the dataset. I just notice the first prediction actually start at 8569(train)+2857(val)+500(i)+48+48 th point. Where the input len is 48, label len is 24, pred len is 24. |
I think maybe your confusion can be explained by code, in code # 48-51
border1s = [0, 12*30*24 - self.seq_len, 12*30*24+4*30*24 - self.seq_len]
border2s = [12*30*24, 12*30*24+4*30*24, 12*30*24+8*30*24]
border1 = border1s[self.set_type]
border2 = border2s[self.set_type]
#78-79
self.data_x = data[border1:border2]
self.data_y = data[border1:border2]
# 95-96
def __len__(self):
return len(self.data_x) - self.seq_len- self.pred_len + 1
# test flag:
# data_x: [12*30*24+4*30*24-self.seq_len: 12*30*24+8*30*24]
# data_x: [train_len+val_len-seq_len: train_len+val_len+test_len]
# test_len = 4*30*24 The length of When you load the first sample of test_dataset, you will get an encoder input with length |
I also still can't reproduce the level of accuracy as reported in the paper in univariate mode. |
Don’t worry, we will update the experimental results and provide experimental parameters in a few days. |
Hi @cookieminions,
In the script, I can see Therefore, |
Yes, your understand is correct. But the size of Train, Validation and Test data you see when running data is the size of dataloader. However, in code So the length of data_x is not the same with the length of dataloader when running the code. if flag == 'test':
shuffle_flag = False; drop_last = True; batch_size = args.batch_size
else:
shuffle_flag = True; drop_last = True; batch_size = args.batch_size
data_loader = DataLoader(
data_set,
batch_size=batch_size,
shuffle=shuffle_flag,
num_workers=args.num_workers,
drop_last=drop_last) |
Great! Thank you very much. |
Hi @cookieminions
|
|
Thank you, also is it possible to output confidence interval or prediction interval? |
The target of Informer is 'point prediction', so our code does not provide the function of prediction confidence interval now, but it is possible. Maybe you can refer to |
Hi @cookieminions, |
Hi, If using new data from test set as seq_len, different seq_len will cause different number of prediction samples because the length of test set series is fixed. But using seq_len in vali set, the data in vali set is just used as encoder's input and not used as prediction series, so it will not influence the test set samples. |
Hello,
Thanks a lot for the publishing your results and code, I enjoyed reading the paper.
While trying to reproduce the paper results, the output was way off especially for the ETTh2 dataset. (Ran it with the same configuration in the Colab notebook)
Could you please let me know if you use a different hyper parameter? or what am i doing wrong .
Thanks in advance.
Regards,
Kiran
The text was updated successfully, but these errors were encountered: