Test result in multivariate dataset without pretrain

[Mingzhe-Han]

Hi,

I'm trying to study your code for multivariate classification dataset without pretrain, and I choose Handwriting for an example.

In order to achieve the paper's performance , I used the hyperparameters which is shown in your paper.

So I train the model with command below.

python src/main.py --output_dir experiments --comment "classification from Scratch" --name HW --records_file Classification_records.xls --data_dir data/Multivariate_ts/Handwriting --data_class tsra --pattern TRAIN --epochs 400 --lr 0.001 --optimizer RAdam --pos_encoding learnable --task classification --key_metric accuracy --val_ratio 0.2 --num_layers 3 --num_heads 16 --d_model 128 --dim_feedforward 256 --batch_size 128

And then I test the model with command below.

python src/main.py --output_dir experiments --comment "classification from Scratch" --name HW --records_file Classification_records.xls --data_dir data/Multivariate_ts/Handwriting --data_class tsra --pattern TRAIN --epochs 400 --lr 0.001 --optimizer RAdam --pos_encoding learnable --task classification --key_metric accuracy --val_ratio 0 --num_layers 3 --num_heads 16 --d_model 128 --dim_feedforward 256 --batch_size 128 --test_pattern TEST --test_only testset --load_model experiments/HW_2022-07-27_20-01-05_axV/checkpoints/model_best.pth

I thought I use the same data split, same model and same hyperparameters, but finally I find the acc is 0.25882352941176473 and it's different from 0.3 in paper. Is there any step I missed?

[Mingzhe-Han]

After reading #11, I search best epoch and train the model with such epoch and the whole train set, the result can be 0.316 now.

[Guanyunlph]

After reading #11, I search best epoch and train the model with such epoch and the whole train set, the result can be 0.316 now.

I encountered the same problem as you did, that is, I couldn't get results similar to the paper. How did you solve it? If you could describe it in detail, it would greatly help me. Thank you.