What is the configs/kt_config.json?

[skewondr]

Hello, Thanks for the nice framework for knowledge tracing.
I'm also impressed with the delicate experiments you've done on the paper(https://arxiv.org/abs/2206.11460)

Now I'm working on reproducing your Table 2 results shown on the paper, with the assist2015 dataset you've also shared.
And, I got confused about getting the best hyperparameters as you mentioned at A.3 Hyperparameter Search Details of Representative DLKT Baselines.

So, there are two different settings on your code, which is,

1. values in configs/kt_config.json
2. default values of every argument in wandb/wandb_{model_name}_train.py
   Which values do I have to follow to get the high result close to table 2?

Thanks.

[Liu-lqq]

Thank you for your attention to our work.

Neither the parameters in "configs/kt_config.json" nor "wandb/wandb_{model_name}_train.py" are the best parameters.
The best parameters of each fold for each dataset are obtained by hyperparameter search.

For example, for dataset assist2015, the best parameters are(dkt):
fold 0: {"dataset_name": "assist2015", "model_name": "dkt", "emb_type": "qid", "save_dir": "dkt_tiaocan_assist2015_rerun", "seed": 3407, "fold": 0, "dropout": 0.5, "emb_size": 200, "learning_rate": 0.001}
fold 1: {"dataset_name": "assist2015", "model_name": "dkt", "emb_type": "qid", "save_dir": "dkt_tiaocan_assist2015_rerun", "seed": 224, "fold": 1, "dropout": 0.5, "emb_size": 200, "learning_rate": 0.001}
fold 2: {"dataset_name": "assist2015", "model_name": "dkt", "emb_type": "qid", "save_dir": "dkt_tiaocan_assist2015_rerun", "seed": 42, "fold": 2, "dropout": 0.5, "emb_size": 200, "learning_rate": 0.001}
fold 3: {"dataset_name": "assist2015", "model_name": "dkt", "emb_type": "qid", "save_dir": "dkt_tiaocan_assist2015_rerun", "seed": 42, "fold": 3, "dropout": 0.5, "emb_size": 200, "learning_rate": 0.001}
fold 4: {"dataset_name": "assist2015", "model_name": "dkt", "emb_type": "qid", "save_dir": "dkt_tiaocan_assist2015_rerun", "seed": 42, "fold": 4, "dropout": 0.5, "emb_size": 200, "learning_rate": 0.001}

The best parameters above are obtained on Tesla V100.

[skewondr]

Thanks for the clarification.
For the issue, I searched all the cases with Table 7 for the dkt and concluded that dropout 0.5 for dkt is the best result, and that was right. :)
If you don't mind, can you share your best parameters for other models?

At least, it would be a great help if you share the values for the models : AKT, DKT, DKVMN, GKT, SAINT, SAKT

Thanks a lot.

[Liu-lqq]

Thank you for your advice.
We will open the optimal parameters of all models for all data sets in the near future.