Unproduce the results of text classification tutorial

[wangyongjie-ntu]

I download the Jupiter file of https://captum.ai/tutorials/IMDB_TorchText_Interpret and run step by step. I use the same training set, the model officially provided. But gives different results. Can you help explain why? What should I do to fix it? Thanks. By the way, the prediction is also different? Is there any random seed in building vocab?

[wangyongjie-ntu]

There seems a bug in this tutorial. As the train-test split is random, the training set used to build vocab is inconsistent. says the indices of input sentences are changing. You should keep the random seeds consistent between training and testing.

[NarineK]

Hi @wangyongjie-ntu ! I originally trained the model long time ago. Most probably it used the default random seed. If you use GPU then you might also need to fix the seed for GPU too.

[wangyongjie-ntu]

Hi @wangyongjie-ntu ! I originally trained the model long time ago. Most probably it used the default random seed. If you use GPU then you might also need to fix the seed for GPU too.

Thanks for your reply. I find PyTorch currently supports the save/load of TEXT field object. To make the tutorial consistent to yours, one option is the save your prebuilt vocab with 'torch.save', and load it in the tutorial with 'torch.load', like loading the pre-trained weights.

[NarineK]

Hi @wangyongjie-ntu, I looked into this again and I see that the dataset that I downloaded aclImdb already contains train / test split. IMDB dataset doesn't seem to be splitting randomly: https://github.com/pytorch/text/blob/4b32cf52768baa73a5d59e11451bb580a8dcf192/torchtext/data/dataset.py#L54

[wangyongjie-ntu]

Hi @NarineK, It seems you are right. But I find the indices of words can change, I have not understand why this happens. And I save/load the TEXT and solve it.

[NarineK]

It's interesting, the vocab size used to be: Vocabulary Size: 101982 and now I see Vocabulary Size: 101867.
Perhaps different versions of torchtext ?