Question about dataset construction

[LeePleased]

Hello Yikang:

Hi~, I'm a research intern of HIT-SCIR lab, Yangming Li. It's great for your contribution about this repository. But I found some problems about the dataset construction (including test set):

1, the use of pytorch API "narrow" will unexpectedly abandon some words and result in incorrect PPL score.

2, It seems that your slide window on the whole corpus is not continuous and thus generate far less data than usual.

Great thanks again for your contribution about this repository.
Yangming, 19/08/02

[yikangshen]

Hi Yangming,
The problems that you mentioned are actually regularization methods introduced in AWD-LSTM (https://github.com/salesforce/awd-lstm-lm). Please refer to their paper for explanations.

[LeePleased]

Very thanks for your response :)