New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

why word_id + 1 in data2pkl.py? #3

Open

eveliao opened this issue Aug 6, 2019 · 1 comment

eveliao commented Aug 6, 2019 •

edited

Loading

In preprocess/data2pkl.py line 43, 44:

sent = f.readline().strip().split(",")
sentences.append([(x+1) for x in list(map(int, sent))])

sent should be the word id list of current sentence. But why x+1 (instead of x)?

Is it because a new token '' is added to vocaburary?

Thanks.

The text was updated successfully, but these errors were encountered:

eveliao changed the title ~~why word id + 1 in data2pkl.py?~~ why word_id + 1 in data2pkl.py?

Owner

ZhixiuYe commented Aug 9, 2019

Index 0 is used for a special token [UNK], which is represented for the unseen words in Glove.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment