data loading

[anxuthu]

Hello! This is a very nice work!

I am looking into your codes and did not figure out why only the first training file ([0]) was loaded in line 146 of translation.py

examples_train = torch.load(data_paths[0])

Thanks!

[tunz]

IIRC, it splits data, and loads one at a time because of memory limitation. The code is just loading the first chunk, and the iterator will pick the next one after completing each one.

It's my old memory, so I could be wrong :)