Multi30k and Dataset download refactor

[bmccann]

No description provided.

[jekbradbury]

Maybe factor out the targz handling into a GZipDataset (or a ZipDataset with ext=’tar.gz’)?

[bmccann]

It's a little weird since the class has multiple urls/tar.gz files.

[bmccann]

Okay, I've got a way to unify the tarfile/gzip datasets, so will push in a few minutes after tests run locally.

[bmccann]

Okay, I changed up the ZipDataset so that I could use it for all our datasets that download a zip file or tar.gz file and got rid of the cls.filename field, since it isn't used.

[bmccann]

So, it annoyed me that I couldn't get the Zipfile functionality to work the way I wanted for all the datasets. I ended up getting rid of Zipfile and adding a generic download function to the Dataset class. Datasets that want to use download() need to have a class variable urls defined.

The upside is that any dataset that needs to download files can do so using the common download() function. Standardizes how we handle downloading and decompressing.

[jekbradbury]

I hadn't read through this yet; this is such a good simplification! I'm still amazed how little code you can get away with.

[jihunchoi]

This is a trivial question, but isn't Multi30k dataset from WMT 2016 shared task, not 2017?
It seems that all links are pointing to the links of 2016.

[nelson-liu]

The shared task was in both years --- but yeah it would probably be more accurate to have the docstring be WMT 2016 (since the splits are from that year).