Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix WT2, restructure splits, and add WMT14 #138

Merged
merged 11 commits into from
Oct 16, 2017
Merged

Fix WT2, restructure splits, and add WMT14 #138

merged 11 commits into from
Oct 16, 2017

Conversation

jekbradbury
Copy link
Contributor

Also adds nonoptional dependencies to setup.py so pip install torchtext will fetch them.

The only known possibly-breaking change (I'd call it a bugfix) is that Dataset.splits will now use os.path.join rather than concatenation to join provided paths.

@bmccann
Copy link
Contributor

bmccann commented Oct 9, 2017

Nice; the added download functionality is great, especially for Google Drive. Is there any way to unify the splits for the other datasets even more?

Copy link
Contributor

@bmccann bmccann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a little odd to have the WMT14 be the preprocessed version from Google, but I suppose we can worry about that later if we want to add other versions of WMT.

@jekbradbury
Copy link
Contributor Author

Agreed, though the others are a lot more complicated and this one allows direct comparison with Google's GNMT/Transformer results

@jekbradbury
Copy link
Contributor Author

jekbradbury commented Oct 9, 2017

Also the confirm_token thing sits there for easily five minutes before actually downloading?? I don't know what's up with that; the logic is exactly the same as what Google uses (and it looks like they got theirs from StackOverflow anyway)

@jekbradbury
Copy link
Contributor Author

Anyone know what to do about the bogus flake8 error? I think I've seen it before but I don't remember what if anything makes it go away (other than #NOQA)

@jekbradbury jekbradbury merged commit 247598d into master Oct 16, 2017
@jekbradbury jekbradbury deleted the wmt14 branch October 17, 2017 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants