Dataset fixes and tests #105
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix writing json data
jekbradbury
approved these changes
Sep 8, 2017
jekbradbury
added a commit
that referenced
this pull request
Sep 8, 2017
This reverts commit 3aa4d13.
jekbradbury
added a commit
that referenced
this pull request
Sep 8, 2017
bmccann
pushed a commit
to bmccann/text
that referenced
this pull request
Sep 11, 2017
This reverts commit 3aa4d13.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds some tests to the
DatasetandExampleclasses. It also sets up aTorchtextTestCasebase test case class that is used as a central repository for writing synthetic test data to a temporary directory when running tests.I've also made a few minor changes to the class behavior:
Previously the dataset code would silently skip input fields that didn't exist in the JSON data. I'm curious if there's a real dataset that this makes easier to work with --- it certainly seems like it could be a pretty big source of bugs for the user, so I've changed it to error when the dataset tries to read a JSON key not actually in the data.
Reading CSV data with Unicode characters previously didn't work with Python 2, since the
csvmodule in python 2 doesn't support unicode input. I've fixed it (by encoding toutf8, running it through the CSV reader, then decoding back tounicode) and added a non-regression test for it.Making examples from CSV and TSV lines would previously remove the last character if it was
\n. I've changed this to useline.rstrip("\n").