Skip to content

Commit

Permalink
Fix numerical field generation.
Browse files Browse the repository at this point in the history
Numerical fields don't get tokenized. Bug hard to detect, because the examples are cached to disk.
  • Loading branch information
cristipp committed Aug 29, 2018
1 parent db64df3 commit 6b3ddd5
Showing 1 changed file with 6 additions and 3 deletions.
9 changes: 6 additions & 3 deletions text/torchtext/data/example.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,12 @@ def fromlist(cls, data, fields):
ex = cls()
for (name, field), val in zip(fields, data):
if field is not None:
if isinstance(val, six.string_types):
val = val.rstrip('\n')
setattr(ex, name, [sys.intern(x) for x in field.preprocess(val)])
if field.numerical:
setattr(ex, name, val)
else:
if isinstance(val, six.string_types):
val = val.rstrip('\n')
setattr(ex, name, [sys.intern(x) for x in field.preprocess(val)])
return ex

@classmethod
Expand Down

1 comment on commit 6b3ddd5

@cristipp
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#847a9dd fixed it. Thanks!

Please sign in to comment.