Skip to content

Commit

Permalink
Fix #139 XLNet example with IMDB fails (#140)
Browse files Browse the repository at this point in the history
- This was due to a bug in the IMDB processor which resulted in an empty
  dataset being created, which caused the PyTorch built-in sampler to
  raise an exception
  • Loading branch information
huzecong committed Aug 5, 2019
1 parent 7f3861e commit f76a2f7
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions examples/xlnet/utils/processor.py
Original file line number Diff line number Diff line change
Expand Up @@ -286,9 +286,9 @@ def _create_examples(data_dir: Path) -> List[InputExample]:
for label in ["neg", "pos"]:
cur_dir = data_dir / label
for filename in cur_dir.iterdir():
if filename.suffix != "txt":
if filename.suffix != ".txt":
continue
with (cur_dir / filename).open() as f:
with filename.open() as f:
text = f.read().strip().replace("<br />", " ")
examples.append(InputExample(
guid=str(filename), text_a=text, text_b=None, label=label))
Expand Down

0 comments on commit f76a2f7

Please sign in to comment.