Data setup: WMT preprocessing in data_setup.py

[priyakasimbeg]

There are currently has to be downloaded manually and there does not exist any documentation with instructions.

Description

The WMT workload requires the wmt17_translate/de-en and wmt14_translate/de-en datasets which have to be downloaded manually with tfds.load. During the download the user may run into issues, for example when downloading wmt14_translate/de-en I ran into ResourceExhausted errors during the generating step. The solution was to manually increase the open file limit on the system as described here.

Steps to Reproduce

VM: tf-agents/in-to-win
In python terminal run `tfds.load('wmt14_translate/de-en')

Source or Possible Fix

We should either:

• Document instructions to download the datasets for WMT workload clearly, including workarounds for anticipated issues.
• Automate the downloading of the data, perhaps as a prerequisite step.

[priyakasimbeg]

From email thread with @runame it seems like the recommended instructions for downloading this dataset are in the flax examples repo.

[runame]

@priyakasimbeg I think this issue was addressed in #235?

[priyakasimbeg]

From Mike's setup looks like download script works fine but there are some outstanding issues w preprocessing:

First time running submission_runner the tfds data loader needs to do some additional work (looks like preprocessing WMT’17 and training a tokenizer from the logs). This results in a NotFoundError on other ranks when they try to load wmt_sentence_piece_model.rntmp
However, after that, subsequent times running the training code work fine.
Recommendation: Consider modifying dataset_setup.py so that it takes care of all preprocessing and training the sentence piece model

[runame]

Fixed by #444.