Pluggable datasets

[noe]

Currently it is possible to plug you own model for it to be evaluated. This can be done by providing a --t2t_usr_dir parameter to t2t-trainer, as explained in the readme, without needing to modify tensor2tensor itself.

However, the same is not possible with datasets. In order to provide your own dataset, you need to modify tensor2tensor itself, as described data generators README.

It would be nice to have a mechanism for custom datasets that is analogous to that for models.

If it is seen as something good but it's not to be implemented in the near future, are contributions in that direction welcome?

[rsepassi]

Thanks @noe. Yes, we're working on making datasets/problems pluggable.

[jgevershed]

I believe that the trainer, as well as the datagen, is not 100% "pluggable".
The data generator needs to modify _SUPPORTED_PROBLEM_GENERATORS.
The trainer may also need to add an entry to problem_hparams.PROBLEM_HPARAMS_MAP.

My "solution" is to have my own copies of t2t-trainer and t2t-datagen with minimal changes.
So, I start training by "python my_trainer.py --t2t_usr_dir......." (in Bash on Ubuntu on Windows).
Luckily the google t2t team has made both these modules very small.
To keep my needed trainer change down to one line, I call a simple helper in my usr dir.

It means I can have my own variant Transformer model and my own problem data, without any hassle in keeping tensor2tensor up to date on my platform.

[peblair]

@jgevershed Wouldn't something like this work to address this without needing a forked t2t-datagen/t2t-trainer (if support was added for it, of course)?

@register_problem("image_mnist")
def mnist_problem():
  return (lambda: <generator>, lambda: <generator>)

(of course, things like hyperparameters would also need to be supported, so perhaps a decorated class would make more sense)

[rsepassi]

Currently models, hyperparameter sets, and modalities can be registered without editing the T2T codebase by passing the --t2t_usr_dir command line flag. Data generators and problems cannot currently be registered in that way; however, we're working on unifying data generators and problems and adding them to the registry so that it becomes easy to plug in your own dataset & problem without editing the T2T codebase.

[rsepassi]

In v1.0.14, users can now register datasets/problems by defining a subclass of Problem and decorating it with @registry.register_problem. See algorithmic.py or wmt.py for examples.

[noe]

From what I see in the updated code of t2t-datagen, there is no equivalent to --t2t_usr_dir, which would be needed to load custom datasets from outside t2t codebase. As the issue is closed, I understand that this is not to be done, is that correct?

[rsepassi]

Ah, you're right. Oversight on my part. Will have it fixed shortly.

…

On Jul 17, 2017, 1:23 AM -0700, Noé Casas ***@***.***>, wrote: From what I see in the updated code of t2t-trainer, there is no equivalent to --t2t_usr_dir, which would be needed to load custom datasets from outside t2t codebase. As the issue is closed, I understand that this is not to be done, is that correct? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.