How to train using my own dataset?

[ndvbd]

I would like to train en->fr using the Transformer model on one GPU, and by using wordpieces (sentencepiece) of 32k vocab, like in GNMT. Now I have my own dataset of English sentences and corresponding French sentences.

What's the right way to do it with T2T? Is there a way to pass the parameters of the test/dev/test source and target files in the command line, or must I register my own Problem, by adding a class to the data-generators folder?

Will the 32k vocab be generated automatically given the source and target corpuses I supply?

[martinpopel]

If you want to train on your own dataset you should define your own Problem, but you can do it with --t2t_usr_dir, so you don't need to modify the versioned files. (Alternatively, you could fake the downloaded archive files for an existing problem, but that's hacky.)

I suggest to use T2T internal subwords, which is the default in most translation problems. The vocabulary file will be created automatically if missing, you can specify its approximate size (e.g. 32k) in the Problem specification.
Note that sentencepiece is a different algorithm than T2T internal subwords. Sentencepiece allows subwords which include the space character.
If you want to use external tokenization/subwords (e.g. BPE or sentencepiece), you need to use TokenTextEncoder instead of SubwordTextEncoder, see e.g. and make sure you preprocess the training data and postprocess the translations accordingly (cf. the --replace_oov option).

[ndvbd]

Thanks. How can ignore the downloading of the .tgz file and make it simply use the .en and .fr files that are defined in the _ENDE_TRAIN_DATASETS variable? Must I hack the code or is there a flag for it? Something like generate-vocab-dont-download.

In the meanwhile, I had to simply place a dummy .tgz file so it will ignore the download, but I'm sure there should be a better way...

[martinpopel]

If the vocab file already exists, it is easy to (re)implement the whole generate function in three lines of code: create the vocab object with vocab = text_encoder.TokenTextEncoder(vocab_filename) and return data_generators.wmt.token_generator(input_path + "en", input_path + "xy", vocab, EOS).
However, there is no easy way for the most common use case when the training data is already downloaded (and maybe not available publicly online), but the vocab file needs to be generated. Downloading is hardwired in get_or_generate_vocab, but this will be skipped if the file exists (I use dummy.tgz). What cannot be skipped is the extraction of the downloaded tgz file and for this reason I use a tiny dummy.tgz which does not actually contain the training data; the training data is already extracted in the expected location, instead. I admit this is very hacky and there should be an easier way. Reimplementing (in your code in t2t_usr_dir) get_or_generate_vocab without the extraction, but supporting compilation from multiple sources would be a big code duplication. I guess PR to make this easier are welcome.

[yudianer]

this is my code:
Generator defined in the same directory as other problems.
All of the language pairs were preprocessed by BPE.

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import os
import tarfile

# Dependency imports

from tensor2tensor.data_generators import generator_utils
from tensor2tensor.data_generators import problem
from tensor2tensor.data_generators import text_encoder
from tensor2tensor.data_generators import translate
from tensor2tensor.utils import registry

import tensorflow as tf

FLAGS = tf.flags.FLAGS

# End-of-sentence marker.
EOS = text_encoder.EOS_ID


@registry.register_problem
class TranslateMnzhBpe32k(translate.TranslateProblem):

  @property
  def targeted_vocab_size(self):
    return 32000

  @property
  def source_vocab_name(self):
    return "vocab.32k.mn.txt"

  @property
  def target_vocab_name(self):
      return "vocab.32k.ch.txt"

  def feature_encoders(self, data_dir):
    source_vocab_filename = os.path.join(data_dir, self.source_vocab_name)
    target_vocab_filename = os.path.join(data_dir, self.target_vocab_name)
    source_encoder = text_encoder.TokenTextEncoder(source_vocab_filename, replace_oov="UNK")
    target_encoder = text_encoder.TokenTextEncoder(target_vocab_filename, replace_oov="UNK")
    return {"inputs": source_encoder, "targets": target_encoder}

  def generator(self, data_dir, tmp_dir, train):
    """Instance of token generator for the mn->zh task, training set."""
    dataset_path = ("train.32k"
                    if train else "valid.32k")
    train_path = os.path.join(data_dir, dataset_path)

    source_token_path = os.path.join(data_dir, self.source_vocab_name)
    target_token_path = os.path.join(data_dir, self.target_vocab_name)

    source_token_vocab = text_encoder.TokenTextEncoder(source_token_path, replace_oov="UNK")
    target_token_vocab = text_encoder.TokenTextEncoder(target_token_path, replace_oov="UNK")
    return translate.token_generator_by_source_target(train_path + ".mn", train_path + ".ch",
                                     source_token_vocab, target_token_vocab, EOS)

  @property
  def input_space_id(self):
    return problem.SpaceID.MN_BPE_TOK

  @property
  def target_space_id(self):
    return problem.SpaceID.ZH_BPE_TOK

token_generator_by_source_target:

def token_generator_by_source_target(source_path, target_path, source_token_vocab, targe_token_vocab, eos=None):
  """Generator for sequence-to-sequence tasks that uses tokens.

  Args:
    source_path: path to the file with source sentences.
    target_path: path to the file with target sentences.
    source_token_vocab: text_encoder.TextEncoder object.
    targe_token_vocab: text_encoder.TextEncoder object.
    eos: integer to append at the end of each sequence (default: None).
  Yields:
    A dictionary {"inputs": source-line, "targets": target-line} where
    the lines are integer lists converted from tokens in the file lines.
  """
  eos_list = [] if eos is None else [eos]
  with tf.gfile.GFile(source_path, mode="r") as source_file:
    with tf.gfile.GFile(target_path, mode="r") as target_file:
      source, target = source_file.readline(), target_file.readline()
      while source and target:
        source_ints = source_token_vocab.encode(source.strip()) + eos_list
        target_ints = targe_token_vocab.encode(target.strip()) + eos_list
        yield {"inputs": source_ints, "targets": target_ints}
        source, target = source_file.readline(), target_file.readline()

[stefan-it]

@NadavB Is this still an issue? Otherwise I guess we could close that issue :)

[yudianer]

ok, I have no question about this one already.

[ndvbd]

thanks @stefan-it

[sugeeth14]

If you want to train on your own dataset you should define your own Problem, but you can do it with --t2t_usr_dir, so you don't need to modify the versioned files. (Alternatively, you could fake the downloaded archive files for an existing problem, but that's hacky.)

I suggest to use T2T internal subwords, which is the default in most translation problems. The vocabulary file will be created automatically if missing, you can specify its approximate size (e.g. 32k) in the Problem specification.
Note that sentencepiece is a different algorithm than T2T internal subwords. Sentencepiece allows subwords which include the space character.
If you want to use external tokenization/subwords (e.g. BPE or sentencepiece), you need to use TokenTextEncoder instead of SubwordTextEncoder, see e.g. and make sure you preprocess the training data and postprocess the translations accordingly (cf. the --replace_oov option).

@martinpopel how to use Tokentextencoder in case I generate vocabulary using sentence piece. I am working on english-chinese translation and getting memory error when using t2t-datagen for larger chinese data. So I read through the issues and thought sentencepiece as a better alternative. In case I use sentencepiece what are the pre or post processings to be done ?

[Diego999]

I'm also wondering how to leverage SentencePiece within tensor2tensor

I think the TranslateMnzhBpe32k class should have generate_data method.

    def generate_data(self, data_dir, tmp_dir, task_id=-1):
        train_paths = self.training_filepaths(data_dir, 100, shuffled=True)
        dev_paths = self.dev_filepaths(data_dir, 1, shuffled=True)
        generator_utils.generate_files(
            self.generator(data_dir, tmp_dir, True), train_paths)
        generator_utils.generate_files(
            self.generator(data_dir, tmp_dir, False), dev_paths)

[prudhviraju535]

Hi Everyone,
I am new to T2T, can anyone provide a complete code on, how to generate translation data and register problem from own CSV file (2 columns of input and target). It will help me in my learning. Thanks in Advance.