Loading text tutorial: fixed OOV handling

[vaharoni]

In the Beginners tutorials > Load and preprocess data > Text, under Example 2, I believe there is an issue handling OOV tokens. The code does this:

keys = vocab
values = range(2, len(vocab) + 2)  # Reserve `0` for padding, `1` for OOV tokens.

init = tf.lookup.KeyValueTensorInitializer(
    keys, values, key_dtype=tf.string, value_dtype=tf.int64)

num_oov_buckets = 1
vocab_table = tf.lookup.StaticVocabularyTable(init, num_oov_buckets)

However, note that StaticVocabularyTable does not return the value 1 for OOV, but rather hash(<term>) % num_oov_buckets + vocab_size per its documentation. In our case, this is equal to vocab_size, which is actually 10000 (and not 1 nor 10002). This in fact collides with the token in the original index 9998. To fix this, I suggest to first map the OOV to 10002 by providing vocabulary that includes the padding and OOV tokens inherently:

# Reserve `0` for padding, `1` for OOV tokens.
keys = ['', '[UNK]'] + vocab
values = range(len(keys))

Then, in the subsequent cell remap the 10002 to 1. I'm using len(keys) here rather than vocab_size + 2 as later in the tutorial we do vocab_size += 2.

def preprocess_text(text, label):
  standardized = tf_text.case_fold_utf8(text)
  tokenized = tokenizer.tokenize(standardized)
  vectorized = vocab_table.lookup(tokenized)
  # StaticVocabularyTable returns the OOV token as vocab_size + 2. We overwrite it to be 1.
  vectorized = tf.where(vectorized == len(keys), tf.constant(1, dtype=tf.int64), vectorized)
  return vectorized, label

[github-actions]

Preview

Preview and run these notebook edits with Google Colab:

• site/en/tutorials/load_data/text.ipynb

Rendered notebook diffs available on ReviewNB.com.

Format and style

Use the TensorFlow docs notebook tools to format for consistent source diffs and lint for style:

$ python3 -m pip install -U --user git+https://github.com/tensorflow/docs

$ python3 -m tensorflow_docs.tools.nbfmt notebook.ipynb

$ python3 -m tensorflow_docs.tools.nblint --arg=repo:tensorflow/docs notebook.ipynb

If commits are added to the pull request, synchronize your local branch: git pull origin tutorial_text_fix_2

[MarkDaoust]

Nice catch. Thanks for all the PRs!

I did a quick edit to use the builtin OOV indexes instead of remapping, and added a cell to test run it.

[vaharoni]

This approach works for that section. However, note that in the subsequent section "Export the model", we do the following:

To make the model capable of taking raw strings as input, you will create a Keras TextVectorization layer that performs the same steps as your custom preprocessing function. Since you have already trained a vocabulary, you can use TextVectorization.set_vocabulary (instead of TextVectorization.adapt), which trains a new vocabulary.

preprocess_layer = TextVectorization(
    max_tokens=vocab_size,
    standardize=tf_text.case_fold_utf8,
    split=tokenizer.tokenize,
    output_mode='int',
    output_sequence_length=MAX_SEQUENCE_LENGTH)

preprocess_layer.set_vocabulary(vocab)

export_model = tf.keras.Sequential(
    [preprocess_layer, model,
     layers.Activation('sigmoid')])

We do not train export_model. We just export it, relying on the weights of model. However, there is a subtle difference here. model was trained to expect 10001 for the OOV token from the StaticVocabularyTable returned from your make_vocab_table function. But exported_model receives 1 for the OOV token from the TextVectorization layer. This could impact inference, and the statement "you will create a Keras TextVectorization layer that performs the same steps as your custom preprocessing function" seems incorrect.

[MarkDaoust]

Thanks again. What a strange setup? I'll fix it.

[MarkDaoust]

Okay, I've removed my commits, let's merge this, and I'll fix the weirdness in a separate PR.