Skip to content

Files

Latest commit

 

History

History
178 lines (141 loc) · 5.19 KB

hashtable.md

File metadata and controls

178 lines (141 loc) · 5.19 KB

How to use TF Lookup ops in TFLite

The objective of this file is to provide examples to demonstrate how to use TF Lookup ops in TFLite.

Supported Tensorflow Lookup ops in TFLite

Here is the supported status of TensorFlow Lookup ops.

TF Python lookup ops Supported status
tf.lookup.StaticHashTable Supported only with tensor initializers.

Supported mapping type: string → int64, int64 → string

tf.lookup.Hashtable Supported only with tensor initializers.

Supported mapping type: string → int64, int64 → string

tf.lookup.index_to_string_table_from_tensor Supported.
tf.lookup.index_table_from_tensor Supported natively when num_oov_buckets=0 and dtype=dtypes.string.

For the oov concept, you will need a Flex delegate.

tf.lookup.StaticVocabularyTable Supported but you will need a Flex delegate.

Use tf.index_table_from_tensor or tf.index_to_string_table_from_tensor instead if possible if you don’t want to use Flex delegate.

tf.lookup.experimental.DenseHashTable

tf.contrib.lookup.MutableHashTable

tf.contrib.lookup.MutableDenseHashTable

Not supported yet.
tf.lookup.IdTableWithHashBuckets Supported but you need a Flex delegate.

Python Sample code

Here, you can find the Python sample code:

  • Static hash table (string → int64)
int64_values = tf.constant([1, 2, 3], dtype=tf.int64)
string_values = tf.constant(['bar', 'foo', 'baz'], dtype=tf.string)

initializer = tf.lookup.KeyValueTensorInitializer(string_values, int64_values)
table = tf.lookup.StaticHashTable(initializer, 4)

with tf.control_dependencies([tf.initializers.tables_initializer()]):
  input_string_tensor = tf.compat.v1.placeholder(tf.string, shape=[1])
  out_int64_tensor = table.lookup(input_string_tensor)
  • Static hash table, initialized from a file (string → int64)
with open('/tmp/vocab.file', 'r') as f:
  words = f.read().splitlines()

string_values = tf.constant(words, dtype=tf.string)

initializer = tf.lookup.KeyValueTensorInitializer(string_values, int64_values)
table = tf.lookup.StaticHashTable(initializer, 4)

with tf.control_dependencies([tf.initializers.tables_initializer()]):
  input_string_tensor = tf.placeholder(tf.string, shape=[1])
  out_int64_tensor = table.lookup(input_string_tensor)
  • Index table (string → int64)
UNK_ID = -1
vocab = tf.constant(["emerson", "lake", "palmer"])
vocab_table = tf.lookup.index_table_from_tensor(vocab, default_value=UNK_ID)

input_tensor = tf.compat.v1.placeholder(tf.string, shape=[5])

with tf.control_dependencies([tf.initializers.tables_initializer()]):
  out_tensor = vocab_table.lookup(input_tensor)
  • Index table, initialized from a file (string → int64)
with open('/tmp/vocab.file', 'r') as f:
  words = f.read().splitlines()

UNK_ID = -1
vocab = tf.constant(words)
vocab_table = tf.lookup.index_table_from_tensor(vocab, default_value=UNK_ID)

input_tensor = tf.compat.v1.placeholder(tf.string, shape=[5])

with tf.control_dependencies([tf.initializers.tables_initializer()]):
  out_tensor = vocab_table.lookup(input_tensor)
  • Index to string table (int64 → string)
UNK_WORD = "unknown"
vocab = tf.constant(["emerson", "lake", "palmer"])
vocab_table = tf.lookup.index_to_string_table_from_tensor(vocab, default_value=UNK_WORD)

input_tensor = tf.compat.v1.placeholder(tf.int64, shape=[1])

with tf.control_dependencies([tf.initializers.tables_initializer()]):
  out_tensor = vocab_table.lookup(input_tensor)
  • Index to string table, initialized from a file (int64 → string)
with open('/tmp/vocab.file', 'r') as f:
  words = f.read().splitlines()

UNK_WORD = "unknown"
vocab = tf.constant(words)
vocab_table = tf.lookup.index_to_string_table_from_tensor(vocab, default_value=UNK_WORD)

input_tensor = tf.compat.v1.placeholder(tf.int64, shape=[1])

with tf.control_dependencies([tf.initializers.tables_initializer()]):
  out_tensor = vocab_table.lookup(input_tensor)

How to Include Hashtable ops in your TFLite.

Currently, hashtable ops are now a part of the TFLite builtin op set. You don't need to add hashtable ops manually.