transform raw categorical features using custom mapping function

[rahulrajpl]

Hello,

As I understand from the documentation here, I can have a call like tft.compute_and_apply_vocabulary(s) in order to convert a categorical column to numerical feature.

As a beginner in tensorflow, I am wondering if there exists a custom mapping of raw feature column to a numerical column? I have already seen that hash_bucket method described here almost does the job I want. But instead of hash of the entries, I need a custom mapping function to be called so that 'm' unique elements in the categorical column is mapped to 'n' unique elements of numerical or string, where n < m.

Use case. I bumped into this issue during my experiment with the KDD CUP 99 dataset where target class of training set contains 23 different attack types where they are required to be identified and categorized into four class of attacks. If there is a transformation function I could use so that all 23 unique elements in target class could be mapped to 4 classes of attack numbered [1,2,3,4]. Including a normal connection that could be mapped to [0], the target class will contain 5 classes and thus I can directly train a multiclass classification model. More on KDD CUP 99 dataset is here

Can anyone help?

[zoyahav]

Note that tft.compute_and_apply_vocabulary (and the underlying analyzer tft.vocabulary), computes a vocabulary over the data.

Instead of using tft.compute_and_apply_vocabulary which just performs a lookup in the vocabulary and returns the index for each value, you could use tft.vocabulary and tft.apply_vocabulary with a custom lookup_fn:
https://www.tensorflow.org/tfx/transform/api_docs/python/tft/apply_vocabulary

in the preprocessing_fn:

def compute_mapped_value(x):
  ....
def lookup_fn(key, vocab_filename):
  table = tf.lookup.index_table_from_file(vocab_filename, ...)
  value = table.lookup(key)
  return compute_mapped_value(value)
vocabulary_filename = tft.vocabulary(x, ...)
x_numerical = tft.apply_vocabulary(x, vocabulary_filename, lookup_fn=lookup_fn, ...)

Would that work for your use case?

[rahulrajpl]

Yeah. I got that working. Thanks for the pointers.

[erikinfo]

Hello,

i'm facing the same problem with probably a newer version (TF 2.0) since tf.contrib.lookup.index_table_from_file is not available.

I've tried it with this:
table = tf.lookup.StaticHashTable(tf.lookup.TextFileInitializer(vocab_filename, tf.string, 0, tf.int64, 1, delimiter="!"),7)

but got
OperatorNotAllowedInGraphError: iterating over tf.Tensor is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.

What is the problem here and what lookups could i use instead?
And more minor: When using lookup_fn it says "return a lookup op along with the table size"? What would be an accurate return function?

Note that tft.compute_and_apply_vocabulary (and the underlying analyzer tft.vocabulary), computes a vocabulary over the data.

Instead of using tft.compute_and_apply_vocabulary which just performs a lookup in the vocabulary and returns the index for each value, you could use tft.vocabulary and tft.apply_vocabulary with a custom lookup_fn:
https://www.tensorflow.org/tfx/transform/api_docs/python/tft/apply_vocabulary

in the preprocessing_fn:

def compute_mapped_value(x):
  ....
def lookup_fn(key, vocab_filename):
  table = tf.lookup.index_table_from_file(vocab_filename, ...)
  value = table.lookup(key)
  return compute_mapped_value(value)
vocabulary_filename = tft.vocabulary(x, ...)
x_numerical = tft.apply_vocabulary(x, vocabulary_filename, lookup_fn=lookup_fn, ...)

Would that work for your use case?