Consistent preprocessing output on all backends #1777
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mattdangerw
force-pushed
the
consistent-preprocessing-outputs
branch
7 times, most recently
from
2008b84 to
5de16cd
Compare
mattdangerw
changed the title
Old behavior: - On TF backend, raggeds and strings returned as tf tensors. - On Jax/Torch bakcnes, raggeds and strings returned as lists. - Preprocessing functions outside of `call`, like `tokenize()`, `detokenize()`, `generate_preprocess()`, will always return tf tensors. This made it hard to write backend agnostic code. TF shows up in random places, and if you are flipping from tf -> jax or vice versa you have to switch between handling tensors and lists. New behavior: - On all backends for all functions, raggeds and strings are returned as lists. - Inside a `tf.data` call or tf compiled function, preprocessing layers always output tf.tensors. This requires a little complexity to avoid over converting back and forth for tf -> python, but thankfully we can hide most of that complexity in a decorator.
mattdangerw
force-pushed
the
consistent-preprocessing-outputs
branch
from
5de16cd to
5fdaa4a
Compare
mattdangerw
force-pushed
the
consistent-preprocessing-outputs
branch
from
c582212 to
331f6a1
Compare
SamanehSaadat
approved these changes
Aug 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Old behavior:
__call__, liketokenize(),detokenize(),generate_preprocess(), will always return tf tensors on all backends.This made it hard to write backend agnostic code. TF shows up in random places, and if you are flipping from tf -> jax or vice versa you have to switch between handling tensors and lists.
New behavior:
tf.datacall or tf compiled function, preprocessing layers always output tf.tensors.This requires a little complexity to avoid over converting back and forth from tf -> python in nested calls, but thankfully we can hide most of that complexity in a decorator.