Skip to content

Stripping the MASK token #876

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Mar 23, 2023
Merged

Stripping the MASK token #876

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
8b8bc19
Stripping the MASK token
TheAthleticCoder Mar 18, 2023
c991718
Stripping the MASK token
TheAthleticCoder Mar 18, 2023
c95e0a9
added unit test
TheAthleticCoder Mar 21, 2023
40231bd
fixed detokenize function
TheAthleticCoder Mar 21, 2023
6152965
check if test unit is correct
TheAthleticCoder Mar 21, 2023
408f1b5
changed MASK token index
TheAthleticCoder Mar 21, 2023
7bc5d18
trial using tokenizer mask id
TheAthleticCoder Mar 22, 2023
ae43476
using tf ragged boolean mask
TheAthleticCoder Mar 22, 2023
1bbb9ed
reformatted the prev commit
TheAthleticCoder Mar 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions keras_nlp/models/deberta_v3/deberta_v3_tokenizer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@

import copy

import tensorflow as tf

from keras_nlp.api_export import keras_nlp_export
from keras_nlp.models.deberta_v3.deberta_v3_presets import backbone_presets
from keras_nlp.tokenizers.sentence_piece_tokenizer import SentencePieceTokenizer
Expand Down Expand Up @@ -120,6 +122,10 @@ def token_to_id(self, token):
return self.mask_token_id
return super().token_to_id(token)

def detokenize(self, ids):
ids = tf.ragged.boolean_mask(ids, tf.not_equal(ids, self.mask_token_id))
return super().detokenize(ids)

@classproperty
def presets(cls):
return copy.deepcopy(backbone_presets)
5 changes: 5 additions & 0 deletions keras_nlp/models/deberta_v3/deberta_v3_tokenizer_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@ def test_detokenize(self):
output = self.tokenizer.detokenize(input_data)
self.assertEqual(output, tf.constant(["the quick brown fox"]))

def test_detokenize_mask_token(self):
input_data = tf.constant([[4, 9, 5, 7, self.tokenizer.mask_token_id]])
output = self.tokenizer.detokenize(input_data)
self.assertEqual(output, tf.constant(["the quick brown fox"]))

def test_vocabulary_size(self):
self.assertEqual(self.tokenizer.vocabulary_size(), 11)

Expand Down