Deberta tokenizer.detokenize() errors out with mask token

[mattdangerw]

When working on the Deberta masked language model, we had to do some special treatment for the mask token in the tokenizer.

We left one outstanding bug on the main PR, which is that detokenize will error out with a mask token. See:
#732 (comment)

Here's a colab:
https://colab.research.google.com/gist/mattdangerw/5164a7cad80e9f5fcbb9a495264f80e1/deberta-detokenize-error.ipynb

We should either strip or properly render the mask token during detokenize so the call does not error out.

[mattdangerw]

@abheesht17 copying you in case you have any thoughts on a proper fix.

[ADITYADAS1999]

/assign

Hey @mattdangerw can I work this issue ?

[mattdangerw]

sg!

[ADITYADAS1999]

So, here it is two ways to tackle this error, strip the mask token or properly rendering during detokenize while calling 🤔

[TheAthleticCoder]

So, an approach I have in mind is that:
In the id_to_token and token_to_id,
we can just use an if-else loop to check if the id is equal to the self.mask_token_id and token is equal to [MASK] and return accordingly. Does that work?

[Mrutyunjay01]

I believe it's a typo as there is no change to the BBPE tokenizer used in DeBerta (earlier versions). Correct me if I'm wrong, thanks

https://github.com/keras-team/keras-nlp/blob/4c94a0d68872a72ed3e27d3f93a3d7eec3ddc0ba/keras_nlp/models/deberta_v3/deberta_v3_masked_lm_preprocessor.py#L141

[abheesht17]

So, an approach I have in mind is that: In the id_to_token and token_to_id, we can just use an if-else loop to check if the id is equal to the self.mask_token_id and token is equal to [MASK] and return accordingly. Does that work?

The issue is with the detokenize function. We have already taken care of the functions you have mentioned: https://github.com/keras-team/keras-nlp/blob/master/keras_nlp/models/deberta_v3/deberta_v3_tokenizer.py#L113-L121.

[Plutone11011]

So, here it is two ways to tackle this error, strip the mask token or properly rendering during detokenize while calling 🤔

The latter approach seems more intuitive to me. Stripping the mask token means that we ignore [MASK] during detokenization?

[ADITYADAS1999]

So, here it is two ways to tackle this error, strip the mask token or properly rendering during detokenize while calling 🤔

The latter approach seems more intuitive to me. Stripping the mask token means that we ignore [MASK] during detokenization?

not sure but stripping removes or truncates extra spaces ,
So, what about rendering the mask token during detokenization ?

[Plutone11011]

not sure but stripping removes or truncates extra spaces , So, what about rendering the mask token during detokenization ?

I misunderstood what you meant, sorry. In my mind, I think @TheAthleticCoder had the right idea, you could override detokenize and use id_to_token for each input, which already takes care of [MASK].

There might be a more efficient method though.

[abheesht17]

I would suggest that we just strip the mask token for now. @ADITYADAS1999, could you please open a PR for that?

Trying to keep the mask token will most probably require modifying the sentencepiece proto. If anyone is willing to take that up, cool! Here is the repo: https://github.com/google/sentencepiece. Here is a notebook to get started: https://colab.research.google.com/github/google/sentencepiece/blob/master/python/sentencepiece_python_module_example.ipynb. This isn't very easy to do since the SP Python API does not have any way to add tokens after training :/