Can load Udop-Dual-Large-224, but not Udop-Unimodel-Large-224

[plamb-viso]

I was able to load the dual-large tokenizer/config/model using the same method described in this comment

However, when I attempt to use the exact same code, but for udop-unimodel-large-224 I get:

Traceback (most recent call last):
    tok = UdopTokenizer.from_pretrained(
    return cls._from_pretrained(
    tokenizer = cls(*init_inputs, **init_kwargs)
    self.sp_model.Load(vocab_file)
    return self.LoadFromFile(model_file)
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
RuntimeError: Internal: unk is not defined.

[zinengtang]

Can I know what transformer and sentencepiece version do you use.

[plamb-viso]

I am using the version from this PR which is based on the main branch and most recently 4.26.1

@zinengtang I realized that I can load the model/config from udop-unimodel-large-224 and the tokenizer from udop-dual-large-224 with no errors. Do you foresee this causing any problems or do the two tokenizers use the same settings?

[zinengtang]

different tokenizers shouldn't cause any issues, all configs are shared.

[zinengtang]

Our model dev uses 4.20.1. it might affect loading I think but I am not sure.