RobertaDot_NLL_LN class not defined?

[ylwangy]

Hi, jingtao

I find that the adore model released is not defined in your code.

The config.json file indicates that the model architecture is "RobertaDot_NLL_LN", however , it seems not defined in model.py.

[jingtaozhan]

Oh, sorry for the confusion. This is quite tricky.
In fact, RobertaDot_NLL_LN is defined in a previous work. To enable a fair comparison, we utilize the same model architecture and the same warmup model. Our released RobertaDot is the same as RobertaDot_NLL_LN except for the name. As for the RobertaDot_NLL_LN in the config.json, it is from the warmup model released by that previous work and I did not change it.

[ylwangy]

Thanks, and when we train adore, which model should we give for the init_path arg?

config = RobertaConfig.from_pretrained(args.init_path)
model = RobertaDot.from_pretrained(args.init_path, config=config)

Should the init_path be the star model we trained previously (say the checkpoint you released? The model architecture is RobertaDot_InBatch, not RobertaDot ), or other PLMs such as roberta_base?

[jingtaozhan]

ADORE further finetunes the query encoder and uses a frozen document encoder.
In our experiments, we use the frozen document encoder as initialization.

[jingtaozhan]

The so-called RobertaDot_Inbatch only means that we use a trick similar to in-batch training. The actual model architecture is still the same as RobertaDot.
In all our experiments, we use the same model architecture defined as RobertaDot

[ylwangy]

I see. Thanks and I'll try.