Questions about the experiments details

[speedcell4]

Hi, thanks for sharing the source code.

1. In Table 2, are these reported numbers the results of the test split or the validation split?
2. In Table 2, for the RoBbase (LoRA) on the RTE task, the reported result is 86.6, is this a typo? cause it is even much higher than the full-tuning results (delta = 7.9).

[edwardjhu]

Thanks for your questions.

1. I believe these are validation numbers since the test set is not public. That is also done by prior work.
2. Nope, that's not a typo. You can verify it with our checkpoint :)

[speedcell4]

Wow, thanks for your quick response. I got two more questions.

1. If I understand correctly, in table 2, the numbers of BitFit were taken from the original paper. But actually, there are some numbers I can not find in the original paper. For example, you mentioned the RoBbase (BitFit) on MRPC task results in 92.7, but I think the original paper reported this number as 92.0 in their Table 2. Could you specify more details about this?
2. Do you fine-tune the bias terms? cause, I understand you don't require gradients for the weight terms, but I did not see you turn off this for bias terms.

LoRA/loralib/layers.py

Line 116 in 33b9536

self.weight.requires_grad = False

[edwardjhu]

You are right. I can't remember where we got 92.7, and it should be 92.0.

Yes, the bias term is learnable here, even though it is not in the code used in our experiments. This seems to be a good idea in practice and has a minimal overhead. The checkpointing utility functions should take care of saving/loading biases. Please let me know if you encounter any issues :)

[speedcell4]

Thanks~