Few-shot NLU: learning rate for model parameters vs. embedding parameters

[nelson-liu]

Hi!

Thanks for the interesting paper and releasing this nice codebase! I had a quick question with respect to the learning rate used for the fewshot NLU experiments. The paper mentions (Section 4.2) that:

We perform grid search of hyper-parameters and take the best combination on Ddev or Ddev32. Specifically, we take learning rates from 1e-5, 2e-5, 3e-5 and batch sizes from 16, 32

However, it seems like the model is updated with a fixed learning rate of 1e-5 in the code ( https://github.com/THUDM/P-tuning/blob/main/PT-Fewshot/pet/wrapper.py#L312 ) , and the learning rate taken from the CLI is only used for the embedding parameters.

Given that the paper and code seem to differ in this regard, I'm not sure if this is a bug in the code (i.e., the model and the embedding parameters should always use the LR taken from the CLI) or if the paper omits this detail (i.e., in reality, the LR grid search is only done on the embedding parameters, and 1e-5 is always used for the model). Could yo clarify which approach was taken in your experiments?

Thanks again!

[nelson-liu]

ah, rereading that passage, am I correct in that the grid-search is not used in the few-shot setting (and the default hyperparameters from PET are used)?

[zheng-yanan]

ah, rereading that passage, am I correct in that the grid-search is not used in the few-shot setting (and the default hyperparameters from PET are used)?

Hi!

Yes, in the few-shot setting, the hyperparameters from PET are used and we additionally select hyperparameters for prompt-related ones. Actually, we've experimented to use the same/different learning rates for both the backbone and prompt embeddings, and find that using different learning rates yields better performance in the few-shot setting. The grid-search mentioned in the paper was used in the fully supervised setting.

Thank you.