The default changed from 1e-3 to 1e-4 in this commit.
For training on ChEBI directly, 1e-3 seems to fare better than 1e-4. See run with lr=1e-3 and run with lr=1e-4.
Todo
Change default to 1e-3 and add a comment that 1e-4 is better for OPT finetuning