Adding evaluation of TURNA-encoder #74
Conversation
| try: | ||
| self.encoder = T5EncoderModel.from_pretrained(pretrained_model_name) | ||
| except Exception as e: | ||
| pretrained_model_name = config._name_or_path | ||
| self.encoder = T5EncoderModel.from_pretrained(pretrained_model_name) | ||
|
|
There was a problem hiding this comment.
Do we really need exception handling here? When does the encoder initialization fail with pretrained_model_name?
There was a problem hiding this comment.
This is really bad code I realize :D but there was a problem with the from_pretrained method, when it calls the init of T5forSequenceClassification, that argument is overwritten as the config.. I couldn't solve it, so I added this.. If I can find a solution, I will fix it! 👍
There was a problem hiding this comment.
Can we predict if it's going to fail beforehand and update the pretrained_model_name accordingly? In what scenarios do we use the first from_pretrained method compared to the other one?
| try: | ||
| return(float(label.strip())) | ||
| except: | ||
| return 0 | ||
| try: | ||
| return(float(label)) | ||
| except: | ||
| return 0 | ||
|
|
There was a problem hiding this comment.
Doesn't float(label.strip()) cover float(label) already?
Did we check the outputs to see if they contain similarity scores where they cannot immediately cast to float?
There was a problem hiding this comment.
When we use conditional generation, the output is a string, so float(label.strip()) works. When we use classification, the output is a number, so the strip() function naturally gives an error, so I try converting into a float.
There was a problem hiding this comment.
I see. Rather than handling it with exceptions, I suggest checking its type to choose the conversion. It would be much clearer.
In this PR, the evaluation of fine-tuned TURNA-encoder models was implemented. Changes include: