Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EC GO results #23

Closed
Heisenburger2020 opened this issue Mar 23, 2024 · 3 comments
Closed

EC GO results #23

Heisenburger2020 opened this issue Mar 23, 2024 · 3 comments

Comments

@Heisenburger2020
Copy link

Dear Sir,

Why is the EC GO result in Saprot so much lower than the original paper "Enhancing Protein Language Model with Structure-based Encoder and Pre-training"? I wonder if the dataset is different.

@LTEnjoy
Copy link
Contributor

LTEnjoy commented Mar 23, 2024

Hello!

Yes, our dataset is different from their original paper. We have mentioned that in our paper:
image

This pre-processing resulted in different datasets in all tasks such as EC and GO. Therefore the results reported in our paper should not be directly compared to other papers.

@LTEnjoy LTEnjoy closed this as completed Mar 28, 2024
@LTEnjoy
Copy link
Contributor

LTEnjoy commented Apr 18, 2024

Hi!

After careful examination, we found out that there was a silght difference for EC and GO evaluation.

Specifically, we copied the evaluation function from GearNet.
1713410596007
This function requires the input shape to be (B, N), where B is the number of proteins and N is the number of labels. However, our predictions and targets were flatten before evaluation, which means their shape were (1, B*N). This wouldn't cause an error but would lead to the reported results to be lower. Intuitively, this evaluation is like on a global level and previous evaluation is like an averaged result among proteins.

We have revised our evaluation code by reshaping the input tensors, as shown below:
image

Kindly note that our key conclusion will not change, namely that Saprot still remains a SOTA model under the new setting (will update new results soon).

Thank you again for pointing out such problem! :)

@LTEnjoy LTEnjoy reopened this Apr 18, 2024
@LTEnjoy
Copy link
Contributor

LTEnjoy commented Apr 19, 2024

Hi!
New results have been updated!

@LTEnjoy LTEnjoy closed this as completed Apr 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants