-
Notifications
You must be signed in to change notification settings - Fork 161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Aggregation Prediction uses select column from the ground truth #12
Comments
Here is the code snippet to replace the ground truth to the prediction.
|
@deepcoord Thanks for the code snippet. If I'm not mistaken, with your fix, sel_score is computed twice:
Wouldn't it be simpler to just reverse the order in which sel_score and agg_score are computed in the forward method in sqlnet.py ? By doing so, we could compute sel_score only one time and add a shorter snippet of code just before predicting the aggregator :
Am I missing something? |
Thanks for recognizing the issue! Yes, this will lead to a wrong break-down result on aggregator but will not affect the overall acc_qm or acc_ex. I will fix the bug soon. |
The below code snippet looks like using the column position for select in evaluation as well as training.
SQLNet/sqlnet/utils.py
Line 204 in 5dfb96e
SQLNet/sqlnet/model/sqlnet.py
Line 131 in 5dfb96e
SQLNet/sqlnet/model/modules/aggregator_predict.py
Line 41 in 5dfb96e
Don't you think the model should predict the column for select instead of the column given as ground truth before aggregation prediction?
In fact, the select column will not be given in prediction time when applying this to real worlds.
You made selection prediction, so the output could be fed with the aggregation prediction.
In result, the evaluation number might be wrong while comparing with the original paper, seq2sql.
The text was updated successfully, but these errors were encountered: