Use eval_result/metadata for comparision in case of dml/ddl

[hardikgu23]

We are currently storing eval_query output for dml queries and complete schema metadata for ddl (#87) . Using the execution result of these to compare if generated query was correct or not (as dml/ddl execution do not return any result on their execution)

[IsmailMehdi]

_ No description provided. _

Please have descriptions in you PRs.

[hardikgu23]

Added description.

[mahyareb]

@hardikgu23 instead of overriding generated and golden result, could you just pass eval_results and golden_eval_results to scorer in comp.compare

And as of right now, we only should use exact_match scorer for DML / DDL. So we could add a flag in the config like

...
scorers:
  exact_match:
     use_eval_sql: true
...

@IsmailMehdi what do you think?

[IsmailMehdi]

@hardikgu23 instead of overriding generated and golden result, could you just pass eval_results and golden_eval_results to scorer in comp.compare

And as of right now, we only should use exact_match scorer for DML / DDL. So we could add a flag in the config like

...
scorers:
  exact_match:
     use_eval_sql: true
...

@IsmailMehdi what do you think?

Yes that sounds reasonable

[hardikgu23]

@hardikgu23 instead of overriding generated and golden result, could you just pass eval_results and golden_eval_results to scorer in comp.compare

And as of right now, we only should use exact_match scorer for DML / DDL. So we could add a flag in the config like

...
scorers:
  exact_match:
     use_eval_sql: true
...

@IsmailMehdi what do you think?

Acknowledged. Have modified the pr to do the same.
Here is the sample run id: cbb68549-323b-4669-b588-bdf084e6d186

[mahyareb]

LGTM! Just left a comment for one minor thing to fix. Thanks @hardikgu23 ! 👏