TOT-SQL Safeguard submission#56
Conversation
|
Hi @NG-VikasV — thank you for your contribution! Would you mind also sharing the traces for all runs across all queries (for example, as a zip file)? Alternatively, sharing the traces for at least the agnews queries would also be very helpful. We’ll use them for validation checks. Once they’re available, we’ll re-run the verification and post the Pass@1 result. |
|
Hi @Ruiying-Ma, I have attached the same (attached all traces including agnews), please let us know if you need anything more. Thank you. |
|
Hi @NG-VikasV — thanks for sending the traces. We went through them and have two things to sort out before we can compute Pass@1. First, run coverage. The leaderboard score averages over 5 runs per query, and we require at least 5 runs per query. The bundle currently contains a single run per query (54 traces total), and the submission JSON repeats that one answer across all 5 run slots. Could you re-run each query at least 5 times and share the per-run traces and answers? Second, the submission JSON does not match the traces for two agnews queries. For query 1 the submission lists Once we have the 5-run traces and a reconciled submission file, we'll re-run the verification and post the Pass@1 result. |
|
Hi, @Ruiying-Ma Thank you for the detailed feedback — apologies for the issues with the initial submission. Both points have been addressed: Please find the updated files committed directly to the PR branch: submissions/TT_SQL_V2_traces_all_runs.zip — all 270 per-run traces (54 queries × 5 runs), structured as traces/{dataset}/query{id}/run{0–4}/ Direct link: Click here Best regards, Vikas |
|
Hi @Ruiying-Ma, We have not yet received any update regarding our submission. We are fully prepared to make any corrections, revisions, or improvements required to support the review process. As a company based in Hyderabad, Telangana, India, we would appreciate it if you could kindly acknowledge receipt of our submission and provide an update on its current status. We look forward to your response and are available to address any feedback promptly. Thanks & Regards,
|
|
Hi @Ruiying-Ma, I think you might have missed our previous results, nevertheless, we are attaching our new runs and their traces, this time we have used any "hint' in contrast to our earlier submission. Please find our submission for "DataAgentBench" leaderboard below. Our Results: 54 queries × 5 runs = 270 total run slots · Pass@1 47.0% · Pass@K (K=5) 70.4% Submission answers (270 runs): click here Model: gpt-oss-safeguard-120b via AWS Bedrock · Thanks & Regards,
|
|
Hi @NG-VikasV, thanks for the traces. Apologies for my late reply. Thank you for your patience. We found a data leakage issue in at least 79/270 trials: At the final answer step (the SELF_CORRECTOR / "CONCISE ANSWER" stage) the prompt handed to the model includes a line like We list all possible leakage we could find in the table below. Happy to re-verify once they're ready! Full list of trials where the gold value was injected (79)
|
…ation Fixes the data leakage flagged in review (issuecomment-4727219915): - Removed db_description_withhint.txt re-injection into the self-corrector context, which was the source of literal ground-truth values appearing as 'GROUND TRUTH HINT (format only, not the answer)' in prompts. - Answer-format guidance (_derive_answer_format_hint) now derives hints purely from the question text, never from ground truth or expected answers. - Full fresh 5-run evaluation (270 trials, 54 official queries x 5 runs) with the fixed pipeline. Average pass@1: 46.7%, pass@5: 55.6%. - Each trial graded by its own official validate.py (dynamic exec).
…json as canonical
|
Hi @Ruiying-Ma, Thanks for catching this — confirmed and fixed. Fix: The final-answer stage (extract_answer()) takes no ground_truth parameter — structurally cannot inject a gold value. Answer-format guidance is derived only from the question's wording, never the answer. ground_truth/validate_src are read only post-hoc for grading, never passed into any prompt. Verification: Scanned all 270 real trace logs (this resubmission) for any occurrence of each query's gold value in the LLM-facing portion of the prompt. Found 103 matches, manually inspected every one — all are either (a) real DB values/stats the agent discovered live, (b) the benchmark question's own multiple-choice wording, (c) the agent's own correct SQL results echoed in self-correction, or (d) short-string coincidences inside unrelated boilerplate (e.g. a 2-char value inside "LIMIT"/"MINIMAL"). None match the original leak pattern (a pre-answer labeled hint). Each graded by its own official validate.py. Updated JSON: submissions/tot_sql_safeguard.json (commit 6a61fb8) Full traces (all 270 trials): |
Stratified Pass@1 0.4547 (PR #56), re-validated leakage-free after the ground-truth hint injection was removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CsYefzESSsuuN3hyWCawK
|
Thank you @NG-VikasV ! We've added your submission to the leaderboard. |
|
Hi @Ruiying-Ma, Thank you very much for the update. There is a small request from our side, I request you to please change the Names in the leader board as """ That will be really helpful for us. Regards, |
Per submitter request in PR #56. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CsYefzESSsuuN3hyWCawK
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018CsYefzESSsuuN3hyWCawK
|
Hi @NG-VikasV ! We've updated the leaderboard with the proposed name. Thanks for your contribution to DAB leaderboard! |
2 participants
Agent Name: TOT-SQL Safeguard
Backbone LLM: openai.gpt-oss-safeguard-120b
Dataset Hints Used: No
A reasoning-based agent architecture executing multi-step Data-to-SQL tasks. It leverages LLM reasoning to dynamically construct SQL queries, execute execution-verification loops, and retrieve correct query results across SQLite, PostgreSQL, DuckDB, and MongoDB databases without using any hardcoded schemas or domain-specific hints.