How to judge "prediction" according to "gold"?模型评测得分是否可靠?如何根据gold判断prediction好坏? #1227
Unanswered
chairmanQi
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
After evaluating _gen data set, how to judge the quality of "prediction" according to "gold" in the output json?
关注研究:
评测xx_gen数据集后,在输出的结果json中,如何根据gold来判断生成prediction的好坏呢?
![image](https://private-user-images.githubusercontent.com/86595682/336808913-bcefb296-53e8-4673-b00f-7b6c0ea90a65.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5NjMwMzMsIm5iZiI6MTcyMzk2MjczMywicGF0aCI6Ii84NjU5NTY4Mi8zMzY4MDg5MTMtYmNlZmIyOTYtNTNlOC00NjczLWIwMGYtN2I2YzBlYTkwYTY1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE4VDA2MzIxM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTQ4ZDMwM2MzODIzNmIyNjNjMTI5NmY3NTA1NGFlOGJiYTVjNjIzMGE3NTY1Mjg4ODZiNDM3M2NkYTAwZGFkMTQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.LASpkjyPLxlY8RRORoqFUGGVuPLzYD7RTlHbhmmF7lA)
背景:
针对于在评估数据集时保存不良案例Save bad cases when evaluating datasets工作(#1056 && #1086),以运行各种模型评测siqa_gen选择题为例,从预测输出文件siqa.json中寻找错例规律,推测gold中的label为正确答案对应的序号(0→2分别对应A→C)
问题描述:
如何判断模型预测的prediction是正确的呢?此前听开发者讲课介绍时有了解一种可能已经应用的方案是截取prediction中第一个字母作为预测答案
重点困惑
gen类数据集
gen类数据集上的评测得分,还存在以下问题:
![image](https://private-user-images.githubusercontent.com/86595682/336810436-d81efd37-f0e7-4b5f-b798-3b7ead21869b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5NjMwMzMsIm5iZiI6MTcyMzk2MjczMywicGF0aCI6Ii84NjU5NTY4Mi8zMzY4MTA0MzYtZDgxZWZkMzctZjBlNy00YjVmLWI3OTgtM2I3ZWFkMjE4NjliLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE4VDA2MzIxM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQ3MThkZjEzMTI4Y2NiN2FiOGMyZDk4MjQ1N2JlZDMxOWUyZmNmZGE0NzAzMmQ5ZDYyNjVmYmRjY2ZhZTU2ZWEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.H_9pZAzLiuYSzM7DWGAL1naI5i_pc9hCtdPE0lIR8pQ)
![image](https://private-user-images.githubusercontent.com/86595682/336810882-efce1a8d-8c04-4f13-945b-7348986a4b3f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5NjMwMzMsIm5iZiI6MTcyMzk2MjczMywicGF0aCI6Ii84NjU5NTY4Mi8zMzY4MTA4ODItZWZjZTFhOGQtOGMwNC00ZjEzLTk0NWItNzM0ODk4NmE0YjNmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE4VDA2MzIxM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWNkMGJjNGIzNDIyYzkzZDJlNDk0MTVhMGZkMDQ2M2Y2NDNjMjJiNmM1ODIxNDI3NWVkMzNlNTk5YjlmY2MwZGImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.K6vpUefMMIe1eeduCUbfMYl1Ev4losuJMpXVnaln8g0)
修改prompt使模型固定输出时依旧能获得37的评分,乃至于出现输出为空还能获得33分,连空输出都能得分吗?非常费解
ppl类数据集
类似的,虽然ppl类有明确的prediction与gold进行对比来判定是否相符,但分析输出结果发现模型经常对于完全相反的问题回复相同的答案,是否存在得分“蒙中”达50%正确以上情况?
![image](https://private-user-images.githubusercontent.com/86595682/336814086-48abb037-2400-4a60-a3fc-855d40831512.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5NjMwMzMsIm5iZiI6MTcyMzk2MjczMywicGF0aCI6Ii84NjU5NTY4Mi8zMzY4MTQwODYtNDhhYmIwMzctMjQwMC00YTYwLWEzZmMtODU1ZDQwODMxNTEyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE4VDA2MzIxM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTlmMmU3Y2E1YjNmODllZTliOGNlM2NjYjQxOTQzZTg1ZGIyYThlOWFmNzk0N2Y4NDVlZDVmZjYxMmE5MDFiNTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.DGwkMWek3nqdHLDHslHVEo51dzQY3_QMwBsxIjVI6eA)
![image](https://private-user-images.githubusercontent.com/86595682/336814684-ccd16e8d-e40b-44b0-b8e1-ddd03c179981.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM5NjMwMzMsIm5iZiI6MTcyMzk2MjczMywicGF0aCI6Ii84NjU5NTY4Mi8zMzY4MTQ2ODQtY2NkMTZlOGQtZTQwYi00NGIwLWI4ZTEtZGRkMDNjMTc5OTgxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE4VDA2MzIxM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTM3YTkwYmM3YmY5MDUxMThhZTU2MjIwNjc0NmI3ZTBkMGI5YmQzODRjMmU2ZGQ1OTI5ZGMxOGQ1YTFlZTQ4NGImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.QouI-Xn5IeEus9WCo3tt470MyXVcjj1pHoDkChH-TrE)
存在能够正确识别问题需求的,
但大多数都是对不同问题回复相同答案:
优先关注:
还是围绕着如何实现官方文档实用工具中的case_analyzer.py来实现错例筛选,延伸出主要想请问对于gen类数据集评测时是如何判定其生成正确?是以怎样的方法识别判定,能否证明模型在数据集评测得分可靠呢?
Beta Was this translation helpful? Give feedback.
All reactions