[FEAT]: Selected scenarios w better valuation score and split into train / test

[sharonwx54]

Description

Adding filtering layer to select dialogue/scenario that produces high overall score.

Additional Information

No response

[lwaekfjlk]

@ProKil @XuhuiZhou Do we have designed scenario or dialogue filtering strategy? I remember that you mentioned using distribution and variance for selection.

[XuhuiZhou]

@lwaekfjlk Yeah, that's more for selecting challenging scenarios. I don't think we have a filtering strategy for quality (if this is what you are asking for?) cuz we manually get rid of those scenarios that are not natural.

[sharonwx54]

Preliminary split by scenario, filtering by overall score, and selection of 2agent dialogues sit in the branch https://github.com/XuhuiZhou/sotopia/tree/feature/log-filtering-for-ft. Currently we are using 3.2 as a threshold value for dropping low performance dialogues, and the value is the overall reward values, not corresponding to any specific metrics such as believability or secret.