Next Steps

[pszemraj]

Next Steps

Until this thread gets deleted/closed, it will be used for notes, thoughts etc on the next steps with this repo/concept

initial list

• clean up & consolidate the dataset: the initial compilation has a lot of metadata columns as a result of the many APIs used to generate the summaries. These need to be cleaned up
• analysis/testing of the computed zero-shot writing quality scores. A start to this is here but things like statistical analysis & controlling for the 'best' params for each run need to be done
• revisit Max Salient Similarity - this should now be much easier to troubleshoot & implement with the dataset structure
• improve the codebase - the initial code was done quickly for the sake of actually getting a version of the dataset out there. It can be improved and cleaned up, and the repo better organized

[pszemraj]

'traditional' evaluation via GPT-4 summaries

idea: pretend the GPT-4 summaries are ground-truth/golden summaries. compute metrics on everything else.

• compute metrics using GPT-4 summaries as "golden"/ground truth (refer to: summeval repo)
• compile multiple human perspectives on how good the GPT-4 summaries actually are (anything obviously wrong/weird in them that we see right away?)