Query on how to test on custom prompts

[yuezhao238]

Hi, thanks for this great work!

I am using this benchmark suite to compare performance of different models. However, I didn't find .py file to extract "subject", "subject_noun", "place", "action" from my original prompt.

Do you have plan to release the code to help me extract these keywords more conveniently?

Thanks again!

[chenzhu01]

Hi, thanks for your interest!

In our benchmark, the "subject", "subject_noun", "place", and "action" are sampled from collected meta information, and then used to generate the corresponding prompts.

To extract these meta fields from existing prompts, our process uses an LLM to parse the prompt, followed by some manual cleaning to ensure accuracy.