AIStorySimilarity: Quantifying Story Similarity Using Narrative for Search, IP Infringement, and Guided Creativity

Author: Jon Chun
Conference: Proceedings of the 28th Conference on Computational Natural Language Learning (CoNLL)
Anthology ID: 2024.conll-1.13
Month: November
Year: 2024
Location: Miami, FL, USA
Editors: Libby Barak, Malihe Alikhani
Publisher: Association for Computational Linguistics
Pages: 161–177

Abstract

Stories are central for interpreting experiences, communicating, and influencing each other across domains like film, medicine, media, and more. Quantifying story similarity has numerous applications, including detecting IP infringement, identifying hallucinations, improving search/recommendation engines, and guiding human-AI collaborations. Traditional NLP similarity metrics are often limited, using short-text distance metrics like n-gram overlaps or embeddings, which can lose significant information when applied to long texts.

This paper introduces AIStorySimilarity, a novel benchmark for measuring semantic distance between long-text stories using narrative theory and scriptwriting fundamentals. We base our method on four core narrative elements—characters, plot, setting, and themes—and 31 sub-features within these categories. Using a state-of-the-art LLM (GPT-3.5-turbo), we extract and evaluate semantic similarity for a diverse set of Hollywood films.

Our approach compares three methods: evaluating extracted elements from film scripts (Elements), directly evaluating entire scripts (Scripts), and extracting elements from the parametric memory of a state-of-the-art LLM without provided scripts (GenAI). To our knowledge, AIStorySimilarity is the first benchmark to measure story similarity over long texts comprehensively grounded in narrative theory.

Code and data are available at: https://github.com/jon-chun/AIStorySimiliarity.

Citation

• ACL Citation:

  Jon Chun. 2024. AIStorySimilarity: Quantifying Story Similarity Using Narrative for Search, IP Infringement, and Guided Creativity. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 161–177, Miami, FL, USA. Association for Computational Linguistics.

• Informal Citation:

  AIStorySimilarity: Quantifying Story Similarity Using Narrative for Search, IP Infringement, and Guided Creativity (Chun, CoNLL 2024)

• Bibkey: Not provided

Links

• ACL Anthology URL: https://aclanthology.org/2024.conll-1.13
• PDF Download: https://aclanthology.org/2024.conll-1.13.pdf

Additional Information

• Volume: 28
• Venue: CoNLL
• Language: English
• SIG: Not provided
• DOI: Not provided
• Note: Not provided