MultiSentimentArcs:

A Novel Method to Visualize Multimodal AI Sentiment Arcs in Long-Form Narratives

Open-Source AI, LLM, LMM, Multimodal AI, Affective AI, Sentiment Analysis, Narrative, Storytelling

Abstract

Affective AI and Sentiment Analysis play critical roles in designing safe and effective human computer interactions. It can be found in applications ranging from tutor chatbots to the physical robots used in eldercare. There is a growing concern, however, that increasingly capable AI will be able to manipulate, persuade, and otherwise compromise human autonomy. Rapid progress in AI is providing a constant stream of new and more capable LLM/LMMs that can better understand nuanced, complex, and interrelated sentiments across different modalities including text, vision, and speech. This paper introduces MultiSentimentArcs, a novel methodology that combines sentiment analysis, time series transformations, narrative studies, and leading open-source AI models to analyze two main modalities of sentiment expression in long-form video narratives. Although we use Hollywood films to demonstrate the technique, it can generalize to any long-form multimodal narrative like those found on social media (text, images), in video conversations (text, image, voice), or in medical settings (text, data, voice, image). To the best of our knowledge, MultiSentimentArcs is the first framework to integrate multimodal Affective AI that enables human-in-the-loop exploration, analysis, and explanation of long-form narratives. To support the democratization of AI, all components are open-source and can run on mid-range consumer gaming laptops. This research can significantly advance the field of Digital Humanities by giving non-AI experts access to directly engage in human-in-the-loop research around Affective AI and human-AI alignment. Code, results, and non-copyrighted data will be available at https://github.com/jon-chun/multisentimentarcs.

Royal Wedding (1951) Video Sentiment Arcs

Royal Wedding (1951) Transcript Sentiment Arcs

Royal Wedding (1951) Video Sentiment KDE Distribution

Royal Wedding (1951) Transcripts Sentiment KDE Distribution

NOTE:

This repo is in the process of being cleaned-up and reorganized. Please be patient