By Leena Mathur from the Language Technologies Institute at CMU's School of Computer Science.
This repository contains resources related to advancing socially-intelligent AI (Social-AI) agents. If there are any topics, papers, books, benchmarks, courses, or dissertations you would like added, please feel free to make a pull request or email lmathur@andrew.cmu.edu. All suggestions or contributions are welcome!
This repository accompanies the position paper Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions by Leena Mathur, Paul Liang, and Louis-Philippe Morency from the Language Technologies Institute and Machine Learning Department at CMU.
The position paper discusses core technical challenges, along with opportunities and open questions, towards advancing social intelligence in AI agents. Our paper is anchored in social intelligence concepts and progress in Social-AI across 6 computing communities: natural language processing, machine learning, computer vision, robotics, human-machine interaction (including human-computer interaction and human-robot interaction), and speech. Social-AI research interest has accelerated across computing communities in recent years:
Cumulative number of Social-AI papers over time, based on 3,257 papers from Semantic Scholar Social-AI queries. Social-AI research interest has been increasingly rapidly!
We believe there are core technical challenges that are particularly relevant to advancing social intelligence in AI agents with a variety of embodiments, social attributes, and roles, interacting in a range of social contexts.
(A) Four core technical challenges in Social-AI research, illustrated in an example context of a Social-AI agent observing and learning from a human-human interaction. (B) Social contexts in which Social-AI agents can be situated, spanning interaction dimensions/structures, social settings, degrees of agent embodiment, and social attributes of humans, with agents in several roles.
C1: Ambiguity in Constructs (Section 4.1 of the paper)
Social constructs have inherent ambiguity in their definition and interpretation in the social world.
C2: Nuanced Signals (Section 4.2 of the paper)
Social constructs are expressed through behaviors and signals that can be nuanced, often manifesting through different degrees of synchrony across actors and modalities. During interactions, small changes in social signals can lead to large shifts in social meaning being conveyed.
C3: Multiple Perspectives (Section 4.3 of the paper)
In social interactions, actors bring their own perspectives, experiences, and roles; these factors can change over time and influence the perspectives of other actors during interactions.
C4: Agency and Adaptation (Section 4.4 of the paper)
Actors learn from social experiences and adapt to social contexts, through interactions, influenced by their own agency, goals, motivations, and identities.
@misc{mathur2024advancing,
title={Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions},
author={Leena Mathur and Paul Pu Liang and Louis-Philippe Morency},
year={2024},
eprint={2404.11023},
archivePrefix={arXiv},
primaryClass={cs.HC}
}Social Intelligence Foundations
Ethics, Safety, and Participatory Social-AI
The Construction of Social Reality, 1995
Social Ontology and the Philosophy of Society, Analyse & Kritik, 1998
The Evolutionary Emergence of Language: Social Function and the Origins of Linguistic Form, 2000
Introduction. Social Intelligence: From Brain to Culture, Philosophical Transactions of the Royal Society, 2007
Social Intelligence, Human Intelligence and Niche Construction, Philosophical Transactions of the Royal Society, 2007
Making the Social World: The Structure of Human Civilization, 2010
Three Kinds of Social Kinds, Philosophy and Phenomenological Research, 2013
Human Social Reality and Language, Phenomenology and Mind, 2012
Moral Principles in Education, 1909
Moral Instruction through Social Intelligence, American Journal of Sociology, 1911
Intelligence and Its Uses, Harper's Magazine, 1920
Measures of Social Intelligence, American Journal of Sociology, 1930
An Evaluation of the Attempts to Measure Social Intelligence, Psychological Bulletin, 1937
Social Intelligence – A Review and Critical Discussion of Measurement Concepts, Emotional Intelligence: An International Handbook, 2005
Theory and Measurement of Social Intelligence as a Cognitive Performance Construct, Susanne Weis PhD Dissertation, 2008
New Findings about Social Intelligence, Journal of Individual Differences, 2013
The Social Shapes Test: A New Measure of Social Intelligence, Mentalizing, and Theory of Mind, Personality and Individual Differences, 2019
We consider the following 6 competencies to be core competencies of social intelligence: Social Perception, Knowledge, Memory, Reasoning, Creativity (Theory-of-Mind), Interaction. This perspective is informed by readings from cognitive science, psychology, and neuroscience.
Social Perception, 1990
Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing, IEEE Transactions on Affective Computing, 2011
Nonverbal Signals, Handbook of Interpersonal Communication, 2011
Social Signals: A Framework in Terms of Goals and Beliefs, Cognitive Processing, 2012
Data-driven Approaches in the Investigation of Social Perception, Philosophical Transactions of the Royal Society B: Biological Sciences, 2016
Thinking about Ourselves and Others: Self-monitoring and Social Knowledge, Journal of Personality and Social Psychology, 1980
A Proposed Model for the Acquisition of Social Knowledge and Social Competence, Psychology in the Schools, 1993
Social Memory in Everyday Life: Recall of Self-events and Other-events, Journal of Personality and Social Psychology, 1991
Self and Social Functions: Individual Autobiographical Memory and Collective Narrative, Memory, 2003
Constraint Satisfaction Processes in Social Reasoning, Proceedings of the 25th Annual Cognitive Science Society, 2003
Reasoning Strategies Explain Individual Differences in Social Reasoning, Journal of Experimental Psychology: General, 2021
Theory of Mind Development and Social Understanding, Cognition and Emotion, 2008
A Social Perspective on Theory of Mind, Handbook of Child Psychology and Developmental Science, 2015
A Theory of Social Interaction, 1988
Interaction, Chapter 13 within Handbook of Symbolic Interactionism, 2003
Can Social Interaction Constitute Social Cognition?, Trends in Cognitive Science, 2010
Social-AI agents can be situated within interactions spanning social units, interaction structures, and timescales. Interactions can span social settings, degrees of agent embodiment, and social attributes of humans, with agents in several roles.
Social identity shapes social perception and evaluation, Neuroscience of Prejudice and Intergroup Relations, 2013
Social Identity Theory,Psychology of Entertainment, 2006
Social Identity Theory, 2016
Difference Matters: Communicating Social Identity, 2023
The Presentation of Self in Everyday Life, 1959
Action and Embodiment within Situated Human Interaction, Journal of Pragmatics, 2000
The Role of Physical Embodiment in Human-Robot Interaction, IEEE RO-MAN, 2006
Grounding in Communication, Perspectives on Socially Shared Cognition, 1991
Shared Reality: Experiencing Commonality with Others' Inner States about the World, Perspectives on Psychological Science, 2009
Embodiment in Socially-Interactive Robots, Foundations and Trends in Robotics, 2019
Models of the Interaction of Language and Social Life, 1972
Interpretation as a Communicative Event: A Look Through Hymes' Lenses, Meta, 2000
Language and Social Relations, 2006
Understanding Dialogue: Language Use and Social Interaction, 2021
Phases, Transitions and Interruptions: Modeling Processes in Multi-party Negotiations, International Journal of Conflict Management, 2003
Social Influence Network Theory: A Sociological Examination of Small Group Dynamics, 2011
Detecting, Measuring, and Testing Dyadic Patterns in the Actor--Partner Interdependence Model, Journal of Family Psychology, 2019
Social Moments: A Perspective on Interaction for Social Robotics, Frontiers in Robotics and AI, 2017
Note: This section will be periodically updated with representative papers. Pull requests are always welcome, too
Elementary Contracts as a Pragmatic Basis of Language Interaction, COLING, 1986
Linguistic Issues in Facial Animation, Computer Animation, 1991
Abductive explanation of dialogue misunderstandings, EACL, 1993
Social Interaction: Multimodal Conversation with Social Agents, AAAI, 1994
Animated Conversation: Rule-Based Generation of Facial Expression, Gesture & Spoken Intonation for Multiple Conversational Agents, SIGGRAPH, 1994
Generating Facial Expressions for Speech, Cognitive Science, 1996
Cooperation Structures, IJCAI, 1997
Modeling Social Action for AI Agents, Artificial Intelligence, 1998
A Computational Model of Social Perlocutions, ACL/COLING, 1998
Multi-agent planning as a dynamic search for social consensus, IJCAI 1993
Designing Emergent Behaviors: From Local Interactions to Collective Intelligence, International Conference on Simulation of Adaptive Behavior: From Animals to Animats, 1993
Learning to Behave Socially, International Conference on Simulation of Adaptive Behavior: From Animals to Animats, 1994
How to Build Robots That Make Friends and Influence People, IROS 1999
Toward Sociable Robots, 2003
Designing Sociable Robots, 2004
Toward Virtual Humans, AI Magazine, 2006
Latent-dynamic Discriminative Models for Continuous Gesture Recognition, CVPR, 2007
Social Signal Processing: Survey of an emerging domain, Image and Vision Computing, 2009
Improving Data Association by Joint Modeling of Pedestrian Trajectories and Groupings, ECCV, 2010
Towards Multimodal Sentiment Analysis: Harvesting Opinions from the Web, ACM ICMI, 2011
AVEC 2012: The Continuous Audio/visual Emotion Challenge, ACM ICMI, 2012
Note: AVEC has occurred several times as a workshop.
Learning the Communication of Intent Prior to Physical Collaboration, IEEE RO-MAN, 2012
Social Signal Classification Using Deep BLSTM Recurrent Neural Networks, ICASSP 2014
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing, IEEE Transactions on Affective Computing, 2015
Coordinate to Cooperate or Compete: Abstract Goals and Joint Intentions in Social Interaction, Cognitive Science, 2016
Commonsense Interpretation of Triangle Behavior, AAAI, 2016
Active Preference-based Learning of Reward Functions, RSS, 2017
Personalized Machine Learning for Robot Perception of Affect and Engagement in Autism Therapy, Science Robotics, 2018
Multimodal Language Analysis in the Wild: CMU-Mosei Dataset and Interpretable Dynamic Fusion Graph, ACL, 2018
Social-bigat: Multimodal Trajectory Forecasting Using Bicycle-gan and Graph Attention Networks, NeurIPS, 2019
Gaitset: Regarding Gait as a Set for Cross-view Gait Recognition, AAAI, 2019
Dialoguernn: An Attentive RNN for Emotion Detection in Conversations, AAAI, 2019
Multimodal Analysis and Estimation of Intimate Self-Disclosure, ACM ICMI, 2019
Social Influence as Intrinsic Motivation for Multi-agent Deep Reinforcement Learning, ICML, 2019
Theory of Minds: Understanding Behavior in Groups through Inverse Planning, AAAI, 2019
Too Many Cooks: Coordinating Multi-agent Collaboration through Inverse Planning, Cognitive Science, 2020
Joint Attention for Multi-agent Coordination and Social Learning, ICRA Workshop on Social Intelligence in Humans and Robots, 2021
Learning To Listen: Modeling Non-Deterministic Dyadic Facial Motion, CVPR, 2022
Gesture2path: Imitation Learning for Gesture-aware Navigation, arXiv, 2022
Observer-aware Legibility for Social Navigation, RO-MAN, 2022
Social-iq: A Question Answering Benchmark for Artificial Social Intelligence, CVPR, 2019
Revisiting the Evaluation of Theory of Mind through Question Answering, EMNLP, 2019
Socialiqa: Commonsense Reasoning about Social Interactions, EMNLP, 2019
Human-centric Dialog Training via Offline Reinforcement Learning, EMNLP, 2020
A Simple Language Model for Task-oriented Dialogue, NeurIPS, 2020
Language Model Transformers as Evaluators for Open-domain Dialogues, COLING, 2020
Exploring RoBERTa's Theory of Mind through Textual Entailment, 2021
Neural Theory-of-mind? On the Limits of Social Intelligence in Large LMs, EMNLP, 2022
Affective Behavior Learning for Social Robot Haru with Implicit Evaluative Feedback, IROS, 2022
Social-iq 2.0 Challenge: Benchmarking Multimodal Social Understanding, ICCV Challenge, 2023
The Socialai School: Insights from Developmental Psychology towards Artificial Socio-cultural Agents, arXiv, 2023
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions, EMNLP, 2023
NormBank: A Knowledge Bank of Situational Social Norms, ACL, 2023
Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models, EACL, 2024
Building Cooperative Embodied Agents Modularly with Large Language Models, ICLR 2024
Sotopia: Interactive Evaluation for Social Intelligence in Language Agents, ICLR, 2024
MMToM-QA: Multimodal Theory of Mind Question Answering,arXiv, 2024
Habitat 3.0: A Co-habitat for Humans, Avatars and Robots, ICLR 2024
Note: This section includes representative work and is being periodically updated.
Human--AI collaboration Enables More Empathic Conversations in Text-based Peer-to-peer Mental Health Support, Nature Machine Intelligence 2023
Wellbeat: A Framework for Tracking Daily Well-being Using Smartwatches, IEEE Internet Computing, 2020
Social Robots in Hospitals: A Systematic Review, Applied Sciences, 2021
Socially Assistive Robotics for Post-stroke Rehabilitation, Journal of NeuroEngineering and Rehabilitation, 2007
Social and Emotional Skills Training with Embodied Moxie, arXiv, 2020
Robots for Use in Autism Research, Annual Review of Biomedical Engineering, 2012
A Robotic Positive Psychology Coach to Improve College Students’ Wellbeing, IEEE RO-MAN, 2020
Lifelong Personalization for Social Robot Learning Companions: Interactive Student Modeling Across Tasks and Over Time, PhD Thesis, 2022
The Social Impact of a Robot Co-worker in Industrial Setting, CHI, 2015
Machines and Mindlessness: Social Responses to Computers, Journal of Social Issues, 2000
Beyond Dirty, Dangerous and Dull: What Everyday People Think Robots Should Do, HRI, 2008
Averting robot eyes, Maryland Law Review, 2016
Social Bias Frames: Reasoning about Social and Power Implications of Language, ACL, 2020
Towards Transparency by Design for Artificial Intelligence, Science and Engineering Ethics, 2020
Towards Understanding and Mitigating Social Biases in Language Models, ICML, 2021
Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases, arXiv, 2021
Envisioning Communities: A Participatory Approach towards AI for Social Good, AIES, 2021
Unmasking the Mask--Evaluating Social Biases in Masked Language Models, AAAI, 2022
Power to the People? Opportunities and Challenges for Participatory AI, EAAMO, 2022
Stable Bias: Evaluating Societal Representations in Diffusion Models, NeurIPS, 2023
Survey of Social Bias in Vision-Language Models, arXiv, 2023
Dall-eval: Probing the Reasoning Skills and Social Biases of Text-to-image Generation Models, ICCV, 2023
Never Trust Anything That Can Think for Itself, if You Can’t Control Its Privacy Settings: The Influence of a Robot’s Privacy Settings on Users’ Attitudes and Willingness to Self-disclose, International Journal of Social Robotics, 2023
Using Design Metaphors to Understand User Expectations of Socially Interactive Robot Embodiments, ACM Transactions on Human-Robot Interation, 2023
Federated Continual Learning for Socially Aware Robotics, IEEE RO-MAN, 2023
Note: This section is being periodically updated. Pull requests are always welcome, too
| Dataset | Modality and/or Domain | Paper | Data/Code |
|---|---|---|---|
Social-IQ |
multimodal video qa | CVPR 2019 paper | data + code |
Social-IQ 2.0 |
multimodal video qa | ICCV 2023 Challenge | data + code |
Social-IQa |
text qa | EMNLP 2019 paper | data + code |
CMU-MOSEI |
multimodal sentiment and emotion intensity | ACL 2018 paper | data + code |
IEMOCAP |
multimodal emotional dyadic motion capture | LREC 2008 paper | data |
GENEA |
virtual agent gesture generation | ICMI 2023 paper | website |
SocNavBench |
robot social navigation simulation | THRI 2022 paper | website |
Habitat 3.0 |
simulated human-robot social navigation and object rearrangement | ICLR 2024 paper | website |
Note: This section is being periodically updated. Pull requests to add courses are always welcome, too
11:866: Artificial Social Intelligence, Carnegie Mellon University
CMU offers a new course 11:866: Artificial Social Intelligence, most recently taught in Spring 2023. There are publicly-available summaries from class discussions and reading lists for anyone interested in Social-AI topics.
Multimodal Probabilistic Learning of Human Communication, University of Southern California
Affective Computing: An Interdisciplinary Approach, University of Southern California
Affective Computing and Ethics, MIT
Note: This section is not exhaustive and is being periodically updated. Pull requests to add dissertations are always welcome, too
Communication and Coarticulation in Facial Animation, 1991, Catherine Pelachaud
Interaction and Intelligent Behavior, 1994, Maja J Matarić
Sociable Machines: Expressive Social Exchange Between Humans and Robots, 2000, Cynthia Breazeal
Foundations for a Theory of Mind for a Humanoid Robot, 2001, Brian Scassellati
Socially guided Machine Learning, 2006, Andrea Thomaz
Context-Based Visual Feedback Recognition, 2006, Louis-Philippe Morency
Vision-Based Multimodal Analysis of Affective Face and Upper-Body Behaviorl 2007, Hatice Gunes
Implicit and Automated Emotional Tagging of Videos, 2011, Mohammad Soleymani
Computers to Help with Conversations: Affective Framework to Enhance Human Nonverbal Skills, 2013, Ehsan Hoque
Measuring college students' sleep, stress, mental health and wellbeing with wearable sensors and mobile phones, 2016, Akane Sano
Nonverbal Communication in Socially Assistive Human-Robot Interaction, Henny Admoni
A Bayesian Theory of Mind Approach to Nonverbal Communication for Human-Robot Interactions, 2017, Jin Joo Lee
Cooperative and Transparent Machine Learning for the Context-Sensitive Analysis of Social Interactions, 2018, Tobias Baur
Computational Foundations of Human Social Intelligence, 2018, Max Kleiman-Weiner
Social and Affective Machine Learning, 2019, Natasha Jaques
Social Scene Understanding: Group Activity Parsing, Human-Robot Interactions, and Perception of Animacy, 2019, Tianmin Shu
Computational Social Roles, 2019, Diyi Yang
Modeling Visual Minutiae: Gestures, Styles, and Temporal Patterns, 2020, Shiry Ginosar
Positive AI with Social Commonsense Models, 2021, Maarten Sap
Towards Human-Centered Optimality Criteria, 2021, Asma Ghandeharioun
Lifelong Personalization for Social Robot Learning Companions: Interactive Student Modeling Across Tasks and Over Time, 2022, Sam Spaulding
Communication Beyond Words: Grounding Visual Body Motion with Language, 2022, Chaitanya Ahuja
Conversation Modeling with Human Values, Social Relations, Mental States, and Structure Learning, 2022, Liang Qiu
Synthesis of Multi-Modal Socially Intelligent Human-Robot Interaction, 2022, Karen Tatarian
VirtualHome: Building Socially Intelligent Agents via Simulation, 2023, Xavier Puig
Towards Artificial Social Intelligence in the Wild: Sensing, Synthesizing, Modeling, and Perceiving Nonverbal Social Human Behavior, 2023, Chirag Raman