Human-AI Co-Learning

Human-AI Interaction vs Human-AI Co-Learning

Three Key Concepts of Human-AI Co-Learning

• Mutual Understanding: Human and AI are developing common knowledge (i.e., a shared mental model) through an iterative, interactive process.

• Mutual Benefits: Human and AI as a team achieves superior results that a single human or AI cannot achieve alone.

• Mutual Growth: Human and AI both have a growth mindset —i.e. they learn together, learn from each other, learn with each other, and grow and evolve over time.

AI sucess vs AI failure

• Slides: Human-AI Co-Learning for Data-Driven AI by Janet Huang, Oct 2019
• In-Progress Writing: Human-AI Co-Learning (Huang et al., 2019) https://arxiv.org/abs/1910.12544

---

Human-AI Collaboration

Literature for Human-AI collaboration, Hybrid Intelligence, Human-AI interaction

Human-Centered Artificial Intelligence

• Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy. (Shneiderman 2020)
• Human-Centered Artificial Intelligence: Three Fresh Ideas. (Shneiderman, 2020)
• Workshop: Human-Centered AI: Reliable, Safe & Trustworthy, University of Maryland, 2020.
• Bridging the Gap Between Ethics and Practice: Guidelines for Reliable, Safe, and Trustworthy Human-centered AI Systems

Interactive Machine Learning

• Power to the People: The Role of Humans in Interactive Machine Learning (AI Magine 2014)

Hybrid Intelligence

• Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence (Karmar, IJCAI 2016)
• Hybrid Intelligence (Dominik et al., 2019)
• The Future of Human-AI Collaboration: A Taxonomy of Design Knowledge for Hybrid Intelligence Systems (Dominik et al., 2019)
• On Facilitating Human-Computer Interaction via Hybrid Intelligence Systems (Lasecki, 2019)

Collective Intelligence / Crowdsourcing

• Human-Computer Interaction and Collective Intelligence (Bigham et al., 2014)
• Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research (2018)

Machine Teaching

• Machine Teaching A New Paradigm for Building Machine Learning Systems (2017)
• The Importance of UX for Machine Teaching (SSS18)
• Interactive Teaching Strategies for Agent Training (IJCAI 2016)

Agency vs Automation

• Man-Computer Symbiosis (Licklider, 1960)
• Direct Manipulation vs. Interface Agents (1997)
• Principles of Mixed-Initiative User Interfaces (Horvitz, 1999)
• Agency plus automation: Designing artificial intelligence into interactive systems (Heer, 2019)

Human-AI Interaction

• Guidelines for Human-AI Interaction (Amershi et al., CHI2019)
• Ask not what AI can do, but what AI should do: Towards a framework of task delegability (Lubars an Tan, 2019)

AI/ML as UX

• UX Design Innovation: Challenges for Working with Machine Learning as a Design Material (CHI 2017)
• Reimagining the Goals and Methods of UX for ML/AI
• The Role of Design in Creating Machine-Learning-Enhanced User Experience
• Investigating How Experienced UX Designers Effectively Work with Machine Learning (DIS 2018)
• Mapping Machine Learning Advances from HCI Research to Reveal Starting Places for Design Research (CHI 2018)
• Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design (Yang et al., CHI 2020)

Intelligibility, Explainability, Explainable AI

• How Good is 85%? A Survey Tool to Connect Classifier Evaluation to Acceptability of Accuracy (CHI 2015)

• "Why Should I Trust You?": Explaining the Predictions of Any Classifier (KDD 2016)

• Towards A Rigorous Science of Interpretable Machine Learning (2017)

• Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda (CHI 2018)

• Designing Theory-Driven User-Centric Explainable AI (CHI 2019)

• Explaining Decision-Making Algorithms through UI (CHI 2019)

• The Challenge of Crafting Intelligible Intelligence (Weld and Bansal, CACM 2019)

• Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning (Kaur et al., CHI 2020)

• Questioning the AI: Informing Design Practices for Explainable AI User Experiences (Liao et al., CHI 2020)

• No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML (Smith-Renner et al., CHI 2020)

• Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. (Buçinca et al., IUI 2020)

• (Tutorial Slides)"Introduction to Interpretable Machine Learning", Deep Learning Summer school at University of Toronto, Vector institute in 2018, by Been Kim

• (Tutorial Slides)Interpretable Machine Learning: The fuss, the concrete and the questions, Harvard university Tutorial by Finale Doshi-Velez and Been Kim, ICML 2017

FATE: Fairness, Accountability, Transparency, and Ethics in AI

• Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI (Madaio et al., CHI 2020)
• Principles for Accountable Algorithms and a Social Impact Statement for Algorithms
• Model Cards for Model Reporting (Mitchell et al., 2019)
• Improving fairness in machine learning systems: What do industry practitioners need? (Holstein et al., 2019)
• Google Cloud Model Cards
• FATE: Fairness, Accountability, Transparency, and Ethics in AI by MSR

Human-AI Team

• Learning to Complement Humans (Wilder et al., IJCAI 2020)
• Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. (Bansal et al., arXiv 2020)
• Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff (AAAI 2019)
• Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance (HCOMP 2019)
• Investigating human+machine complementarity for recidivism predictions. (Tan et al., 2018)

Human-in-the-loop vs Machine-in-the-loop (or AI-in-the-loop)

Human-in-the-loop around AI

• https://livebook.manning.com/book/human-in-the-loop-machine-learning/ (in-progress book, Robert Munro)
• A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores. (De-Arteaga et al., CHI 2020)

Machine-in-the-loop around Humans

• [Creative writing with a machine in the loop: Case studies on slogans and stories](IUI 2018)

Applications for Human-AI Co-Learning

Healthcare

• Human-centered tools for coping with imperfect algorithms during medical decision-making. (Cai et al., CHI 2019)
• Unremarkable AI: Fitting intelligent decision support into critical, clinical decision-making processes (Yang et al., CHI 2019)

Writing

• Metaphoria: An Algorithmic Companion for Metaphor Creation (CHI 2019)
• Creative writing with a machine in the loop: Case studies on slogans and stories (IUI 2018)
• LISA: lexically intelligent story assistant (AIIDE 2017)

Creativity (Drawing, Music, etc)

• In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound (CHI 2019)
• I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence (CHI 2018)

Design

• VisiBlends: An Interactive Pipeline for Creating Visual Blends (CHI 2019)
• Human Errors in Interpreting Visual Metaphor (C&C 2019)

Conversational Agents

• What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents (CHI 2019)
• Caring for Vincent: A Chatbot for Self-compassion(CHI 2019)

Teaching & Learning

• Co-designing a real-time classroom orchestration tool to support teacher–AI complementarity. (Holstein et al., 2019)
• A conceptual framework for human–AI hybrid adaptivity in education. (Holstein et al., 2020)

Others

• Calendar.help: Designing a workflow-based scheduling agent with humans in the loop
• AI-Mediated Communication: How the Perception that Profile Text was Written by AI Affects Trustworthiness (CHI 2019)

---

Courses

• Human-AI Interaction, by Chinmay Kulkarni an Mary Beth Kery, CMU (former class by Jeff Bigham and Joseph Seering, Fall 2018)
• CS 889: Human-AI Interaction by Edith Law, U.Waterloo
• CS 294: Fairness in Machine Learning by ,UC Bekeley
• CS598RK: HCI for Machine Learning, by Ranjitha Kumar and Jinda Han,UIUC
• Human-centered Machine Learning (2018 Spring), by Chenhao Tan, University of Colorado Boulder
• CS6724: Advanced Topics in Human-Computer Interaction: Human-AI Interaction, by Kurt Lurther, Virginia Tech
• Designing AI to Cultivate Human Well-Being, by Jennifer Aaker and Fei-Fei Li, Stanford
• CS279R: Research Topics in HCI: Human-AI Interaction
• CS492F: Human-AI Interaction, by Jean Young Song and Juho Kim, KAIST
• CS279R Research Topics in HCI: Human-AI Interaction, by Elena L. Glassman, Harvard

Labs

• Polo Club of Data Science @ Georgia Tech
• Glassman Lab @ Harvard SEAS
• CMU Co-Augmentation, Learning, & AI (CoALA) Lab
• AI4LIFE @ Harvard

Workshops & Tutorials

• PAIR 2019 @Google
• HAI 2019 @Stanford
• Machine Teaching Group @MSR
• HAI-GEN 2020 @ IUI 2020 Workshop on Human-AI Co-Creation with Generative Models
• AAAI 2020 Tutorial – Guidelines for Human-AI Interaction
• Participatory Approaches to Machine Learning, ICML 2020 Workshop (July 17, 2020)

Others Links

• https://pair.withgoogle.com
• AI Guidelines in the Creative Process
• https://hcil.umd.edu/human-centered-ai-a-second-copernican-revolution/
• https://hcil.umd.edu/human-centered-ai-trusted-reliable-safe/
• https://machinelearning.design

Other Awesome HAI list

• https://github.com/bwang514/awesome-HAI/blob/master/README.md

---

For artist

• https://ml4a.github.io/classes/itp-F18/