BADA : The social Robot for Hearing-impaired people.

2019-2020 Robot Capstone Project

2020 CAPSTONE DESIGN FAIR Grand prize @ Hanyang University ERICA.

Team : [사과가 쿵!]

Member : 유호연, 김민우, 배종학, 이현우, 최수진, 황지원

Directed by Prof. 한재권

Overview

BADA is a Social Robot made for Hearing-impaired people. The word 'BADA' is derived from "Buddy for Auditory Detection Assistance" and South Korean first assistance dog's name.

This project is inspired by hearing dog which can assists hearing-impaired people by acting from the situation of things and modeled on their behaviors. If BADA detects dangerous sounds such as Fire alarm, this robot firstly send sound information to BADA Web page and KAKAO Talk message for people in thier home and displays pictogram of the sounds source and find person who live in. However, to prepare in situations that people cannot interact with those interfaces, BADA will bump into him/her to delivery sound information.

BADA can hear 6 sounds source that is selected on the basis of Korean hearing-dog education guide. these are Fire Alarm, Door Knocking, Water Boiling, Water, Baby crying and Door Bell. By utilizing YAMNet which is a machine learning algorithm of sound recognition, this detection system can be constructed.

RTAB Map, one of the state of the arts 3D SLAM package, helps BADA roam around house.

Hardware Architecture

Raspberry pi 3B+, NUC, ReSpeaker Microphone Array, Coral USB accelerator, Intel Realsense D435, T265, Adafruit Dot LED Matrix, Arduino mega

Action Flowchart

Interface

Kakao Talk

Web

Raspberry pi 3B+

install pigpiod

wget https://github.com/joan2937/pigpio/archive/master.zip
unzip master.zip
cd pigpio-master 
make
sudo make install

Nodes

Bada_node

Published Topic

/bada/odom(nav_msgs/Odometry)

Odometry Information of Bada. Pose and velocity are the Position and Velocity of Bada in Global reference frame.

Subscribed Topic

/cmd_vel(geometry_msgs/Twist)

The linear and angular velocity of Bada in Bada's Local reference fream. Twists.linear.x is the forward velocity [m/s], Twists.angular.z is the angular velocity [rad/s].

Parameters

signal_node

Published Topic

/signal

Detected sound signal. type: std_msgs/String

/audio

Realtime sound detection informations. JSON array contains detected sound data with [name, probability] format.

type: std_msgs/String format: {'data': <JSON array>}

usage

var detectionData = JSON.parse(msg.data);

TodoLIST

Final

• refactoring

Robot System Core

• Architecture Design
• Coding
• Debug

Mobile Robot

• Encoder value setting
• Publish Twist message
• PID Control
• Code Refactoring
• Dead Reckoning Information (Odometry / Pose)
• Code Refactoring
• TF message Generation
• Trajectory
• Navigation
• TF Change
• Making roslaunch

Sound Localization

• Measure the direction of sound using TDoA Sensor
• Merge TDoA sensor and mobile robot.
• Create movement toward where sound is ocurred.

Sound Recognition

• YamNet
• Make Yamnet as ROS Program.

SLAM

• Pointcloud using RealSense T265 & D435
• Making SLAM Information
• GMAPPing
• Navigation

WebAPP

• Interaction Design
• Connect WebAPP with ROS
• Visualizing SLAM Information

Human-Robot-Interaction

• What if the person sleeps.
• What if the person is far from their phone.
• How to transfer information about danger situation.