Skip to content
Year 3 Group Project in collaboration with IBM
Branch: master
Clone or download
Latest commit ebfaecb Jun 26, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
1-Setup Minor tweaks Jun 26, 2019
2-MachineLearning Add server requirements.txt Jun 26, 2019
3-Pi Add audio_files and audio generation to work with naming differences Jun 26, 2019
4-Hardware Update Jun 26, 2019
5-Administrative Slight formatting changes Jun 26, 2019
6-Miscellaneous Update Jun 26, 2019
imgs Add files via upload Jun 25, 2019
.gitignore Clean up May 28, 2019 tweak overview Jun 26, 2019



People with learning disabilities can encounter difficulty in expressing themselves in speech. Makaton is a language programme that uses signs and symbols to enable users of all ages to communicate effectively. WALDO is a machine learning-enabled assisted living device meant for use by Makaton users and their carers in a care home setting. WALDO’s capabilities include:

  • Makaton sign language interpretation using machine learning
  • Easy-to-use interface, and a cute and endearing package to encourage use by people with learning disabilities and their carers
  • Programmable buttons for users to quickly express pre-set phrases or emotions

Intended Use

WALDO is intended for use in a care home setting in the following ways:

Makaton sign interpretation

  1. Makaton user (care home patient) signs to WALDO
  2. WALDO uses machine learning to interpret sign
  3. WALDO vocalises sign for carer to understand what user intends to express
  4. Carer reacts accordingly, completing the interaction between the user and carer This implementation enables the carer to understand the user without having to undergo the time-consuming process of learning Makaton.

Expression of pre-set phrases/emotions

  1. User (by him/herself, or via carer) sets common phrases that he/she uses frequently to correspond to physical buttons on WALDO
  2. User labels buttons accordingly for ease of use
  3. User presses the relevant button, WALDO vocalises pre-set phrase
  4. Carer responds accordingly This implementation enables people with learning disabilities that have difficulty expressing themselves in speech to easily express everyday sentiments.

Virtual Design History File Overview

This is an overview of the various sections and subsections of WALDO's Virtual Design History File (VDHF). The VDHF details the developmental process of WALDO, the necessary steps to recreate it and the ethical and sustainability considerations of WALDO.

Section Subsection Description
Overview Overview of WALDO, contents of Virtual Design History File
1. Setup 1.1 Jetson Nano Setup Jetson Nano Setup Guide
1.2 Pi Setup Raspberry Pi Setup Guide
1.3 Hardware Setup Diagram of device hardware connections
2. Machine Learning 2.1 Machine Learning Details model design history and evolution as well as implementation on the edge device
2.2 Server Environment Setup Setting up server environment for further model training
Server Training Code Code to be run on server for model training
Jetson Nano Execution Code Code to be run on Jetson Nano for Makaton sign recognition
Model and Weights Saved weights of the final trained model
3. Raspberry Pi 3.1 Overview of Pi Function Outline of functions and connections of Pi
Pi Execution Code Code to be run on Raspberry Pi for high level tasks and audio generations through the IBM Watson text to speech module
4. Hardware 4.1 3D printing files Contains Standard Tessellation Language (STL) files of the Jetson Nano case, the Pi case, the button support structure and WALDO’s eyepiece
4.2 Power Supply Summary of considerations when choosing WALDO’s power supply
4.3 Hardware List List of all hardware components used in assembly of WALDO
5. Administrative 5.1 Record of Meetings Meeting minutes and decisions made
5.2 Gantt Chart Intended timeline of project (plotted during first meeting at IBM Hursley)
5.3 Bill of Materials Record of expenditure and outline of cost of device
5.4 Data Collection Outline of procedure, considerations and product of data collection
6. Miscellaneous 6.1 Ethical Considerations Outline of ethical considerations in the design, building and use of WALDO
6.2 Sustainability considerations Outline of sustainability considerations in the design, building and use of WALDO
6.3 Blog Rationale behind the blog and links to the blog posts
6.4 Future Work Discussion of present implementation and possible extensions to the project
6.5 Leaflet Informational leaflet on WALDO
6.6 Poster Poster for hackbooth and presentation


The WALDO team consists of 5 students from the Electrical Engineering (EE) Department of Imperial College London. The key tasks performed by each member is as follows,

  1. Joshua Chan : Administrative, Data Collection, Peripheral Sensors Implementation
  2. Lua Ying Hao : Jetson Nano Implementation, Machine Learning
  3. Ng Yi Song : Data Collection, Peripheral Sensors Implementation
  4. Patrick John Chia : Jetson Nano Implementation, Machine Learning
  5. Yeow En Kai Joel : Administrative, Data Collection and Processing, Peripheral Sensors Implementation

We would like to thank:

  1. Dr Steve Wright, for his invaluable guidance and encouragement throughout the project
  2. Louise Cooper, John McNamara & Emily Larkin from IBM for their clear direction and unwavering support in the implementation of this project
  3. Precious Homes, for supporting this project by providing valuable insight into the use of Makaton, as well as training videos
  4. Vic, May & Amine at the teaching labs, for their untiring technical support
  5. All who volunteered their time to help us in the collection of training data
You can’t perform that action at this time.