Hatschi (cutesy german way of saying "bless you") is an application that automatically tracks respiratory symptoms of the cold, the flu or COVID-19 in particular. It automatically assesses these symptoms through usage of a microphone and gives the user a statistical overview of the amount of symptoms shown throughout the day. That data is furthermore used to give health recommendations to the user, i.e. to stay at home or to get tested for COVID-19.
Try it out here.
Note: The application doesn't actually detect coughs yet (see explanation below). You can say "three" to make it count "coughs" though.
Note: The application doesn't send data anywhere. The code runs locally in your browser and historic data is stored in your browser using IndexedDB. Nobody else will ever get access to the audio stream analyzed here neither see historic data of your health status.
We cough, sneeze and sniff subconsciously throughout the day. Sometimes we don't even know if we're healthy or sick. We might feel healthy but actually we're a walking germ-bomb. This application puts an end to that and tracks your respiratory symptoms throughout the entire day. It's meant to run in the background and go unnoticed. Based on the data it gathers, it's able to tell if you're healthy (having a very low symptom count), kinda unhealthy or if you should go see a doctor immediately. This is especially important in the impeding COVID-19 outbreak but not necessarily limited to it at all. It could be used in a day-to-day office setting as well, where it would prevent workers from being in the office if they are infectuous.
Moreover, we can also qualitatively analyze the type of coughs. A dry-cough is characteristic for COVID-19 for example. We can use our detection to either urge the user to go to see a doctor or to calm them down by stating that the cough pattern is not indicative of COVID-19.
There is quite a bit of scientific research in this area.
- Continuous Sound Collection Using Smartphones and Machine Learning to Measure Cough
- The automatic recognition and counting of cough
- Recognizing bodily noises (ie, not spoken words) [StackExchange Post]
- Accurate and Privacy Preserving Cough Sensing using a Low-Cost Microphone
- MobiCough: Real-Time Cough Detection and Monitoring Using Low-Cost Mob
- A prospective multicentre study testing the diagnostic accuracy of an automated cough sound centred analytic system for the identification of common respiratory disorders in children
- This AI system listens to coughs to learn where the coronavirus is spreading
Based on this research it should be possible to accurately detect different respiratory symptoms. Given the shortage of time (the hackathon had 48 hours of code-time) we went for a Tensorflow.js based approach to make it runnable in the browser and thus available on mobile and laptop/desktop pcs.
We collected sample data from
Google Research and
trained a model to detect coughs. Unfortunately this is where the time
constraint bit us and we had to leave it up to be implemented post-hackathon if
needed. We have a rough idea on how to implement a model now, as can be seen by
the model directory in this repository. The model in there is a very crudely
trained model of the spoken words model used in the application (see below)
though. Training a proper cough model would take longer than the deadline
permits at this point.
Currently, the implementation uses a Tensorflow.js sample model, that is able to detect spoken words. The word "three" is used as a synonym of a cough in the current implementation.
We started at 0, with a team of 8. We divided the team into a visualization and a model team. The former would design and implement the application, get the statistics and visualize it for the the user. It also implemented any decision making logic. The model team was responsible for collecting training data and producing a model to be used in TensorflowJS.
Working in parallel has greatly improved the throughput of our work and we've been able to make great progress on both fronts, as can be seen in the example above. We underestimated the interface between a generic Tensorflow model and TensorflowJS though which at the end was a dealbreaker for not being able to import the trained model into the application.
As mentioned above, we didn't succeed in training the cough model though in theory, given more time, that should absolutely be possible. Relevant resources can be found here and here. These examples need to be adjusted to not work on the list of words they are based on now, but to be trained to detect "cough". This repository does not contain the cough sample set we created due to potential copyright infringement. This repository started a collection on covid/non-covid coughs to train models for example.
The model could not only detect cough, but also which kind of cough (i.e. dry-cough) or other respiratory symptoms like sneezes or rattling. All of that data can greatly improve the recommendation engine to give deeper insight and more nuanced guidance.
The recommendation engine is currently very crude and only take a few values into account. You might find the recommendations it gives very strange sometimes, especially when using a randomized data set. This should be improved upon using more insight from actual medical professionals and respective statistical handling of the data at hand.
As a lot of symptom-gathering apps are built these days, this application could of course be used as well to feed a centralized data store to determine hot-spots of flu-like outbreaks and even COVID-19 spread. It could even help to narrow down the number of unreported cases and give a clearer picture on how many people are actually likely infected.
This app has been created as part of the "WirVsVirus" Hackathon held by the federal government of Germany to fight the corona virus disease.