TODO Android

TODO for Android developers

App shell

Create a wrapper app shell that can run our SPA web app.

Options

• Cordova
• PhoneGap
• Others

Tasks

• Create the wrapper app
• Package web app within APK
• Communication between app parts:
  • speech recognition
  • speech generation
  • LLM
  • intent parser
  • notification
  • home screen widget
• File access, including download of AI models

Home screen widget

Create widget that can be placed on the home screen, similar to weather, clock or search field.

Record audio

• Can we record audio from the widget? Whenever the widget is displayed?

Device button

• We need a physical device button that the user can press and activate the voice recognition.
• It needs to work on all phones.
• Solution: Use the "Volume Down" button.

Tasks

• Implement an audio app like Spotify that captures the volume buttons
  • If Volume Down is pressed once, activate the voice recognition, without wake word
  • If Volume Down is pressed twice or more, change system volume, as the buttons would normally do. Ditto for Volume Up.
• Show a notification, so that the app stays active and can react to the volume button, even when the user uses other apps, and even if the phone is in standby.
  • The notification can show the state of the assistant, to make it useful for the user.
• Test that this works on various phones.

LLM

• See section AI | LLM speed
• Run the chosen model on Android

Speech recognition

• Run Mozilla DeepSpeech on Android
• Test speed, must be real time speed. If that's true:
• The speech recognition can be done in real time while the user is speaking, so the result can be ready 100ms-200ms after the voice stopped.

Mitigation

• For very old phones: Online speech recognition

Speech generation

• Find variants of Mozilla's TTS models (various GANs)
  • https://github.com/mozilla/TTS/wiki/Released-Models
  • https://github.com/mozilla/TTS/issues/221
  • https://github.com/erogol/WaveRNN
  • There are more, find them
  • Ask erogol
• MyCroft Mimic2
  • https://github.com/MycroftAI/mimic2
• MaryTTS
  • Very fast, but bad quality
• Test which one works well and is the fastest
• Run it on Android

Mitigation

• Show result visually on display immediately, and delayed per voice once generated
• Use MaryTTS (fast, bad quality)
• Online speech generation