lily

An efficient voice assistant for human beings.

Lily is written in Rust🦀.

Obtaining Lily

Lily uses git LFS which means it needs to be cloned with that installed beforehand.

First install git LFS, for Debian/Ubuntu it is:

sudo apt install git-lfs

On Fedora:

sudo dnf install git-lfs

Then, no matter which OS you are under you need to initialize git LFS:

git lfs install

Now, you can clone the repo:

git clone https://github.com/sheosi/lily

Alternatively, if you already have cloned the repository but did not had lfs installed, on the root folder of this folder do:

git lfs pull

And you'll be good to go.

Building

Dependencies

Dependencies needed at compile time and runtime:

On Debian:

sudo apt install libssl-dev libasound2-dev libpocketsphinx-dev libsphinxbase-dev clang libgsl-dev

On Fedora:

sudo dnf install openssl-devel alsa-lib-devel pocketsphinx-devel clang-libs llvm-devel clang gsl-devel libtool

Optional dependency for feature extra_langs_tts (Languages not provided by Pico Tts for local Tts):

Debian

sudo apt install libespeak-ng-dev

Fedora

sudo dnf install espeak-ng-devel

Note: The first time that you use a language it needs to be downloaded by the NLU, so it needs internet at that time. Also, installing them as system would make this download fail, and you would need to install the languages on your own, for english: snips-nlu download en

Build process

Once you have at least the compile time dependencies you can compile lily, you'll need Rust and cargo (bundled alongside Rust) for this.

cargo build

Docker image

Lily can be made into docker images, one for the default client and another for the assistant itself:

Assistant:

docker build -f Dockerfile -t lily

Client:

docker build -f Dockerfile.client -t lily-client

Features

• Overall shell: some user interfaces, stt, tts ...
• Intent and slot parsing
• Multilanguage
• Client/Server architecture
• Interactivity (asking for something after the initial trigger)
• Question answering
• Semantic parsing

Multilanguage

Lily can be configured to support multiple languages at the same time, this means al of the languages can be used for queries, however, this doesn't mean they can be used in the same phrase (this depends on the ASR model). Part of this is automatic language detection: for voice it exists, but it needs heavy testing, while for text there's some code but te implementation is pending.

Usable components

TTS, STT and NLU are components important enough that having multiple of them makes sense, here's what's supported by Lily right now:

TTS: Pico, Espeak, Google (using optional feature), Larynx, IBM.

STT/ASR: PocketSphinx, Deepspeech (using optional feature), IBM.

NLU: Snips, Rasa (using optional feature, not functional).

Remember that both for TTS and STT/ASR Lily will pair any online service with an offline one as fallback, so that even in case of the connection failing it will continue to work.

Where will it run?

Lily is meant to be run on-device (mostly) even on constrained hardware like a Raspberry. Of course, it will still work on standard PCs and more powerful hardware.

Current state:

• Smartness: Can trigger actions, with a dialog system on the works
• Modularity: Clients connect to the assistant through a custom MQTT protocol, while skills connect via VAP.
• Multilanguage [Needs testing]: Can work with multiple languages and detect by voice which is being used. Detection is only made for voice (not text). Languages can`t be mixed in the same question.