Eliza implementation using the combot framework.
This application also serves as a blue-print for applications in the combot framework.
Follow the install and run instructions in the Eliza parent.
After starting the application, wait until all services and the backend server are started. Until that point is reached, there will be warnings about failed connections to the backend in the logs, which can be ignored. This can take a while, especially at the first run of the application, as models for the ASR module may need to be downloaded.
Once the application is fully started, it will greet the user. You can interact with the application either by speech or using the chat UI, which is available at http://localhost:8000/chatui/static/chat.html. Currently, when running locally, output will be only to stdout or the Chat UI, as there is no text to speech implementation in place.
The application is composed of the following components:
The Backend Server is a standalone application that provides a REST API with the basic raw signals.
The Backend Container connects to the backend server to retrieve raw signals and processes them to make them available in the application. This involves:
- Managing access to the signal resources (e.g. mute mic while speaking).
- Store raw signal data.
- Publish events for the incoming signals, pointing to the storage location where the raw data can be retrieved.
- Subscribe to events that result in outgoing signals and send them to the backend server.
Subscribes to audio signal events and detects voice activity in the audio data. For detected voice activity, an event with the respective Mention on the audio signal is published.
Subscribes to voice activity events and transcribes audio data referenced in the voice activity annotation to text. For the transcribed text, a text signal event is published, referencing the respective segment in the AudioSignal.
Subscribes to text signals where the robot is not the speaker, processes the text and publishes a new text signal with a response.
Subscribes to text signals where the robot is the speaker, and converts the text to an audio signal.
Subscribes to text signals and publishes text signals from user input.
The event payploads used to communicate between the individual modules follow the EMISSOR framework. To be continued..
The application is setup for multiple runtime systems where it can be executed.
The simplest is a local Python installation. This uses Python packages built for each module of the application and has a main application script that configures and starts the modules from Python code.
The advantage is that communication between the modules can happen directly within the application, without the need to setup external infrastructure components, as e.g. a messaging bus. Also, debugging of the application can be easier, as everything run in a single process.
The disadvantage is, that this limits the application to first, use modules that are written in Python, and second, all modules must have compatible requirements (Python version, package versions, hardware requirements, etc.). As much as the latter is desirable, it is not always possible to fulfill.
The local Python application is setup in the py-app/ folder and has the following structure:
py-app
├── app.py
├── requirements.txt
├── config
│ ├── default.config
│ └── logging.config
└── storage
├── audio
└── video
The entry point of the application is the app.py script and from the py-app/ directory after running make build
from the eliza-parent it can be run via
source venv/bin/activate
python app.py
The Python application provides the Chat UI at http://localhost:8000/chatui/static/chat.html.
Alternatively to the local application, modules can be run in a containerized runtime system. In this setup each (or a subset of) module(s) used in the application runs in a separate containerized runtime systems that provides the requirements needed by the module. This provides full separation between the modules in terms of software and hardware requirements, as well as isolation of their runtime state. In exchange this requires additional infrastructure to enable the communication between the modules, as for instance a messaging bus, resource management, container orchestration etc. In exchange this can be harder to setup and to debug, as each module and the communication between them has to be inspected separately.
Container management can be done different tools. The next two sections describe two such setups. Also see the Docker documentation.
Docker provides Docker compose and Docker swarm as a tool to orchestrate applications with multiple containers.
A widely used tool to run containerized applications is Kubernetes.
For the development workflow see the cltl-combot project.