Creating HTML voice widgets for the SEPIA client

Creating HTML voice widgets for the SEPIA client

Introduction

So called view frames have been a tool of the SEPIA cross-platform HTML client for a long time. They are used to display the tutorial and various settings pages by loading content dynamically from HTML files placed inside the client folder.
Since v0.23.0 the cross-platform HTML client (included in SEPIA-Home v2.5.1) now has support for improved custom view frames that fully integrate into the event-queue of the client. That means they have direct access to the microphone, to the client states (recording, talking, loading, etc.) and can manipulate the input text before it is sent to the SEPIA server. In addition the HTML files that contain the frame code can now be loaded from a specific client folder AND from the SEPIA server using a new command available via the Teach-UI called control custom frame.
All this features make custom view frames a powerful tool to create voice widgets and extend the client with unique new abilities. Combined with custom services developers can go even further, building a full-fledged, domain-specific assistant within the existing SEPIA client.

How-to

The best way to get started is to look at the examples that can be found in the SEPIA server folder for custom views: ~/SEPIA/sepia-assist-server/Xtensions/WebContent/views/ and the info/examples for the new control custom frame (frame_control) command inside the Teach-UI (the ? button next to the command).

Download/update example (Linux):

cd ~/SEPIA/sepia-assist-server/Xtensions/WebContent/views
wget -N https://raw.githubusercontent.com/SEPIA-Framework/sepia-extensions/master/client-views/clock.html

Open/use a custom view:

• Define a "control custom frame" (frame_control) command via Teach-UI
• Choose action <on> or <show>
• As URL use <assist_server>/views/path/to/my-view.html (e.g. <assist_server>/views/clock.html or <assist_server>/views/custom/my-view.html)
• Optionally add some "data" that will be sent to the to view during 'onOpen' or 'onFinishSetup' event

Backup

NOTE: Put your own views in the '/views/custom/' folder to ensure they are properly included in a SEPIA backup.