mmir

Overview for the open source mmir (Mobile Multimodal Interaction and Relay) Framework*.

Related repositories

• core:
  • mmir-lib : the core library (this is usually included in the directory mmirf/ of MMIR-based applications)
  • mmir-tooling : scripts and resources for building and generating resources (e.g. compiling JSON grammars)
  • mmir-webpack : extended tooling for integrating mmir-lib in webpack-based apps
• plugins (e.g. for different speech recognition/synthesis engines):
  • mmir-plugins-*: multiple speech input (ASR) and speech oupt (TTS) plugins that use various web services
  • mmir-plugin-speech-android: a Cordova 5.x plugin that integrates Android's default speech recognizer and text-to-speech engines
  • mmir-plugin-speech-nuance: a Cordova 5.x plugin that integrates the Nuance SpeechKit (Dragon Mobile SDK) for speech recognition and text-to-speech
  • see also module overview in the wiki
• tools:
  • scion-queue-plugin: a Cordova 5.x plugin for android platform that provides an event-queue for SCION (used by the dialog manager)
• examples:
  • mmir-cordova: resources / example for creating a minimal MMIR-based application (for Cordova 5.x)
  • mmir-starter-kit: a small MMIR-based example application (for Cordova 5.x)

Documentation

See the wiki for the main documentation.

See API documentation for information about the mmir-lib interface (or the even more detailed API doc with private-visibility).

See the README in mmir-cordova for step-by-step instruction on how to (1) create a new Cordova project and (2) add the MMIR framework.

Older Documenation:

• API mmir-lib v6.x (and private-visibility)
• API mmir-lib v5.x (and private-visibility)
• API mmir-lib v4.x (and private-visibility)
• API mmir-lib v3.x (and private-visibility)

Introduction

The MMIR (Mobile Multimodal Interaction and Relay) framework* aims to provide a lightweight multimodal dialog manager that -- for instance -- can run on mobile devices. The base technology is HTML5 (JavaScript).

A core concept for the framework is the MVC (Model View Controller) pattern. While the framework provides a template mechanism similar to JSP (Java Server Pages), ASP (Active Server Pages), the MMIR framework can also be combined/used with other GUI frameworks, e.g. see Ionic/Angluar starter-kit example.

The state of the dialog system can be tracked and manipulated based on a finite-state machine (FSM). The framework uses SCXML (State Chart XML) via SCION: SCION is a JavaScript-based interpreter for SCXML files.

The framework core and its extensions are implemented as AMD (Asynchronous Module Definition) using RequireJS.

The MMIR library provides basic capabilities for recognizing speech input (ASR, e.g. using Google Speech Recognition service), and speech output (TTS, e.g. using MARY).
For more details of available MMIR Speech Plugins, see the overview on the speech processing wiki page.

NOTE: Requirements for browser-based speech plugins

• web-service-based ASR plugins
  • microphone access: plugins require access to the microphone resource for speech input, i.e. getUserMedia API
  • silence detection: plugins require support of the AudioContext API, for the implementation of the end-of-speech detection (aka silence detection)
• Web Speech-based ASR plugins: plugins that use Web Speech for speech recognition, require support of the Web Speech API
• web-service-base TTS plugins:
  • playback of speech synthesis: plugins require support of the Audio Object API

Prerequisites

For target platforms Android and iOS, Cordova 5.x or later is required (see Cordova documentation on CLI based development for more details).

In addition, the development tools for the targeted platforms need to be installed (see also the Cordova documentation).

Custom Cordova Plugins

If you plan to use the MMIR framework in combination with SCION (that is: make use of the framework's DialogManager and/or InputManager) within an environment that does not support HTML5 WebWorkers, you need the Queue Plugin for extending the framework's SCION
integration with an event queue.

---

*: previously " Mobile Multimodal Interaction and Rendering ", this has been changed since version 6 of the framework, to account for the changed focus in that regard, namely that (graphical) rendering is not a major focus of the framework anymore, but only considered an optional/add-on feature.