ARIA Framework Tutorial

Jelte edited this page Mar 30, 2017 · 4 revisions

Running the default scenario

Set up Dialogue Manager

Go to the folder of the dialogue manager, which is located at ..\ARIA\Agent-Core-New\. There are two configuration folders in the DM folder, config and configStandalone. The former is for configuring the dialogue manager in the integrated system, whereas the configStandalone is for configuring the dialogue manager without SSI or a behavior realizer, mostly useful for demo purposes.

In the config folder, open the managers.xml file and make sure that all managers are uncommented (e.g Intent Manager, Turn Manager, etc). To run the simple question answering, you have to make sure the templates loaded in the IntentManager’s templates are: info-rules.xml, opening-rules.xml and closing-rules.xml. In the BehaviourManager’s templates, make sure to activate info-responses.xml, opening-responses.xml and closing-responses.xml. By default this should be the case. In this particular scenario you can greet Alice, ask about the proposed three characters and say goodbye. Be sure to only include a single noun when using anaphora. See in the 'Create your own scenario' section below the reference to the example dialogue that can be held with this scenario.

If you want to run the more extensive (not yet bug-free) QA matcher, be sure to make the lines active of opening-rules.xml, closing-rules.xml and question-intent.xml. Also, adjust the Behaviour Manager’s templates to include opening-responses.xml, closing-responses.xml and question-response.xml. Here you can ask questions which contain specific keywords (see QAPairs.xlsx in the tutorial section 'Create your own scenario'), for which Alice knows a response. These have not been fully tested and might contain some weird sentences.

After you have configured the managers.xml, you can go to the root folder and start the System according to the instructions in the README.md. You can always kill each component separately with the launcher, the Agent Input, the Agent Core or the Agent Output. For the Dialogue Manager, you can kill the Agent Core in between, update your managers.xml or templates if need be and restart it via the launcher (or the run.bat in the Agent Core folder, which gives more explanation or possible syntax errors).

Using NOVA to visualise and analyse an interaction

The ARIA-Framework can save all input audio and video, as well as all SSI output. You can use the NOVA visualisation and annotation tool to view all these information streams to help analyse the system, debug it, and potentially create new annotations to retrain particular modules.

To use this, you must first set-up the ARIA-Framework to record all system information. You do this by [ALEX: explain].

Next, you must download and install NOVA, which is developed by ARIA-VALUSPA but as a separate tool. To load an ARIA-Interaction, go the ARIA\System\Agent-Input\ssi\pipes\all-in-one\log folder and select a session (each session is stored under a unique timestamp, e.g. 2016-12-05_12-45-06). Within the folder you will find a file 'project.nova'. Simply drag'n drop this file to NOVA.

Creating your own scenario

This tutorial will teach you some of the basics of Flipper for the ARIA-VALUSPA project using example templates (please find them in the accompanying Examples.zip). The setup is as follows: for each example we specify some simple rules for the system, we provide the Flipper templates for the example, and we give you a task (optional) to change the template in order to practise. A general introduction to and manual for Flipper can be found here. The tutorial folder contains next to the tutorial files, also documents for:

  • This tutorial (FlipperTutorial.docx)
  • The possible information states in Flipper (Information States.xlsx)
  • The possible tags for emotion in the behaviour class (FMLParameters.docx)
  • The QAPairs used for the more 'complex' QA-matching in Flipper (QAPairs.xlsx)
  • Example of dialogue with the simple QA conversation (Example Dialogue SimpleQA.docx)

With each example we offer our solution in a zip-file. Please extract this zip to: “..\ARIA\System\Agent-Core-New\templates\Examples\” In “..\ARIA\System\Agent-Core-New\config\manager.xml” edit the BehaviourGenerator manager to include the appropriate templates for your example.

For example, set: <template path="../templates/Example/tutorial1.xml"/>.

Example 1: Simple Q&A

This first example shows very basic question-answer behaviour. Flipper is intended to switch (flip) information between modules based on abstract rules. These rules can be based on input the system receives and/or on information state updates the system generates autonomously. Flipper is not intended for use as a QA matcher, but it can be used in this manner. This example shows how, based on some user input, appropriate FML/BML output can be created for response behaviour generation and how this behaviour can depend on the dialogue history.

We specify: greeting and valediction by the agent

A user says ‘Hi’ (or some synonym), we want the agent to say ‘Hello to you, $user_name’. A user says ‘Bye’ (or some synonym), we want the agent to say ‘Bye’ (or some synonym). After the initial greeting, the agent does not respond to ‘Hi’ until the conversation is closed by the user saying ‘Bye’. Examine the accompanying Flipper templates to see our solution, which is tutorial1.xml. Be sure that in the managers.xml file, the template path to tutorial1.xml is uncommented.

Task: change the behaviour of the agent by adding the following specification:

A user says ‘weather’ (or some synonym), we want the agent to say ‘I’m a computer, as long as I’m inside I don’t care about the weather’. Hint: the synonym list can be found here: “..\ARIA\System\Agent-Core-New\data”. The words on each line are all ‘translated’ to the first word on that line.

Example 2: Primitive counselor

Taking output from the SSI to make a compassionate remark can be enough to have a very primitive counselor, as we show in this example. We look at the valence value from the SSI framework (it is based on both visual and auditory information of the user). Remember to adjust the managers.xml to only include the behaviour template of tutorial2.xml.

We specify: Response to non-verbal cues

When the user looks happy (valence > 0.5), the agent says ‘You look happy! Why are you happy?’ When the user looks unhappy (valence < 0.3), the agent says ‘You seem sad. What happened?’. The agent does not say the same thing twice (i.e. the user has to change valence before the agent makes another remark). Examine the accompanying Flipper templates to see our solution, tutorial2.xml.

Task: change the template so that the agent responds after each user utterance with an encouraging remark:

When a user says something, after the agent made a remark about the user’s state, the agent responds with ‘Really? Tell me more!’. Hint: look at Example 1.

Example 3: Funny or not

In this example we show some logic (e.g. random) and how to use SSI input to select agent behaviour. We want the agent to tell jokes and show that it understands whether or not the user thought the joke was funny.

We specify: induced non-verbal response of the user

The agent takes the initiative. The agent selects a random joke (from three options) to crack. The agent tells the joke and waits some time before delivering the punch line from that joke. Based on the SSI input (valence) we want to user to respond differently: A user is smiling after the joke (happiness >= 0.8), the agent says ‘Good joke, made you laugh!’ A user is not smiling after the joke (happiness < 0.8), the agent says ‘Sorry, not funny?’ Examine the accompanying Flipper templates to see our solution, tutorial3.xml.

Task: change the behaviour of the agent by adding the following specification:

If user is present according to SSI, the agent tells jokes. Otherwise the agent does nothing. Hint: find the appropriate Information State in the Information States.xlsx (found in the tutorials folder)

Custom scenario

If you want to write your own templates, it's good to use dialogue acts for structuring the dialogue within Flipper in the preconditions of the agent behaviour. For the ARIA-framework, we take the taxonomy from DIT++, found on https://dit.uvt.nl. The framework helps you to determine which dialogue acts to recognize and how to respond to them.

Using the SSI output for other projects

The following files are created during a session:

  • audio.wav: A PCM audio stream with a sample rate of 16kHz
  • video.avi: A AVI video file with a frame rate of 25 fps
  • voiceactivity.stream: A SSI stream file with the calculated voice activity (1 dimension, 33.3 Hz)
  • arousal.stream: A SSI stream file with the estimated arousal level (1 dimension, 25 Hz)
  • valence.stream: A SSI stream file with the estimated valence level (1 dimension, 25 Hz)
  • interest.stream: A SSI stream file with the estimated interest level (1 dimension, 25 Hz)
  • emax.stream: A SSI stream file with the output of emax (13 dimension, 25 Hz)
  • transcription.annotation: A transcription with the results of the ASR
  • agent-fml.annotation: A log file with the FML messages sent to the agents
  • agent-bml.annotation: A log file with the BML messages sent to the agents
  • dialog.annnotation: A log file storing the dialog acts generated by the DM

A SSI stream is stored in two files and the one ending on ~ contains the stream data in ASCII format. Each line stores a sample and sample values are separated by blank. Likewise, in the annotation ~ file each line stores a segment defined by a start and end point (in seconds) and a label name.

Extracting Video Features using eMax