Skip to content

XML specification

Jump to bottom
Brecht De Man edited this page Nov 17, 2020 · 21 revisions

Each test is defined by its XML configuration file. Examples of these can be seen in the ./tests/examples/ directory. This file can be generated by the test creator (./test_create.html), modified from an existing file, or written from scratch.

The test uses XML validation, so the ordering of nodes is important to pass this validation. Some nodes also have specific attributes which must be set and may even have a certain format to apply them. This is done so error checking can be performed to catch easy-to-find errors before loading and running a test session. If your project XML fails this validation, all the errors will be listed.

This section will walk you through the available nodes, their function and their attributes.

The general structure (with number of occurrences of each node) is as follows:

<?xml version="1.0" encoding="utf-8"?>
<waet xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="test-schema.xsd">
    <setup>
        <exitText><\exitText>
        <survey></survey>                     (0-2)
        <metric></metric>
        <interface></interface>
    </setup>
    <page>                                    (1 or more)
        <title></title>                       
        <commentboxprefix></commentboxprefix>
        <interface></interface>               (1 or more)
        <audioelement />                      (1 or more)
        <commentquestions></commentquestions>
        <survey></survey>                     (0-2)
    </page>
</waet>

Each element occurs 0 or 1 times, unless otherwise specified.

Table of contents

Root (<waet>)

The root node is <waet>. It must have the following attributes: xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="test-schema.xsd". This will ensure it is checked against the XML schema for validation.

Setup (<setup>)

The first child node, <setup>, specifies any one-off and global parameters. It takes the following attributes:

  • interface: string, mandatory, specifies the interface to load (e.g. interface="APE", see Interfaces).
  • projectReturn: URL, mandatory, specifies the return point. Can be a 3rd party server or the local server. Set to "null" to disable automatic saving. Specifying "save.php" will trigger the return if either the PHP or Python servers are used. On error, it will always default to presenting the save on page.
  • returnURL: Upon successful completion and submission of the test, this URL will be opened. This can be a presentation of the results thus far, some type of reward, or a page with links to other tests.
  • randomiseOrder: Boolean, optional, if true it will randomise the order of the test pages. Default is false.
  • poolSize: non-negative integer, optional. Specifies the number of test pages to actually include in test. Combined with randomiseOrder being true this will give a random set of test pages per participant from the given pool of nodes. Specifying 0 disables this option, default is 0.
  • loudness: non-positive integer, optional. Set the default LUFS target value.
  • sampleRate: positive integer, optional. If set, the sample rate reported by the Web Audio API must match this number. See Sample rate.
  • calibration: boolean, optional. If true, a simple hearing test is presented to user to gather system frequency response (DAC, listening device and subject hearing). Default is false.
  • crossFade: decimal greater than or equal to 0.0, optional. Define the crossFade between fragments when clicked, in seconds. Default is 0.0s.
  • preSilence: decimal greater than or equal to 0.0, optional. Add a portion of silence to all elements in the test at the beginning of the buffer, in seconds. Default is 0.0s.
  • postSilence: decimal greater than or equal to 0.0, optional. Add a portion of silence to all elements in the test at the end of the buffer, in seconds. Default is 0.0s.
  • playOne: boolean, optional. Only allow playback of one fragment at a time in its entirety. Will stop synchronous from working.

The <setup> node takes the following child nodes, note these must appear in this order:

Exit text (<exitText>)

This tag simply contains the text which is shown to indicate successful submission at the very end of the test. If the tag is not present, the default exit message ('Thank you!') is shown.

Survey (<survey>)

A survey can appear at the very start and/or end of the test, as well as at the start and/or end of any page (in case the <page> node has a <survey> child). It has only one attribute, location, which can be set to "before" or "after". There are several types of node which you can use:

  • <surveystatement> - Display a statement to the participant
  • <surveyquestion> - Ask a question and receive the response as text
  • <surveyradio> - Give the user several options and ask them to pick only one
  • <surveycheckbox> - Give the user several options, the user can select multiple
  • <surveynumber> - Ask a question and with a number box
  • <surveyslider> - Display a number slider
  • <surveyvideo> - Display a video player
  • <surveyyoutube> - Link to an embedded youtube URL player

Each node has the following attributes:

  • id: ID, mandatory. Must be unique across the entire XML, used to identify the response in the results.

  • mandatory: Boolean, optional. Defines if the survey must have a response or not. Does not apply to statements. Default is false.

  • minWait: Number, optional. Prevent progressing until the number of seconds defined has elapsed. Default is zero seconds.

  • <surveyquestion>:

    • boxsize: String, optional. Only applies when type="question" and must be one of the following: normal (default), small, large or huge.

  • <surveycheckbox>:

    • min: Number, optional. The minimum number to select before progressing. Default is 0
    • max: Number, optional. The maximum number that can be selected to progress. Default is length of options.
    • randomise: Boolean, optional. If true, randomise the order of the options in the list. Default is false

  • <surveyradio>:

    • randomise: Boolean, optional. If true, randomise the order of the options in the list. Default is false

  • <surveynumber>, <surveyslider>:

    • min: Number, optional. The minimum valid response.
    • max: Number, optional. The maximum valid response.

  • <surveyvideo>, <surveyyoutube>:

    • url: String, mandatory: The URL of the video resource.

The nodes have the following children, which vary depending on the survey type wanted.

  • <statement>: Must appear only once. Its text content specifies the text to appear as the statement or question for the user to respond to.
  • <option>: Only valid if the parent node is surveycheckbox or surveyradio. Has attribute name to identify the selected option in the results. The text content is the text to show next to the radio/checkbox.

Example survey node in configuration file

<survey location="before">
    <surveyquestion id="sessionId" mandatory="true">
        <statement>Please enter your name.</statement>
        <conditional check="equals" value="John" jumpToOnPass="test-intro" jumpToOnFail="checkboxtest" />
    </surveyquestion>
    <surveycheckbox id="checkboxtest" mandatory="true" min="2" max="4" randomise="true">
        <statement>Please select with which activities you have any experience (example checkbox question)</statement>
        <option name="musician">Playing a musical instrument</option>
        <option name="soundengineer">Recording or mixing audio</option>
        <option name="developer">Developing audio software</option>
        <option name="hwdesigner">Designing or building audio hardware</option>
        <option name="researcher">Research in the field of audio</option>
    </surveycheckbox>
    <surveyquestion id="instrument" mandatory="false">
        <statement>What instrument did you play</statement>
    </surveyquestion>
    <surveystatement id="test-intro">
        <statement>This is an example of an 'APE'-style test, with two pages, using the test stimuli in 'example_eval/'.</statement>
    </surveystatement>
</survey>

Metric (<metric>)

The metric node contains the metrics to be tracked during the complete test. Their names can be found back in the saved results files, where the various logged measurements can be found in <metricresult> tags.

When in doubt about which metrics to include, err on the inclusive side, as one never knows which information is needed in the future. Most of these metrics are necessary for diagnostics and analysis scripts such as timeline_view_movement.py.

The following metrics are supported:

  • testTimer: One per test page. Presents the total test time from the first playback on the test page to the submission of the test page (excluding test time of the pre-/post- test surveys). This is presented in the results as <metricresult id="testTime"> 8.60299319727892 </metricresult>. Time in seconds.
  • elementTimer: One per audio fragment per test page. This totals up the entire time the audio fragment has been listened to in this test and presented <metricresult name="enableElementTimer"> 1.0042630385487428 </metricresult>. Time in seconds.
  • elementInitialPosition: One per audio fragment per test page. Tracks the initial position of the sliders, especially relevant when these are randomised. Example result <metricresult name="elementInitialPosition"> 0.8395522388059702 </metricresult>. Value from 0.0 (start of scale) to 1.0 (end of scale).
  • elementTracker: One per audio fragment per test page. Tracks the movement of each interface object. Each movement event has the time it occured at and the new value. Example: <metricresult name="elementTrackerFull"><movement time="0" value="0.5895316804407713"/><movement time="9.967165532879818" value="0.5917355371900826"/></metricresult>.
  • elementFlagListenedTo: One per audio fragment per test page. Boolean response, set to true if listened to: <metricresult name="elementFlagListenedTo">true</metricresult>.
  • elementFlagMoved: One per audio fragment per test page. Binary check whether or not a the marker corresponding with a particular fragment was moved at all throughout the experiment.
  • elementListenTracker: One per audio fragment per test page. Tracks the playback events of each audio element pairing both the time in the test when playback started and when it stopped, and it also gives the buffertime positions:
<metricresult name="elementListenTracker">
    <event>
        <testtime start="3.7093877551020284" stop="4.963265306122437"/>
        <buffertime start="3.7093877551020284" stop="4.963265306122437"/>
    </event>
    <event>
        <testtime start="11.708662131519276" stop="14.315102040816328"/>
        <buffertime start="11.708662131519276" stop="14.315102040816328"/>
    </event>
</metricresult>

Example metric node in configuration file

<metric>
    <metricEnable>testTimer</metricEnable>
    <metricEnable>elementTimer</metricEnable>
    <metricEnable>elementInitialPosition</metricEnable>
    <metricEnable>elementTracker</metricEnable>
    <metricEnable>elementFlagListenedTo</metricEnable>
    <metricEnable>elementFlagMoved</metricEnable>
    <metricEnable>elementListenTracker</metricEnable>
</metric>

Interface (global) (<interface>)

The interface child node of the setup node specifies test-wide options (as opposed to page-wide, see Interface (page)) related to the displayed elements and enforced behaviour.

Specifically, optional show items are

  • playhead: Shows the playhead (also 'scrubber bar'), indicating where in the file the participant is currently listening.
  • page-count: Shows the current test page number and the total number of test pages, to show the participant's progress.
  • volume: Shows a master volume control to the user to manipulate the output gain of the page. Volume slider movements are always logged in the results file.

and check items are

  • fragmentPlayed: Checks that all fragments have been at least partially played before the participant can submit and go to the next page.
  • fragmentFullPlayback: Checks that all fragments have been fully played. (NOTE: This will always clear if the page is looping as it is not possible to know every sample has been played.)
  • fragmentMoved: Checks that all fragments have been moved. In AB(X) interfaces this will always clear as there is no movement.
  • fragmentComments: Cheks that all fragments have a comment. Will clear if there are no on page comments but with a console warning.
  • scalerange: Has two extra attributes “min” and “max”. Checks that at least one element is below the min value and at least one element is above the max value.

Example interface node in configuration file

<interface>
    <interfaceoption type="check" name="fragmentMoved"/>
    <interfaceoption type="check" name="fragmentPlayed"/>
    <interfaceoption type="check" name="scalerange" min="25" max="75"/>
    <interfaceoption type="show" name='playhead'/>
    <interfaceoption type="show" name="page-count"/>
    <interfaceoption type="show" name="comments"/>
</interface>

Page (<page>)

The only other type of first level child nodes, <page>, tags specify the test pages. It takes the following attributes:

  • id: ID, mandatory. A string which must be unique across the entire XML. It is used to identify the page on test completion as pages are returned in the results in the order they appeared, not specified.
  • hostURL: URL, mandatory. Used in conjuction with the <audioelement> url to specify where the audio files are located. For instance if all your files are in the directory ./test/ you can set this attribute to "/test/" and the <audioelement> url attribute only needs to file name. Set to "" if no hostURL prefix desired.
  • randomiseOrder: Boolean, optional. If true the audio fragments are presented randomly rather than the order specified. Default is false.
  • repeatCount: non-negative integer, optional. Specify the number of times to repeat the test page (re-present). Each presentation will appear as an individual page in the results. For a test page to be presented n times, repeat count should be n-1. Default is 0.
  • loop: Boolean, optional. If true, the audio elements will loop to the beginning when reaching the end of the fragment. Default is false.
  • synchronous: Boolean, optional. If true, the audio elements will play synchronously with each other. This means that if sample A is at 5.1 seconds when sample B is clicked, sample B will start playing immediately from 5.1 seconds as well. If the samples are similar, this results in seamless crossover between the different versions. Stopped from working if playOne is true. Default is false.
  • loudness: non-positive integer, optional. Set the LUFS target value for this page. Supersedes the <setup> loudness attribute for this page.
  • label: enumeration, optional. Set the label to one of the following
    • default: The default by the interface (Default if undefined)
    • none: Show no labels
    • number: Show natural numbers starting at index 1
    • letter: Show letters starting at 'a'
    • capital: Show letters starting at 'A'
    • samediff: Show 'same' for first sample and 'different' for second sample, all other samples will have no name - only avaliable on AB tests
  • poolSize: non-negative integer, optional. Determine the number of <audioelement> nodes to take from those defined. For instance if poolSize="3" and there are 4 audio elements, only 3 will actually be loaded and presented to the user.
  • alwaysInclude: boolean, optional. If the parent <setup> node has poolSize set, you can enforce the page to always be selected by setting alwaysInclude to true. Default is false.
  • preSilence: decimal greater than or equal to 0.0, optional. Add a portion of silence to all elements in the page at the beginning of the buffer in seconds. Supercedes any value set in <setup>. Default is 0.0s.
  • postSilence: decimal greater than or equal to 0.0, optional. Add a portion of silence to all elements in the test at the end of the buffer in seconds. Supercedes any value set in <setup>. Default is 0.0s.
  • position: integer greater than or equal to 0, optional. Force the page to be placed at a specific position, even after randomisation. Pages with position set will also have alwaysInclude="true". Default is null. (See MWE on issue tracker)

The <page> node takes the following child nodes (note that these must appear in this order):

  • <title>: Appear once or not at all.
  • <commentboxprefix>: Appear once or not at all.
  • <interface>: Minimum of 1.
  • <audioelement>: Minimum of 1.
  • <commentquestions>: Appear once or not at all.
  • <survey>: Minimum of 0, maximum of 2 occurrences (i.e before and/or after).

Title (<title>)

The text content of this node specifies the title of the test page, for instance <title>John's Audio Quality Test</title>

Comment box prefix (<commentboxprefix>)

When comments are enabled (i.e. the global or page-specific <interface> node contains <interfaceoption type="show" name="comments"/>), there is a comment box below the main interface for every fragment on the page.

The text content of the optional <commentboxprefix> tag specifies the prefix of the comment boxes associated with the different audio elements. By default, i.e. in absence of this tag, this text will read "Comment on fragment" followed by the fragment identifier.

For instance, <commentboxprefix>Please describe fragment </commentboxprefix> will lead to comment boxes with text "Please describe fragment 1", "Please describe fragment 2", and so on.

Interface (page) (<interface>)

The interface node (when child of a <page>, see Interface (global) above for the <setup> interface node!) specifies any interface specific options and test parameters. It has an optional name attribute used to set the various axis names (where applicable), such as the multi-axis APE interface. Specifying multiple interface nodes in a <page> node will introduces multiple axes where applicable, otherwise only the first <interface> node will be used and the rest ignored. Use this name attribute to retrieve the rating values, comments and metrics associated with the specified interface. If no names are given, you can still use the automatically given interface-id, which is the interface index starting at 0 and corresponding to the order in which the rating scales appear.

The node has the following children, note the order these must appear in is as follows:

  • title: Optional, maximum 1 occurrence. The text content specifies the name of the axis as shown to the test participants.
  • image: Optional, maximum 1 occurrence. Show a static, on-page image to associate with the test page / axis. The test must be a URL to the image file.
  • interfaceoption: Optional, any number. Specifies the interface options as specified in Interface (global), which will then be applied to this page only.
  • scales: Optional, maximum 1 occurence. Contains <scalelabel> nodes which define the displayed scales.

Audio Element (<audioelement>)

Appear as children of the page node. Each of these specify an individual interface fragment to display. Multiple fragments can reference the same file (allowing for repetition with different parameters, or blind doubles). The node has the following attributes:

  • id: ID, mandatory. Must be unique across the test page. Used to identify the specific fragment in the results.
  • name: String, optional. If you wish to group fragments across pages when performing result analysis, set the group name here. For instance, when using the same sample in different pages, or a sample with the same origin, you cannot assign the same id, but you can give it the same name.
  • url: URL, mandatory. Used with the parent page nodes' hostURL attribute to get the full url of the audio file to load.
  • label: string, optional. Use this if you want to specify how this sample is labelled, e.g. when it is a reference you can label the reference button 'Example' or 'Click to hear the original'.
  • gain: float, optional. Specify the gain in decibels to apply to the node after loudness normalisation. Default is 0.
  • type: string, optional. Must be one of the following:
    • normal (default when not specified)
    • anchor
    • reference
    • outside-reference (presents the node as a separate interface option)
  • marker: integer between 0 and 100, optional. Only used when type="anchor" or type="reference".
  • loudness: Set the loudness of this element in LUFS. Supersedes all other set values.
  • alwaysInclude: boolean, optional. If the parent <page> node has poolSize set, you can enforce the element to always be selected by setting alwaysInclude to true. Default is false.
  • preSilence: float greater than or equal to 0.0, optional. Add a portion of silence to all elements in the page at the beginning of the buffer in seconds. Supercedes any other value. Default is 0.0s.
  • postSilence: float greater than or equal to 0.0, optional. Add a portion of silence to all elements in the test at the end of the buffer in seconds. Supersedes any other value. Default is 0.0s.
  • image: url string, optional. An image to display when the fragment is playing.

Comment questions (<commentquestions>)

As opposed to the audioelement-specific comments (see Comment box prefix), these are containers for commentquestion, commentradio, and commentcheckbox elements, and relate to the complete set of stimuli of a certain page.

For instance, a possible commentquestions element could be

<commentquestions>
    <commentquestion id='difficulty'>
        <statement>Did you find this task challenging?</statement>
    </commentquestion>
    <commentradio id="preference">
        <statement>Please enter your overall preference</statement>
        <option name="worst">Very Bad</option>
        <option name="bad"></option>
        <option name="OK">OK</option>
        <option name="Good"></option>
        <option name="Great">Great</option>
    </commentradio>
    <commentcheckbox id="character">
        <statement>Please describe the overall character</statement>
        <option name="funky">Funky</option>
        <option name="mellow">Mellow</option>
        <option name="laidback">Laid back</option>
        <option name="heavy">Heavy</option>
    </commentcheckbox>
</commentquestions>

Survey (page) (<survey>)

See the Survey section above.

Anything unclear or inaccurate? Please let us know at mail@brechtdeman.com

Overview

Quick start

Examples

Features

Test Design

Interfaces

XML specification

Test_create tool

Running a test

Local test

Remote test

Pooling tests

Sample rate

Other

Analysis and diagnostics

Troubleshooting

Clone this wiki locally