Melodic Discrimination Test (MDT), psychTestR implementation
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
R Researcher email now defaults to NULL Sep 26, 2018
data-raw Adding French to the dictionary Sep 22, 2018
data
man Updated doc Sep 26, 2018
tests Finishing initial implementation Jun 27, 2018
.Rbuildignore Moving item bank to lazy-load data Sep 4, 2018
.gitignore Beginning with code for the PIAT, which we'll convert over to the AMDT Jun 26, 2018
DESCRIPTION Increment version number Sep 26, 2018
LICENSE Update LICENSE Sep 30, 2018
NAMESPACE
NEWS.md
README.md Update README.md Sep 26, 2018
installation-instructions.txt Adding installation instructions Jun 28, 2018
mdt.Rproj

README.md

Melodic Discrimination Test (MDT)

DOI

The MDT is an adaptive test of the ability to discriminate melodies.

We invite you to try the test here and to read the paper here.

Citation

When using the MDT in your own research, you can cite the original MDT research paper:

Harrison, P. M. C., Collins, T., & Müllensiefen, D. (2017). Applying modern psychometric techniques to melodic discrimination testing: Item response theory, computerised adaptive testing, and automatic item generation. Scientific Reports, 7, 1–18. https://doi.org/10.1038/s41598-017-03586-z.

and this implementation:

Harrison, P. M. C., & Müllensiefen, D. (2018). Melodic Discrimination Test (MDT), psychTestR implementation. Zenodo. https://doi.org/10.5281/zenodo.1300950

We also advise mentioning the software versions you used, in particular the versions of the mdt, psychTestR, and psychTestRCAT packages. You can find these version numbers from R by running the following commands:

library(mdt)
library(psychTestR)
library(psychTestRCAT)
if (!require(devtools)) install.packages("devtools")
x <- devtools::session_info()
x$packages[x$packages$package %in% c("mdt", "psychTestR", "psychTestRCAT"), ]

Installation instructions (local use)

  1. If you don't have R installed, install it from here: https://cloud.r-project.org/

  2. Open R.

  3. Install the ‘devtools’ package with the following command:

install.packages('devtools')

  1. Install the melody discrimination test:

devtools::install_github('pmcharrison/mdt')

Usage

Quick demo

You can demo the melodic discrimination test at the R console, as follows:

# Load the mdt package
library(mdt)

# Run a demo test, with feedback as you progress through the test,
# and not saving your data
demo_mdt()

# Run a demo test, skipping the training phase, and only asking 5 questions
demo_mdt(num_items = 5, take_training = FALSE)

Testing a participant

The standalone_mdt() function is designed for real data collection. In particular, the participant doesn't receive feedback during this version.

# Load the mdt package
library(mdt)

# Run the test as if for a participant, using default settings,
# saving data, and with a custom admin password
standalone_mdt(admin_password = "put-your-password-here")

You will need to enter a participant ID for each participant. This will be stored along with their results.

Each time you test a new participant, rerun the standalone_mdt() function, and a new participation session will begin.

You can retrieve your data by starting up a participation session, entering the admin panel using your admin password, and downloading your data. For more details on the psychTestR interface, see http://psychtestr.com/.

The MDT currently supports English (EN), French (FR), and German (DE). You can select one of these languages by passing a language code as an argument to standalone_mdt(), e.g. standalone_mdt(languages = "DE"), or alternatively by passing it as a URL parameter to the test browser, eg. http://127.0.0.1:4412/?language=DE (note that the p_id argument must be empty).

Results

MDT scores are given on an item response theory metric. These scores are similar to z-scores: an average score is about 0, and the typical standard deviation is around 1. See Harrison et al. (2017) for more precise benchmarks.

psychTestR provides several ways of retrieving test results (see http://psychtestr.com/). Most are accessed through the test's admin panel.

  • If you are just interested in the participants' final scores, the easiest solution is usually to download the results in CSV format from the admin panel.
  • If you are interested in trial-by-trial results, you can run the command compile_trial_by_trial_results() from the R console (having loaded the MDT package using library(mdt)). Type ?compile_trial_by_trial_results() for more details.
  • If you want still more detail, you can examine the individual RDS output files using readRDS(). Detailed results are stored as the 'metadata' attribute for the ability field. You can access it something like this:
x <- readRDS("output/results/id=1&p_id=german_test&save_id=1&pilot=false&complete=true.rds")
attr(x$MDT$ability, "metadata")

Installation instructions (Shiny Server)

  1. Complete the installation instructions described under 'Local use'.
  2. If not already installed, install Shiny Server Open Source: https://www.rstudio.com/products/shiny/download-server/
  3. Navigate to the Shiny Server app directory.

cd /srv/shiny-server

  1. Make a folder to contain your new Shiny app. The name of this folder will correspond to the URL.

sudo mkdir mdt

  1. Make a text file in this folder called app.R specifying the R code to run the app.
  • To open the text editor: sudo nano mdt/app.R
  • Write the following in the text file:
library(mdt)
standalone_mdt(admin_password = "put-your-password-here")
  • Save the file (CTRL-O).
  1. Change the permissions of your app directory so that psychTestR can write its temporary files there.

sudo chown -R shiny mdt

where shiny is the username for the Shiny process user (this is the usual default).

  1. Navigate to your new shiny app, with a URL that looks like this: http://my-web-page.org:3838/mdt

Usage notes

  • The MDT runs in your web browser.
  • By default, audio files are hosted online on our servers. The test therefore requires internet connectivity.

Implementation notes

Versions <= 1.3.0 of this package experimented with weighted likelihood ability estimation for item selection. However, current versions of the package revert to Bayes modal ability estimation for item selection, for consistency with the original MDT paper.

Example description for paper

If you use the test in your own research, you could use the following paragraphs as a template for describing the test in your subsequent write-up.

We tested melodic working memory using the melodic discrimination test of Harrison, Collins, and Müllensiefen (2017). This test uses a three-alternative forced-choice (3-AFC) paradigm. In each trial, the participant is played three versions of a new and unfamiliar melody, with successive versions being transposed a semitone higher in pitch. In one of these versions, a note has been altered, making that version the 'odd one out'. The participant's task is to identify which melody was the odd one out. The melodic discrimination task is argued to rely heavily on auditory working memory, as melodies must be held in working memory if they are to be compared and discriminated (Dowling, 1978; Harrison & Mülllensiefen, 2016; Harrison et al., 2017).

We used an implementation of the melodic discrimination test provided by the original researchers and available at https://doi.org/10.5281/zenodo.1300951. Participants complete the test at a computer, using a mouse to select responses. The test uses an adaptive item selection procedure, administering harder items to higher-ability participants and easier items to lower-ability participants. Participant abilities are estimated using Item Response Theory (de Ayala, 2009). We used the test with default settings: length of 20 items, computing intermediate and final abilities with weighted-likelihood estimation, and using Urry's rule for item selection (Magis & Gilles, 2012).

References:

de Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: The Guilford Press.

Dowling, W. J. (1978). Scale and contour: Two components of a theory of memory for melodies. Psychological Review, 85(4), 341–354.

Harrison, P. M. C., Musil, J. J., & Müllensiefen, D. (2016). Modelling melodic discrimination tests: Descriptive and explanatory approaches. Journal of New Music Research, 45(3), 265–280. https://doi.org/10.1080/09298215.2016.1197953

Harrison, P. M. C., Collins, T., & Müllensiefen, D. (2017). Applying modern psychometric techniques to melodic discrimination testing: Item response theory, computerised adaptive testing, and automatic item generation. Scientific Reports, 7, 1–18. https://doi.org/10.1038/s41598-017-03586-z

Magis, D., & Gilles, R. (2012). Random generation of response patterns under computerized adaptive testing with the R package catR. Journal of Statistical Software, 48(8), 1–31.