Home > Preparation > Preparation for the P.835]
The following steps should be performed to prepare the P.835 test setup.
Note: make sure to first perform steps listed in the general preparation process.
-
Upload your speech clips in a cloud server and create
rating_clips.csv
file which contains all URLs in a column namedrating_clips
(see rating_clips.csv as an example).Note about file names:
- Later in the analyzes, clip's file name will be used as a unique key and appears in the results.
- In case you have 'conditions' which are represented with more than one clip, you may consider to use the condition's name in the clip's file name or in the URL e.g. xxx_c01_xxxx.wav. Latter you can use regex pattern to extract the condition identifier from the URLs.
Note on Reference Conditions
- It is strongly recommended to include Reference Conditions in your study to cover the entire range of MOS on all three scales. Results of our studies showed that Reference Conditions based on teh ITU-T Rec. P.835 does not cover the entire range of scales, rather the framework propose in ETSI 103 281 Annex D can cover the entire range. We recommend to use 3gpp_p501_FB which is created base on the ETSI/3GPP framework.
-
Upload your training clips in a cloud server and create
training_clips.csv
file which contains all URLs in a column namedtraining_clips
(see training_clips.csv as an example).Hint: Training clips are used for anchoring participants perception, and should represent the entire dataset. They should approximately cover the range from worst to best quality to be expected in the test. It may contain about 5 clips.
-
Upload your gold standard clips in a cloud server and create
gold_clips.csv
file which contains all URLs in a column namedgold_clips
and expected answer to each clip in a column namedgold_clips_ans
(see gold_clips.csv as an example).Hint: Gold standard clips are used as a hidden quality control item in each session. It is expected that their answers are so obvious for all participants that they all give the
gold_clips_ans
rating (+/- 1 deviation is accepted) to the "overall quality" question. It is recommended to use clips with excellent (answer 5) or very bad (answer 1) quality. -
Create trapping stimuli set for your dataset.
-
Configure the
create_trapping_stimuli.py
in your config file. See configuration of create_trapping_stimuli script for more information. An example is provided inconfigurations\trapping_p835.cfg
. -
Delete all files from
trapping clips\source
directory
cd "src\trapping clips\source" del *.*
-
Add some clips from your dataset to
trapping clips\source
directory. Select clips in a way that- Covers fair distributions of speakers (best couple of clips per each speaker)
- Covers entire range of quality (some good, fair and bad ones)
-
Run
create_trapping_stimuli.py
cd src python create_trapping_stimuli.py ^ --cfg your_config_file.cfg
- Trapping clips are stored in
trapping clips\output
directory. List of clips and their correct answer can be found intrapping clips\source\output_report.csv
. You can replace file names (appears in column namedtrapping_clips
) with the URLs pointing to those files to create thetrapping_clips.csv
file (see below).
-
-
Upload your trapping clips in a cloud server and create
trapping_clips.csv
file which contains all URLs in a column namedtrapping_clips
and expected answer to each clip in a column namedtrapping_ans
(see trapping_clips.csv as an example). -
Create your custom project by running the master script:
-
Configure the project in your config file. See master script configuration for more information.
-
Run
master_script.py
with all above-mentioned resources as inputcd src python master_script.py ^ --project YOUR_PROJECT_NAME ^ --method p835 ^ --cfg your_configuration_file.cfg ^ --clips rating_clips.csv ^ --training_clips training_clips.csv ^ --gold_clips gold_clips.csv ^ --trapping_clips trapping_clips.csv
Note: file paths are expected to be relative to the current working directory.
-
Double check the outcome of the script. A folder should be created with YOUR_PROJECT_NAME in current working directory which contains:
YOUR_PROJECT_NAME_p835.html
: Customized HIT app to be used in Amazon Mechanical Turk (AMT).YOUR_PROJECT_NAME_publish_batch.csv
: List of dynamic content to be used during publishing batch in AMT.YOUR_PROJECT_NAME_acr_result_parser.cfg
: Customized configuration file to be used byresult_parser.py
script
-
Now, you are ready for Running the Test on Amazon Mechanical Turk.