Subvocalization

• Files for the capture and processing of subvocal data as audio.
• This project is currently the combination of two iterations of the project
  • One of which used an external application (such as Audacity) to record the audio files as single large WAV files for post-processing
    • check_audio.py, process_audio.py, upload_audio.py, flash_new.py
  • The other uses a data stream from an Arduino to capture and split audio in real time
    • main.ino, record_raw_data.py

Iteration 1

flash_new.py

• This script uses a pygame GUI to display words on the screen for the user to vocalize or subvocalize
• The script saves timestamps and words into a CSV file.
• It is also capable of logging keypresses for synchronizing with forearm EMG readings for a gesture-control dataset.
• The vocabulary is contained in the configuration.json file, but also has defaults to a basic set of control words.
• python flash_new.py --csv /path/to/08_25_subvoc/outputs.csv
• Run python flash_new.py -h for more usage help

process_audio.py

• This script takes in two arguments, the CSV generated by flash_new.py, and the dated directory to be processed
• Inside the audio directory should be a subdirectory called original_audio where the files are located
• The script will use the timestamps and labels from the CSV to split and organize the monolithic audio files into smaller ones.
• These split audio files will be put into a subdirectory called split_audio
• Running python process_audio.py /path/to/08_25_subvoc should result in this:

[08_25_subvoc]
|
L__ outputs.csv
L__ original_audio
|   |
|   L__ 01.wav
|   L__ 02.wav
|   ...
|
L__ split_audio
    |
    L__ LABEL_0
    |   |
    |   L__ ch1
    |   |   |
    |   |   L__ 000.wav
    |   |   L__ 001.wav
    |   |   ...
    |   |
    |   L__ ch2
    |   ...
    |
    L__ LABEL_1
    ...

upload_audio.py

• This script will upload the project files to a Google Storage Bucket, as determined by the configuration.json file
• Prefixes all the files with the same date as the parent directory
• python upload_audio.py /path/to/08_25_subvoc
• Stores the split audio as 08_25_audio.zip
• Stores the original audio as 08_25_original_audio.zip
• Stores the CSV file as 08_25_outputs.csv

check_audio.py

• This script samples 10% of each label in a dataset and plays it back
• It allows the users to take note of mislabeled or otherwise corrupt samples
• python check_audio.py /path/to/08_25_subvoc

Iteration 2

main.ino

• This arduino code uses timer interrupts to send a timed data stream
• At timed intervals, it will read from an analog source (such as a microphone or EMG electrode) on A0
• It sends the signal over Serial as a bytestream

record_raw_data.py

• This script takes reads in the bytestream from an Arduino and saves it as split audio (WAV) files
• Work in progress
• The script searches for an Arduino automatically, and will raise an exception if it cannot find one
• Once the Arduino is found, it will read the data into a buffer and save the buffered data into 1-second clips
• The clips are stored in a timestamped audio directory

To-Do

• record_raw_data.py
  • The script currently has no customization options
  • It saves files wherever the script is run. The output directory should be user selectable
  • It truncates the buffer to 1 second of data and saves it. The length should be user selectable
• main.ino
  • The timing may be slightly off due to the imprecise nature of the Arduino crystal clock and prescalers used in the timer1 interrupt. Further research/experimentation is required
• process_audio.py
  • The script should have command line arguments for:
    • Sample Rate
    • Dry run
    • The number of threads to spawn
    • A post-processing configuration file, postprocess.json, that can store operations to run on the dataset
  • The script should be able to read in and parse a post-processing configuration file so the user can define directory-specific post-processing requirements
• check_audio.py
  • The script should generate an output file that stores the names of the files marked as incorrect or corrupt
• upload_audio.py
  • The script should also upload the yet-to-be-implemented postprocess.json as 08_25_postprocess.json

Authors

• Wesley To - Initial work - Subvocalization
• Harrison Smith - Initial work - Subvocalization