A listing of publicly available audio samples of a large number of different languages.
The Wide Language Index is an attempt to curate a set of examples of a wide variety of languages from public podcasts. The index itself contains a listing of samples and information about them. The audio files themselves are not bundled in, but can be downloaded as needed.
The full dataset is meant to be downloaded from OS X or Linux. If your system meets the required dependencies, fetching examples of different languages should be as simple as running:
make env && make fetch.
To use the tools that accompany this dataset, your computer should have installed:
make env && make audit to check if the tools are working.
index/ folder contains references to publicly accessible audio samples for which the principal language has been pre-determined. Each sample is represented by a JSON file containing, at minimum:
language: the ISO 693-3 code for the language
media_urls: the URL of the raw sample file
source_name: what venue or outlet published the audio file
title: the title of the podcast or sample
date: the date of publication of the sample
checksum: an md5 checksum of the language
It may also contain the optional fields:
source_url: the URL of the page containing the podcast, for context
description: a description of the contents, in English or in the source language
Each sample's file is named as
Fixing a sample
A helper script exists for fixing samples, but it will only work with access to the underlying S3 bucket.
python src/recode_sample.py index/<language>/<language>-<checksum>.json <new-language>
This index is provided under the Creative Commons Attribution-NonCommercial 4.0 International License. In short, you are free to use this data non-commercially as long as you provide a reference back to this project. If you have a use case where this would be onerous, please send me an email.
Lars Yencken email@example.com