Repository navigation
This repository contains the full implementation of our semester project “An Exploration of Measuring Information Rate Across Human Languages” (ETH/UZH, Spring 2025).
Included in this release:
- End-to-end pipeline for computing information density (ID), speech rate (SR), and information rate (IR)
- Support for multiple linguistic units (phones, syllables, words)
- Configurable n-gram models (orders 1–4) with Jelinek–Mercer smoothing
- Within-word and across-sentence modeling options
- Corpus preprocessing (IPA conversion, syllabification, tokenization)
Key findings:
- IR depends on unit choice (phones > syllables > words)
- IR decreases with increasing n-gram order
- Sentence-level models yield systematically higher IR values than within-word models