-
Notifications
You must be signed in to change notification settings - Fork 1
3) Research Documentation
Recognizing Uncertainty in Speech
- published in 2011 (57 citations)
The study investigates how prosody (intonation, stress, and speech rate) differs when a speaker is confident versus uncertain, and whether these cues appear across the whole utterance or are localized around critical/target words.
-
Participants / utterances
- 20 speakers: all native English
- 5 raters
- 600 utterances total
-
Scripts
- Boston public transportation fill-in-the-blank sentences
- Lexical fill-in-the-blank sentences (with unfamiliar words inserted)
-
Procedure
- Speakers recorded their self-rated confidence (1–5)
- Raters judged perceived confidence (1–5)
-
Features Used
-
Pitch (f0)
- minimum, maximum, mean, std, range
- relative position in utterance of min pitch
- relative position in utterance of max pitch
- absolute slope
-
Intensity (RMS)
- minimum, maximum, mean, std
- relative position in utterance of min intensity
- relative position in utterance of max intensity
-
Temporal
- total silence, percent silence
- total duration
- speaking duration (utterance length − pauses)
- speaking rate
-
-
Normalization
-
z-score normalization per speaker
- For each speaker, all utterances were used to compute ((x - \mu)/\sigma)
- Each speaker’s mean centered to 0 and std to 1
- This cancels baseline pitch/loudness/speaking style differences
⚠️ Problem: What happens in a service’s early stage, when insufficient per-speaker data is available for reliable normalization? -
-
Total duration (−0.653)- Sentence length was nearly fixed in this setup; not relevant for our service
-
Total silence (−0.644)
-
Speaking duration (−0.515)- Same issue as above; discarded
-
Percent silence (−0.459)
-
Absolute slope of f0 (+0.312)
- Larger values correspond to final rising intonation (questions) or sharp final falls (statements)
- Likely: final falls (r < 0) = confidence
- Suggestion: compute f0 slope specifically at the end of the utterance as a feature
-
f0 range
- Larger range = more variation in pitch → likely uncertainty
-
min f0, max f0
- Very low minima or very high maxima = uncertainty (similar effect as large range)
-
min RMS
- Very low intensity suggests uncertainty (quiet speech perceived as low confidence)
- Mismatch: sometimes speakers reported low confidence, but raters perceived them as confident
- Indicates that apparent confidence ≠ actual confidence; speakers may project certainty despite inner doubt
- How to adapt speaker-wise normalization when user data is sparse remains an open question
- No explicit model/weights were suggested for combining features into an ML predictor of uncertainty
- Study limited to English speakers; Korean prosody may differ
- Sentence length was artificially fixed, so features tied to total duration are unsuitable for deployment
-
Correct + Uncertain: “You identified the right idea. Let’s review it once more to solidify your reasoning.”
-
Incorrect + Confident: “This is a common misconception. Let’s carefully compare the key differences.”
Response latency as a predictor of the accuracy of children's reports
- published in 2011 (69 citations)
As obvious as it may sound,
the longer a student takes to select an answer, the more uncertain they are.
Although the study was conducted in a multiple-choice selection scenario (not directly aligned with our speech-based service),
the analogy is straightforward:
Problem presented → student begins recording / begins speaking
The time lag between these two points can reasonably be considered a meaningful feature for uncertainty.
Fluency issues in L2 academic presentations: Linguistic, cognitive and psychological influences on pausing behaviour
- published in 2024 (4 citations)
This study, conducted in an EAP (English for Academic Purposes) class, explored where and why pauses occur in academic presentations by L2 (English as a foreign language) learners, and how they influence fluency. Unlike previous research that mainly emphasized quantitative measures (e.g., number/length of pauses), this work sought to explain the underlying causes—linguistic, cognitive, and psychological.
- (1) What are the types, positions, and frequencies of pauses?
- (2) What are the reasons for pauses?
-
Data
- 22 EAP students at an Australian university
- Each gave ~15-minute presentations
- A 1-minute segment was sampled for each; some 5-minute samples were also analyzed to test representativeness (t-test showed no significant difference)
-
Acoustic & Transcript Analysis
- Tools like Praat were used to measure the frequency, duration, and position of silent pauses and filled pauses (um, uh)
- Position categories: within-clause (MOC) vs. end-of-clause (EOC)
-
Stimulated Recall Interviews (SRI)
- Students re-watched their own presentation videos and were asked: “Why did you pause here?”
- 7 participants took part
- Researchers compared self-reports with observer interpretations
-
332 pauses total across 22 speakers
- 210 filled pauses, 122 silent pauses
- → Filled pauses more frequent
-
Distribution by position
- End-of-clause (EOC): 65.6%
- Mid-clause (MOC): 34.4%
-
Types of pauses
- Silent pause: complete break in sound, only breathing or silence
- Filled pause: hesitation sounds (um, uh, er, hmm, “like,” “so,” “you know”)
-
Planned pauses
- Typically at clause ends or after formulaic phrases (“first of all,” “generally speaking”)
- Functions: breathing, emphasis, giving processing time to listeners
-
Unplanned pauses
- Occurred during repetition, self-repair, lexical retrieval, planning, or unclassified cases
- Lexical retrieval/planning pauses often co-occurred with fillers like um/uh
The study identified overlapping linguistic, cognitive, and psychological causes for pauses:
-
Linguistic
- Pauses due to word search, pronunciation difficulty, or L1–L2 translation
- Self-monitoring and self-repair also triggered pauses
-
Cognitive
- Burden of recalling and organizing content (especially interpreting graphs) → more mid-clause pauses and repetitions
- Observers sometimes assumed “grammar checking,” but SRIs revealed it was often conceptual recall/restructuring instead
-
Psychological
- Anxiety/nervousness disrupted language and cognitive processing → increased silent/filled pauses and repetitions
- Conversely, confident speakers used clause-final pauses strategically (checking audience reaction, emphasis, giving processing time)
A single pause may have multiple overlapping causes (e.g., lexical search + anxiety). The same type of pause may stem from different reasons depending on the speaker. This shows how observer-only analysis can be misleading.
- Fluency is not just about speed or number of pauses. Effective presentations require simultaneous conceptual processing (cognitive), language production (linguistic), and psychological management.
- Clause-final pauses can be strategic and beneficial, serving discourse segmentation, emphasis, or audience processing time.
- Mid-clause silent pauses generally indicate processing difficulties, though causes vary by individual.
-
Differentiating pause types is insightful:
-
Silent pause
- Short, clause-final → marks discourse boundaries
- Long, mid-clause → failed recall, conceptual overload
-
Filled pause
- Short → minor planning strategy, not problematic
- Long → lexical difficulty, insufficient mastery to express concepts fluently
-
Planned pause
- Often for emphasis or breathing
-
Unplanned pause
- Common during recall, self-repair, or at ungrammatical break points
-
Prosodic Manifestations of Confidence and Uncertainty in Spoken Language — Structured Summary
- Published: 2008 (Interspeech/ICSLP, Brisbane)
- Author: Heather Pon-Barry (Harvard SEAS)
The paper asks where prosodic cues to confidence/uncertainty live in an utterance—are they concentrated on the target word/phrase that triggered the decision, or distributed in the surrounding context? It further contrasts self-reported certainty with perceived (listener-rated) certainty.
- Speakers: 20 native English speakers (14F/6M).
- Annotators: 5 native English raters (perceived certainty).
- Utterances: 600 total: 200 transit Q&A + 400 vocabulary sentences.
-
Boston public transit: fill-in-the-blank responses with constrained options (e.g., “Take the red line to the ___ and get off at ___”).
- Procedure included viewing the context alone, then context+options; a beep at 1500 ms cued reading aloud; speakers then self-rated certainty (1–5).
- Vocabulary: sentences completed by choosing 1 of 4 words from a small pool (incl. rare words to induce uncertainty). Same timing as above.
- Five raters judged perceived certainty (1–5) for all 600 utterances, presented without any textual context. Inter-rater agreement (κ) was modest and in line with prior affect work.
- Pitch (f0): min, max, mean, stdev, range, relative positions (min/max), absolute slope.
- Intensity (RMS): min, max, mean, stdev, relative positions (min/max).
- Temporal: total/percent silence, total duration, speaking duration (minus pauses), speaking rate.
- Normalization: all features z-scored per speaker (centered/standardized within speaker).
To localize cues, authors manually removed the target word region (including any immediately preceding pause) from each recording, producing separate context and target segments for parallel feature extraction.
- Perceived certainty exceeded self-reported certainty in 67% of utterances. This gap cautions that listeners often over-estimate a speaker’s confidence relative to the speaker’s own rating.
From Table 1 (correlations with mean perceived rating), the strongest effects are temporal:
- Total duration: −0.653 (longer ⇒ less certain)
- Total silence: −0.644 (more silence ⇒ less certain)
- Percent silence: −0.459
- Speaking duration: −0.515
-
Speaking rate: +0.134 (faster ⇒ slightly more certain)
Among f0/RMS: absolute f0 slope: +0.312 (steeper global slope associates with higher perceived certainty).
- Percent silence is much stronger in the target region (−0.568) than in context (−0.198): localized hesitations around the decision word flag uncertainty.
- Range f0 is stronger in context (−0.247) than target (≈0): broader pitch excursions in the surrounding phrase relate to uncertainty.
- Similar context-dominant patterns appear for min f0, max f0, f0 stdev, min RMS.
- Some features (e.g., absolute f0 slope, total duration, total silence) are best at the whole-utterance level; splitting doesn’t add predictive value for those.
- Temporal rhythm dominates: pauses, silence proportion, and overall length are the most reliable global indicators of perceived certainty.
-
Local vs. global cues:
- If you want to locate the uncertainty source, inspect target-region features—especially percent silence.
- If you’re classifying certainty for the whole utterance, global temporal features and absolute f0 slope are high-value.
- Perception ≠ self-report: Systems optimized only for perceived certainty risk missing actual uncertainty that speakers feel but listeners don’t detect.
- Read (non-spontaneous) speech was used to tightly control lexical content and repeat target words across certainty levels; generalization to spontaneous speech is future work.
- Additional planned work includes within-speaker analyses and feature-set comparisons for classification accuracy.
- ↓ Certainty: more/longer pauses (total/percent silence), longer utterances.
- ↑ Certainty (weak-moderate): steeper absolute f0 slope, slightly faster rate.
- Localization: percent silence peaks in target; f0 range patterns dominate in context.