Level Management System
The Level Management System is a framework designed to continuously evaluate and update a user’s English listening ability. This system forms the foundation of our personalized content pipeline. Only by understanding the user’s current proficiency, we can generate audio content that matches their level. Because our target users are casual learners who prefer light, low-effort daily listening experiences, maintaining an accurate proficiency level is essential for providing content that feels neither too easy nor too difficult.
In the initial version of the system, we delegated level evaluation entirely to an LLM. After each piece of content, the user provided an understanding score from 0 to 100, and we passed this information to the LLM to determine their updated CEFR-based level. However, this approach revealed several limitations. First, the LLM’s decision-making lacked transparency and consistency. Given limited context, the model often produced non-deterministic results, making it difficult to validate or explain why the level was updated in a particular way. Second, using a single CEFR score obscured the multi-dimensional nature of listening difficulty. Vocabulary complexity, sentence structure, and speech rate all influence comprehension, but collapsing these factors into a single number removed essential information. As a result, cases with very different underlying challenges were treated as equivalent. These issues motivated us to design a more linguistically grounded, heuristic-based algorithm
To accommodate casual learners, the system must minimize forced interactions. Requiring quizzes or comprehension tests would introduce friction and reduce usability. Therefore, level updates should rely primarily on naturally occurring signals during content consumption. At the same time, personalization must be precise enough for learners to feel that the content matches their actual ability. This requires the system to treat different sources of difficulty independently rather than relying on a single aggregated score.
To address these challenges, we turned to principles from Human–Computer Interaction, particularly the concept of user modeling. User modeling refers to the process of building an internal representation of a user’s knowledge, preferences, goals, and abilities so that the system can adapt to them. HCI literature distinguishes two approaches: explicit modeling, where users directly provide information through tests or surveys, and implicit modeling, where the system infers user characteristics from natural interactions.
Figure 1. Knowledge-based HCI framework, adapted from Fischer (2001).
Fischer (2001) illustrates that user modeling operates through both explicit and implicit communication channels. Explicit information is consciously provided by the user, whereas implicit signals arise naturally from interaction and often reveal more authentic patterns of knowledge and difficulty. This perspective emphasizes that effective user modeling cannot rely solely on explicit inputs. Instead, implicit interaction logs play a central role in building richer and more adaptive representations of the user. Because explicit assessments would burden casual learners, we adopt an these implicit approach. In our system, behaviors such as pause and rewind usage, dictionary lookups, adding words to a personal vocabulary list, and optional feedback on comprehension or speech-rate difficulty function as implicit indicators of how challenging the content was. These natural interactions provide continuous, lightweight information about user difficulty without requiring any disruptive actions. By interpreting these signals, the system maintains three distinct dimensions of listening difficulty—lexical, syntactic, and auditory. Managing these dimensions separately allows the system to understand why a user struggled with specific content and to deliver more accurate personalized audio in the future.
1. The "Why": Our Guiding Philosophy
The CEFR standards are highly trustworthy for describing a user's language ability, but they lack the concrete, quantitative measurements needed for a dynamic and reliable system (especially when working with AI). To accurately assess and adapt to a user's level, LingoFit required absolute and powerful metrics.
We found that the DLI-FLC (Defense Language Institute Foreign Language Center) focuses its studies on this exact matter. In their "Guide to Measuring Listening," they highlight several critical factors for measuring listening difficulty [1].
While we could not implement all factors due to time limitations, we selected the three most impactful metrics as our core:
-
Lexical Difficulty
- This refers to the challenge of the vocabulary used in a text or speech. High lexical difficulty means the words are rare, specialized, or very diverse (a wide range of different words), making it harder for a reader or listener to understand.
-
Syntactic Complexity
- This describes the complexity of the sentence structures. High syntactic complexity involves long, intricate sentences with multiple clauses (e.g., "The man, who was tired from a long day, decided to skip the event, which he later regretted"). Simple sentences (e.g., "The tired man skipped the event.") have low syntactic complexity.
-
Auditory Difficulty
- This is the challenge a person faces in understanding and processing sounds, especially spoken language. It's not about hearing loss but about how the brain interprets sounds. This can include difficulty understanding speech in noisy environments, following multi-step verbal instructions, or telling the difference between similar-sounding words.
LingoFit evaluates a user's English level across these three independent axes and uses them with different weights to generate perfectly tailored audio content.
2. The "What": Defining Our Three Metrics
Here are is how LingoFit measures each metric:
- Lexical Difficulty: We measure this by mapping every word in a script to its CEFR level and then calculating the statistical distribution of all CEFR levels (e.g., 54% A1, 18% A2, 10% B1) within that script.
- Syntactic Complexity: We measure this by calculating the Average Sentence Length (ASL) of the script.
- Auditory Difficulty: We measure this by calculating the narrator's speech rate in Words Per Minute (WPM).
3. The "How": Our Application & Ground Truth
We quantify each metric using the following logic, which is based on established academic research rather than simple intuition.
- Lexical & Syntactic Logic
For our "ground truth," we adopted the findings of a large-scale study that analyzed over 17,000 sentences to find the average CEFR word distribution and ASL for each CEFR level [2].
When LingoFit generates a script, it aims to match these data-driven profiles.
- Auditory Logic
For our "ground truth," we referenced research on the speech rate of authentic materials. This research identified the natural English speech rate (120-180 WPM) and the average rate of TED Talks (147.4 WPM) as key benchmarks [3].
LingoFit uses a "success range" based on these findings to generate audio at a speed that is appropriate for the user's level.
| Level | WPM |
|---|---|
| A1 | 90-140 (0.7) |
| A2 | 100-160 (0.85) |
| B1 | 110-180 (1.0) |
| B2 | 130-200 (1.1) |
| C1 | 160-240 (1.2) |
| C2 | 160+ (1.2) |
As described earlier in Section 1 (Motivation & Requirements), our system adopts a user modeling approach based on implicit behavior logs that naturally arise during audio-learning sessions. To accomplish this, we collect four types of behavioral interaction logs and combine them with two explicit inputs—user-reported difficulty regarding speech rate and overall comprehension. These six inputs are treated as a single vector and serve directly as the input to our level-update feedback logic.
A pause refers to the action of stopping audio playback during listening, whereas a rewind indicates moving the playback bar backward to listen to a segment again. Merkt et al. (2022) demonstrated that pauses in online learning environments typically occur when learners perceive a segment as difficult to understand. In other words, pausing behavior naturally increases when learners experience cognitive load or feel the need to process information more deeply. In addition, Ozcelik (2021) conducted an experiment in which learners listened to a four-minute CEFR B1-level BBC audio clip and completed related tasks over a total of 26 minutes. During approximately 19 minutes of this period, learners were free to use pause, rewind, and fast-forward controls. On average, individual learners performed 10.6 rewinds, 7.0 pauses, and 1.7 fast-forwards—about 19 self-control actions in total. This corresponds to roughly one action per minute, meaning that pause and rewind behaviors each occur approximately 0.5 times per minute under typical learning conditions. Based on these findings, we use this average behavioral pattern as a baseline to evaluate user difficulty. By measuring whether a learner performs pause or rewind actions significantly more or less frequently than the expected baseline, we infer how challenging the content was for that learner.
A vocabulary lookup occurs when a learner taps a word during the audio session to check its meaning. This action indicates that the learner does not immediately recognize the word and therefore reflects lexical difficulty. Lookup frequency serves as a useful signal for estimating how often the learner encounters unfamiliar vocabulary. Prior research in applied linguistics shows that learners generally need to understand about 95% of the words in a script to achieve adequate comprehension (Nation, 2006). This threshold is widely used as a baseline for determining whether a text or audio clip is accessible to learners. Thus, deviations from this baseline—observedl through increased lookup behavior—can be interpreted as meaningful evidence of lexical challenge in the given content. Vocabulary save actions represent an even stronger indication of difficulty. Saving a word to a personal vocabulary list suggests not only that the learner did not know the word, but also that they have identified it as requiring future study. For this reason, save events are treated as high-confidence signals of lexical difficulty, stronger than simple lookups. By analyzing the frequency and distribution of lookup and save actions, the system can better infer the learner’s lexical struggle within the content and integrate these signals into level-updating logic.
Understanding feedback is a self-assessed measure in which the learner evaluates how well they understood the audio content after completing the session. This feedback directly reflects the learner’s perceived difficulty and overall comprehension, making it an important explicit indicator of their subjective experience. Speech-rate feedback captures whether the learner felt that the speaker’s pace was too fast, appropriate, or too slow. Because speech rate is one of the most influential factors affecting listening difficulty, this feedback serves as a reliable signal of the learner’s auditory load and how manageable the pacing was for them. Both of these feedback types are explicit inputs, intentionally provided by the learner, and therefore serve as high-confidence signals that complement the implicit behavior logs—such as pause, rewind, lookup, and save—collected during the session. When combined with these behavioral indicators, understanding and speech-rate feedback play a crucial role in forming a more precise picture of the learner’s actual experience and perceived difficulty of the content.
This section describes the algorithm we implemented to update a user's lexical, syntactic, and speed levels after each listening session. We implemented the level update mechanism using the following algorithm:
Figure 2. Level Update by Session Feedback
In the level-update algorithm, two predefined constants play an essential role in transforming raw session feedback into stable and interpretable level adjustments. The first is the DEFAULT_WEIGHT_MATRIX, which is implemented as a fixed 6×3 matrix. Its six input dimensions correspond to the behavioral and explicit signals collected in each session—pause count, rewind count, vocabulary lookups, vocabulary saves, self-rated understanding difficulty, and self-rated speed difficulty. Each column of the matrix represents one of the three internal level dimensions (lexical, syntactic, and speed), and each entry w_ij encodes how strongly the i-th input signal should influence the j-th level. In other words, the matrix determines how the six feedback channels are combined to produce meaningful adjustments across the three CEFR-aligned skill components.
The second constant is DEFAULT_CLIP_RANGES, which defines the allowable range for a single session’s level change. We set this range to [–8, +8] for all three level dimensions. The purpose of this bound is to prevent sudden spikes or drops in a user’s level due to a single session with unusually high or low interaction counts. By enforcing this cap, the system ensures that level progression remains smooth and pedagogically reasonable, reflecting gradual learning rather than volatile fluctuations.
Step 1: Normalize Feedback Inputs All raw feedback values are converted into normalized scores: pause/rewind counts are normalized by content duration. vocabulary lookup/save counts are normalized by content vocabulary statistics. understanding and speed difficulty are normalized from user self-ratings. These normalized factors form six comparable input features.
Step 2: Construct Input Vector The normalized values are aggregated into a vector which serves as the complete representation of session feedback.
Step 3: Compute Level Deltas The algorithm multiplies the input vector with the weight matrix and applies the clip range, producing the updates for lexical, syntactic, and speed levels.
Step 4: Apply and Finalize Updates Each level is updated and clipped to the global range [0, 300], ensuring long-term stability. The updated user profile is then committed to the database, completing the session-based level adjustment.
How the three user proficiency metrics (Lexical Difficulty, Syntactic Complexity, and Auditory Difficulty) are integrated throughout the audio generation pipeline.
Audio Generation Pipeline Flow Overview
- Voice Selection: Weighted combination (30% lexical + 20% syntactic + 50% speed) → CEFR level → voice accent/style
- Script Generation: Separate lexical and syntactic levels → targeted vocabulary and grammar complexity
- TTS Generation: Speed level → playback speed multiplier → audio generation speed
- Output: Personalized audio content matching user's three-dimensional proficiency profile
1.1 Voice Selection
Code Location: AudioService._select_voice_algorithmically()
The voice selection algorithm uses a weighted combination of all three levels to determine the appropriate voice complexity:
# Convert user levels to float and handle legacy None values
lexical_level = float(user.lexical_level) if user.lexical_level is not None else 50.0
syntactic_level = float(user.syntactic_level) if user.syntactic_level is not None else 50.0
speed_level = float(user.speed_level) if user.speed_level is not None else 50.0
# Weighted scoring for voice selection
weighted_score = round(
(lexical_level * 0.3) + # 30% lexical weight
(syntactic_level * 0.2) + # 20% syntactic weight
(speed_level * 0.5), # 50% speed weight (primary factor)
1
)Rationale:
- Speed level has the highest weight (50%) as it most directly impacts voice accent/style difficulty
- Lexical level (30%) affects vocabulary complexity in voice selection
- Syntactic level (20%) has lower impact on voice characteristics
The weighted score is converted to a CEFR level using get_cefr_level_from_score() and mapped to appropriate voice accent/style combinations via LEVEL_CHALLENGE_MAP.
1.2 Script Generation
Code Location: AudioService._generate_script()
An extensive prompt is used to generate a script that is tailored for the user using the two metrics lexical and syntactic:
This prompt is a highly engineered, multi-component instruction designed to control AI output with precision. Its key components are:
- Role, Context, and Constraints
The prompt begins by setting a clear role (
You are a scriptwriter) and a specific task (Generate a script for an audio narration). It immediately establishes non-negotiable constraints and contextual variables:
You are a scriptwriter. Generate a script for an audio narration.
The script must be between 1 and 2 minutes long (around {TARGET_SCRIPT_WORDS} words).
The narration will be read by a single speaker: {voice_detail}.
Theme: {theme}
Mood: {mood}
It also receives the two distinct target levels from the user's profile:
User's Current English Profile:
- Lexical Level: {lexical_cefr_str} (for vocabulary)
- Syntactic Level: {syntactic_cefr_str} (for sentence structure)
- Master Generation Principles (Ground Truth) This is the core of the prompt. It commands the model to follow a specific, data-driven methodology rather than just its general knowledge.
Principle 1: Lexical Difficulty ("Closest Profile Match")
The model is forbidden from interpolating. It must strictly adhere to a "Ground Truth" table mapping CEFR levels to exact word-type percentages.
This rule is explicitly tied to the user's {lexical_cefr_str} level.
**1. Lexical Difficulty: "Closest Profile Match"**
Your primary goal is to match the vocabulary distribution of the **Target Lexical Level ({lexical_cefr_str})**.
You MUST strictly follow the lexical profile for **{lexical_cefr_str}** from the table below. Do not interpolate; match this row precisely.
**Ground Truth (Lexical Profile for Content Words)**
| CEFR Level | A1 Word % | A2 Word % | B1 Word % | B2 Word % |
| :--- | :--- | :--- | :--- | :--- |
| **A1** | 66.3% | 15.2% | 4.8% | 1.3% |
| **A2** | 54.6% | 18.2% | 10.1% | 3.2% |
| **B1** | 41.7% | 20.1% | 15.5% | 5.9% |
| **B2** | 31.9% | 19.1% | 17.8% | 7.9% |
| **C1** | 23.7% | 16.9% | 17.3% | 8.5% |
| **C2** | 16.5% | 15.2% | 16.3% | 6.8% |
Principle 2: Syntactic Complexity ("ASL Target")
The model is instructed to target a specific "Success Range" for Average Sentence Length (ASL).
This rule is tied to the user's {syntactic_cefr_str} level.
**2. Syntactic Complexity: "ASL Target"**
Your second goal is to control the Average Sentence Length (ASL).
You MUST target the ASL "Success Range" for the **Target Syntactic Level ({syntactic_cefr_str})**.
**Ground Truth (ASL Range)**
| CEFR Level | ASL "Success Range" | (Reference Avg.) |
| :--- | :--- | :--- |
| **A1** | 4.0 ~ 10.0 words | (7.7) |
| **A2** | 8.0 ~ 14.0 words | (10.9) |
| **B1** | 11.0 ~ 19.0 words | (15.2) |
| **B2** | 14.0 ~ 22.0 words | (18.0) |
| **C1** | 15.0 ~ 25.0 words | (19.0) |
| **C2** | 15.0 ~ 25.0 words | (19.2) |
(Note: For B2-C2 levels, complexity is defined more by the **Lexical Profile** (Metric 1) than by ASL, as their ASL values naturally converge.)
Principle 3: Qualitative CEFR Descriptors
The prompt recognizes that high-level proficiency (C1/C2) is not just quantitative.
It must also incorporate qualitative features from a provided list. For example, if the user's level is C1, the model is forced to include idiomatic expressions and implied relationships, directly referencing the official CEFR descriptors.
**3. Qualitative CEFR Descriptors**
For C1 and C2 levels, relying *only* on the quantitative tables is insufficient.
You MUST *also* incorporate the qualitative features corresponding to **both** of the user's target levels ({lexical_cefr_str} and {syntactic_cefr_str}).
For example, if the lexical level is C1, you MUST include idiomatic expressions. If the syntactic level is B2, you MUST ensure arguments are complex but on a familiar topic.
* **A1:** Follow language which is very slow and carefully articulated, with long pauses. Recognize concrete information (e.g., places, times) delivered slowly and clearly.
* **A2:** Understand enough to meet needs of a concrete type. Understand phrases and expressions related to areas of most immediate priority (e.g., personal information, shopping, local geography).
* **B1:** Understand straightforward factual information about common or job-related topics. Understand the main points in clear standard language, including short narratives.
* **B2:** Understand standard language, live or broadcast. Understand the main ideas of complex abstract topics, including technical discussions. Follow extended discourse and complex arguments if the topic is reasonably familiar.
* **C1:** Understand enough to follow extended discourse on abstract and complex topics. **Recognize a wide range of idiomatic expressions and colloquialisms.** Follow extended discourse even when it is not clearly structured and **when relationships are only implied and not signalled explicitly.**
* **C2:** Understand with ease virtually any kind of language, whether live or broadcast, **delivered at fast natural speed.**
- Lexical Calibration (Few-Shot Examples) To ensure the model's internal understanding of word difficulty aligns with the system's specific wordlist, the prompt provides extensive few-shot examples.
Purpose: It explicitly states this is to calibrate your internal knowledge.
Data: It provides long lists of "representative examples" for each level to ground the model's generation.
*** LEXICAL CALIBRATION (FEW-SHOT EXAMPLES) ***
To help you calibrate your internal knowledge to our specific wordlist, here are representative examples of content words at each level.
* **A1 Examples:** play, bicycle, poor, news, pizza, cream, shopping, around, five, cover, reporter, card, picture, excited, judge, science, snow, more, street, button, buy, well, case, mouth, no, glass, late, black, happy, Wednesday, hotel, all right, grass, outside, umbrella, history, spot, Thursday, cold, taxi
* **A2 Examples:** difficulty, pal, trust, car park, mosque, frightening, competition, appearance, scale, angel, claim, cross, ruin, search, fantastic, ourselves, normal, fence, talent, high, exhibition, north, fault, appreciate, superlative, mysterious, shampoo, possible, few, hey, entertainment, view, pride, spaceship, around, journey, IT, grandson, clerk, aged
* **B1 Examples:** timely, laughter, interact, weakness, forehead, refusal, nutritious, dump, historian, strain, board, sunrise, compose, stream, tragic, net, through, incredible, complete, currently, unexpected, toothpaste, nervousness, anyhow, facility, monitor, substitute, direct, twist, southeast, analysis, tremendous, publisher, adviser/advisor, jug, continuous, remainder, transport, roadside, experience
* **B2 Examples:** faint, reinforce, fatal, fine, upgrade, elemental, flash, mother-in-law, inch, gently, tolerant, royalty, weakly, grim, sufficiently, observer, conductor, innovation, remaining, cherry, martial art, imperative, flash, lest, transitive, editorial, exclusion, nervously, soliloquy, win, disappearance, trivial, retard, bumper, hyphen, cuff, cubism, cascade, disrupt, inspector
* **C1 Examples:** rudimentary, facilitation, vegetation, preacher, detriment, blankness, reenact, sacrifice, inexplicable, prolific, contextual, aimlessly, dither, conditionally, revere, render, bribery, premise, fanatic, provocative, prophet, exuberant, insensitively, carrier, isolated, formulate, overdraft, pertinent, somersault, quirky, jersey, rustle, anthropology, dismay, violet, absolute, commercially, stoke, commission, maneuver/manoeuvre
* **C2 Examples:** kinetically, philanthropic, angsty, facsimile, colloquium, flit, agility, infernally, extant, wistful, posterity, ferocity, ingrate, circuit, thicket, consternation, all-encompassing, enabler, maelstrom, testimonial, daunt, stringently, avian, adversely, blurb, diffuse, annex, drudgery, formidably, solitariness, tetchy, reverb, salivary, tactic, incipient, hazard, incumbent, bona fide, lassitude, extracurricular
- Strict Output Formatting The prompt concludes with rigid formatting rules to ensure the output is machine-parsable and ready for use.
-
Title: Must start with TITLE: [Your Title Here].
-
Sentence-per-Line: A critical rule (Every single sentence MUST be a new line.) allows the system to easily split the script for processing.
-
Exclusion: It explicitly forbids any non-dialogue text.
*** IMPORTANT FORMATTING RULES ***
1. Start with a title on the first line formatted as: TITLE: [Your Title Here]
2. Add one blank line after the title.
3. (Special Point 3) Every single sentence MUST be a new line.
A sentence ends with a period, question mark, or exclamation mark.
4. Do NOT include speaker names (e.g., "Narrator:"), scene directions,
or any text other than the dialogue itself.
Code Location: AudioService._generate_audio_with_timestamps()
The TTS generation incorporates the speed level to control audio playback speed:
# Handle legacy users and convert Decimal to float
speed_level = float(user.speed_level) if user.speed_level is not None else 50.0
# Calculate target speed multiplier
target_speed = get_speed_from_level_score(speed_level)Speed Mapping:
The get_speed_from_level_score() function maps user speed levels to playback multipliers:
- A1 (0-25): 0.75x - 0.80x (slower speech)
- A2 (25-50): 0.80x - 0.90x
- B1 (50-100): 0.90x - 1.00x (normal speech)
- B2 (100-150): 1.00x - 1.08x
- C1 (150-200): 1.08x - 1.12x
- C2 (200+): 1.12x - 1.13x (faster speech)
Auditory Difficulty Factors:
- Lower speed levels → slower, clearer speech for beginners
- Higher speed levels → faster, more natural speech pace for advanced learners
- Speed adjustment is applied during ElevenLabs TTS generation
This three-level approach enables more precise content adaptation compared to the previous single CEFR level system, allowing for nuanced personalization based on distinct language learning competencies.
[1] Defense Language Institute Foreign Language Center (DLI-FLC). (n.d.). DLI-FLC Guide to Measuring Listening. Retrieved from https://www.govtilr.org/Publications/DLI-FLC%20Guide%20to%20Measuring%20Listening.pdf
[2] Arase, Y., Uchida, S., & Kajiwara, T. (2022). CEFR-Based Sentence Difficulty Annotation and Assessment.
[3] Hidayati, Y., & Suharyadi, S. (2023). The Speech Rate and Vocabulary Profile of TED-Ed Videos as Extensive Listening Materials for EFL Learners. LLT Journal, 26(2), 483-495.
[4] Merkt, M., Weigand, S., Heier, A., & Schwan, S. (2022). Pushing the button: Why do learners pause online videos? Computers in Human Behavior, 128, 107–148.
[5] Ozcelik, E. (2021). Clickstream Data Analysis in Listening Comprehension Tasks: Pause, Rewind, and Fast-forward Behaviors of Language Learners. Journal of Educational Technology & Society, 24(3), 15–30.
[6] Fischer, G. (2001). User Modeling in Human–Computer Interaction. User Modeling and User-Adapted Interaction, 11(1), 65–86.
[7] Nation, P. (2006). How large a vocabulary is needed for reading and listening? The Canadian Modern Language Review, 63(1), 59–82.