Scale the default Whisper batch size to the detected GPU memory - #136
Conversation
The fixed default of 16 needs roughly 8 GB VRAM with the large-v2 model; on smaller GPUs the run crashed out-of-memory unless the user knew to lower --whisper_batch_size by hand. --whisper_batch_size now defaults to auto: after GPU detection the batch size resolves to 16 (8+ GB), 8 (6 GB) or 4 (4 GB), field-tested values that run stable on the respective cards. A lower batch size only slows transcription down - the result is unchanged. CPU runs and explicit --whisper_batch_size values behave exactly as before; the flag also accepts the literal 'auto'. GUI: the Batch Size spinner gains an 'Auto' position (0, new default) that omits the CLI flag so the runtime scaling applies; saved explicit values keep working. The scary sub-6GB VRAM warning now explains the automatic scaling instead of predicting a crash, and the installer summaries and README no longer tell small-GPU users to tune the batch size by hand.
|
Warning Review limit reached
Next review available in: 33 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (9)
📝 WalkthroughWalkthroughWhisper batch size now defaults to automatic GPU-memory scaling, with explicit CLI, interactive, and GUI overrides. Runtime detection selects sizes from available VRAM, tests cover fallback and threshold behavior, and help or installer guidance reflects the new defaults. ChangesAutomatic Whisper batch sizing
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant init_settings
participant check_requirements
participant auto_whisper_batch_size
participant torch.cuda
User->>init_settings: select auto or explicit batch size
init_settings->>check_requirements: provide settings
check_requirements->>auto_whisper_batch_size: resolve selected device
auto_whisper_batch_size->>torch.cuda: query GPU availability and VRAM
torch.cuda-->>auto_whisper_batch_size: VRAM information or error
auto_whisper_batch_size-->>check_requirements: batch size
check_requirements-->>User: print selected batch size
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/modules/common_print.py`:
- Line 68: Synchronize all VRAM guidance with the runtime batch-size resolver:
update src/modules/common_print.py lines 68-68 and README.md lines 264-264 to
use the actual 16/8/4 GB thresholds, revise README.md lines 857-865 to remove
the contradictory 8 GB fit claims, and update install/auto_install.bat lines
223-229 and 241-249 plus install/auto_install.sh lines 249-255 and 266-273 so
unknown VRAM is not guaranteed to fit and 7 GB GPUs receive accurate guidance.
In `@src/modules/init_interactive_mode.py`:
- Around line 62-66: In the Whisper batch-size input flow, rename the misspelled
whipser_batch_size_response variable and only assign an explicit batch size when
the response is a positive integer; treat empty, non-numeric, and zero or
negative inputs as None so auto mode remains enabled.
In `@src/UltraSinger.py`:
- Around line 2007-2009: Validate the explicit value assigned to
settings.whisper_batch_size in the argument-parsing branch: preserve “auto” as
None, but require numeric values to be positive integers and reject zero or
negative inputs before inference proceeds.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 43430c75-42fc-4f71-bff1-7d54d24aef6d
📒 Files selected for processing (12)
README.mdinstall/auto_install.batinstall/auto_install.shpytest/gui/test_freestyle_threshold_widgets.pypytest/modules/DeviceDetection/test_device_detection.pysrc/Settings.pysrc/UltraSinger.pysrc/gui/config.pysrc/gui/settings_tab.pysrc/modules/DeviceDetection/device_detection.pysrc/modules/common_print.pysrc/modules/init_interactive_mode.py
- Reject zero/negative --whisper_batch_size with a clear error (matching the existing range-validation pattern); 'auto' still maps to None. - Interactive mode: only a positive integer selects an explicit batch size (zero/empty/non-numeric keep auto); fix the misspelled variable. - Raise the top tier threshold to 7.5 GB so 7 GB cards get batch size 8 (16 needs ~8 GB) while true 8 GB cards that report slightly less than 8.0 still get 16; add tests for both boundary cases. - Sync every user-facing description (help, README, GUI tooltip, installer summaries) to the actual tiers and soften the fit claims.
1 participant
Problem
The fixed default batch size of 16 needs roughly 8 GB VRAM with the default Whisper model (
large-v2). On smaller GPUs, runs crashed out-of-memory unless the user knew to lower--whisper_batch_sizeby hand - the installer could only print a tip. Field testing showed the values that run stable: 8 on a 6 GB card, 4 on a 4 GB card.Change
--whisper_batch_sizenow defaults to auto: after GPU detection, the batch size resolves to the detected VRAM:A lower batch size only slows transcription down - the result is unchanged, so this is safe to do automatically. The resolution happens at runtime (not install time), so it adapts if the hardware changes and works for CLI and GUI alike. The resolved value is printed at startup ("Whisper batch size: 4 (auto, scaled to GPU memory)") and is part of the transcription cache key as before (only resolved ints reach it).
Details:
--whisper_batch_size <int>behaves exactly as before and wins over auto; the flag also accepts the literalauto. A failed VRAM query (driver in a bad state) falls back to the historic default of 16.auto_install.bat/.sh) and the README no longer tell small-GPU users to tune the batch size by hand; the remaining tips (batch size 2/1,int8) stay for the still-out-of-memory case.Testing
Summary by CodeRabbit
New Features
Bug Fixes
Documentation