fix: replace panic with proper error handling in get_tokenizer #7175
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DOsinga
approved these changes
Feb 12, 2026
Collaborator
DOsinga
left a comment
DOsinga
left a comment
There was a problem hiding this comment.
thanks. this had been bugging me for a long time!
Collaborator
|
can you do the DCO thing so we can get this merged? |
The get_tokenizer() function returns Result<Arc<CoreBPE>, String> but was using panic!() when tokenizer initialization failed, which would crash the program instead of returning an error. Changed: - Updated TOKENIZER static to store Result<Arc<CoreBPE>, String> - Modified get_tokenizer() to return Err() instead of panic!() on failure - This allows proper error propagation instead of causing a crash Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: hobostay <110hqc@gmail.com>
hobostay
force-pushed
the
fix/panic-to-error-in-tokenizer
branch
from
3cec137 to
463aced
Compare
Contributor
Author
|
Done! I've added the |
jh-block
added a commit
that referenced
this pull request
Feb 12, 2026
* origin/main: (33 commits) fix: replace panic with proper error handling in get_tokenizer (#7175) Lifei/smoke test for developer (#7174) fix text editor view broken (#7167) docs: White label guide (#6857) Add PATH detection back to developer extension (#7161) docs: pin version in ci/cd (#7168) Desktop: - No Custom Headers field for custom OpenAI-compatible providers (#6681) feat: edit model and extensions of a recipe from GUI (#6804) feat: MCP support for agentic CLI providers (#6972) docs: keyring fallback to secrets.yaml (#7165) feat: load provider/model specified inside the recipe config (#6884) fix ask-ai bot hitting tool call limits (#7162) fix flatpak icon (#7154) [docs] Skills Marketplace UI Improvements (#7158) More no-window flags (#7122) feat: Allow overriding default bat themes using environment variables (#7140) Make the system prompt smaller (#6991) Pre release script (#7145) Spelling (#7137) feat(mcp): upgrade rmcp to 0.15.0 and advertise MCP Apps UI extension capability (#6927) ...
tlongwell-block
added a commit
that referenced
this pull request
Feb 12, 2026
…provenance * origin/main: (68 commits) Upgraded npm packages for latest security updates (#7183) docs: reasoning effort levels for Codex provider (#6798) Fix speech local (#7181) chore: add .gooseignore to .gitignore (#6826) Improve error message logging from electron (#7130) chore(deps): bump jsonwebtoken from 9.3.1 to 10.3.0 (#6924) docs: standalone mcp apps and apps extension (#6791) workflow: auto-update cli-commands on release (#6755) feat(apps): Integrate AppRenderer from @mcp-ui/client SDK (#7013) fix(MCP): decode resource content (#7155) feat: reasoning_content in API for reasoning models (#6322) Fix/configure add provider custom headers (#7157) fix: handle keyring fallback as success (#7177) Update process-wrap to 9.0.3 (9.0.2 is yanked) (#7176) feat: support extra field in chatcompletion tool_calls for gemini openai compat (#6184) fix: replace panic with proper error handling in get_tokenizer (#7175) Lifei/smoke test for developer (#7174) fix text editor view broken (#7167) docs: White label guide (#6857) Add PATH detection back to developer extension (#7161) ... # Conflicts: # .github/workflows/nightly.yml
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
get_tokenizer()would panic instead of returning a proper error when tokenizer initialization failsResulttypes as intended by its signatureDetails
The
get_tokenizer()function incrates/goose/src/token_counter.rsreturnsResult<Arc<CoreBPE>, String>, but was usingpanic!()when thetiktoken_rs::o200k_base()call failed. This would crash the program instead of gracefully handling the error.Changes made:
TOKENIZERstatic fromOnceCell<Arc<CoreBPE>>toOnceCell<Result<Arc<CoreBPE>, String>>Err(...)instead ofpanic!(...)ResultdirectlyTest plan
🤖 Generated with Claude Code