fix: Avoid BadRequestError due to invalid max_tokens #3667
Conversation
luis5tb
requested review from
ashwinb,
bbrowning,
ehhuang,
hardikjshah,
leseb,
mattf,
raghotham,
reluctantfuturist,
slekkala1,
terrytangyuan and
yanxi0830
as code owners
meta-cla
bot
added
the
CLA Signed
luis5tb
changed the title
luis5tb
force-pushed
the
max_tokens
branch
4 times, most recently
from
43fb189 to
ea42cf6
Compare
|
@luis5tb see CI failure: Thanks! |
|
The error in the unittest is weird
I trigger the recording workflow and now is green |
|
@luis5tb the amount of changes files seems off? |
I used the remote recording option (scripts/github/schedule-record-workflow.sh), there is no option for record-if-missing there. From what I see, using that option the "record" is hardcoded: https://github.com/llamastack/llama-stack/blob/main/.github/workflows/record-integration-tests.yml#L70 |
luis5tb
force-pushed
the
max_tokens
branch
2 times, most recently
from
edf6e86 to
4c68892
Compare
luis5tb
force-pushed
the
max_tokens
branch
3 times, most recently
from
cec6d76 to
3cba12d
Compare
To try to workaround this I created it with remote, fetch the commit, reset it and only add the new files (similar to the record + git restore + git add). It is still adding a lot of files though |
This patch ensures if max tokens is not defined it is set to None. This avoid some providers to fail, as they don't have protection for it being set to 0 Issue: llamastack#3666
Removed the modification, just adding the new content
@mattf, any more inputs on this? Is the PR ready? I already add only the new additions and not the minor changes (though still a long list of files) |
This patch ensures if max tokens is not defined, then is set to None instead of 0 when calling openai_chat_completion. This way some providers (like gemini) that cannot handle the `max_tokens = 0` will not fail Issue: llamastack#3666
## Summary Cherry-picks 5 critical fixes from main to the release-0.3.x branch for the v0.3.1 release, plus CI workflow updates. **Note**: This recreates the cherry-picks from the closed PR #3991, now targeting the renamed `release-0.3.x` branch (previously `release-0.3.x-maint`). ## Commits 1. **2c56a8560** - fix(context): prevent provider data leak between streaming requests (#3924) - **CRITICAL SECURITY FIX**: Prevents provider credentials from leaking between requests - Fixed import path for 0.3.0 compatibility 2. **ddd32b187** - fix(inference): enable routing of models with provider_data alone (#3928) - Enables routing for fully qualified model IDs with provider_data - Resolved merge conflicts, adapted for 0.3.0 structure 3. **f7c2973aa** - fix: Avoid BadRequestError due to invalid max_tokens (#3667) - Fixes failures with Gemini and other providers that reject max_tokens=0 - Non-breaking API change 4. **d7f9da616** - fix(responses): sync conversation before yielding terminal events in streaming (#3888) - Ensures conversation sync executes even when streaming consumers break early 5. **0ffa8658b** - fix(logging): ensure logs go to stderr, loggers obey levels (#3885) - Fixes logging infrastructure 6. **75b49cb3c** - ci: support release branches and match client branch (#3990) - Updates CI workflows to support release-X.Y.x branches - Matches client branch from llama-stack-client-python for release testing - Fixes artifact name collisions ## Adaptations for 0.3.0 - Fixed import paths: `llama_stack.core.telemetry.tracing` → `llama_stack.providers.utils.telemetry.tracing` - Fixed import paths: `llama_stack.core.telemetry.telemetry` → `llama_stack.apis.telemetry` - Changed `self.telemetry_enabled` → `self.telemetry` (0.3.0 attribute name) - Removed `rerank()` method that doesn't exist in 0.3.0 ## Testing All imports verified and tests should pass once CI is set up.
4 participants
This patch ensures if max tokens is not defined, then is set to None instead of 0 when calling openai_chat_completion. This way some providers (like gemini) that cannot handle the
max_tokens = 0will not failIssue: #3666