fix(cli/serve): import serve handlers lazily so the CLI works without the serve extra#46473
Open
Anai-Guo wants to merge 1 commit into
Open
fix(cli/serve): import serve handlers lazily so the CLI works without the serve extra#46473Anai-Guo wants to merge 1 commit into
serve extra#46473Anai-Guo wants to merge 1 commit into
Conversation
4 tasks
Member
|
cc @LysandreJik for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
Fixes the
transformersCLI crashing withNameErrorwhen the optionalserveextra (which providesopenai) is not installed.Importing any CLI command triggers this chain:
CompletionCreateParamsStreaming(and the otheropenai.typessymbols) are only imported insideif is_serve_available():, but the handler classes reference them at class-definition time. With theserveextra absent,is_serve_available()isFalse, the symbol is never imported, and the module raises:Because
server.pyimports the four handler modules unconditionally (only thefastapiimports were guarded), the whole CLI becomes unusable withoutservedeps — e.g.transformers env, as reported in #46468.The fix
In
cli/serving/server.py:from __future__ import annotationssobuild_server's parameter/return annotations (ChatCompletionHandler,FastAPI, …) are not evaluated at definition time.chat_completion,completion,response,transcription) into the existingif is_serve_available():guard, next to thefastapiimports.ModelManagerand.utilsstay unconditional — they have no serve-only dependencies.This is sufficient and minimal:
build_serveronly uses the handlers in annotations, never as values in its body, so the guarded imports aren't needed at import time.serve.pyalready imports the handler classes itself (lazily, inside the run path) when it actually starts a server, so guarding them inserver.pydoes not affect serving.tests/cli/test_serve.pyimports onlybuild_serverfrom this module (and is gated with@require_serve), so it is unaffected.Verification
With the
serveextra absent (is_serve_available()isFalse):python -c "import transformers.cli.serving.server"→NameError: name 'CompletionCreateParamsStreaming' is not definedtransformers env/version/downloadwork again.With the extra present, serving behaviour is unchanged.
Note: the
TypeErrorin the issue's main traceback originates in the model's remotemodeling_nemotron_h.py(trust_remote_code), which is outside this repo; this PR addresses thetransformers envNameErrorregression reported in the same issue.🤖 Generated with Claude Code