fix(gemini): raise on refusal/safety-blocked responses instead of reporting success - #507
fix(gemini): raise on refusal/safety-blocked responses instead of reporting success#507mittalpk wants to merge 1 commit into
Conversation
…orting success _process_single_prompt returned ScoredOutput(score=1.0, output=response.text) unconditionally. google.genai's response.text returns None (not an exception) when: the prompt was blocked before generation (prompt_feedback.block_reason set, no candidates), a candidate stopped for a non-STOP finish_reason (SAFETY, RECITATION, PROHIBITED_CONTENT, etc.) with no content, or the response contains only non-text parts. Every one of these was silently reported as a successful empty extraction, discarding the actual block/refusal reason. This is the same class of bug already fixed for the OpenAI realtime provider in google#491/PR google#496 -- that issue explicitly noted "The Gemini realtime path has the same class of gap... worth folding the same guard in there too," but PR google#496 only touched openai.py, leaving this side unaddressed. Verified directly against the installed google-genai 2.11.0 SDK source (GenerateContentResponse._get_text) that None is returned, not raised, in all three cases. Fix: check output_text is None after reading response.text, and raise InferenceRuntimeError with a diagnostic built from whichever information is available -- prompt_feedback.block_reason(+message) for prompt-level blocks, candidates[0].finish_reason for candidate-level stops, or a generic message otherwise. Also re-raises InferenceRuntimeError before the generic retry handler, matching the OpenAI fix's intent: retrying a safety-blocked prompt with identical content won't produce a different result. Added 5 tests to GeminiRefusalHandlingTest in tests/provider_schema_test.py: successful text, prompt-level block, candidate-level non-STOP finish reason, no-diagnostic-available fallback, and a check that a refusal is not retried. All 4 error-path tests confirmed to fail against the pre-fix code (git stash isolation of just gemini.py). Full verification: - tests/provider_schema_test.py: 32 passed (up from 27 baseline, exactly the 5 new tests), same 1 pre-existing unrelated failure as clean upstream/main. - tests -k gemini: 137 passed, 7 skipped (live API creds unavailable), 0 failed. - Full suite (-m "not live_api"): 673 passed (up from 668 baseline), same 39 pre-existing unrelated failures (OpenAI kwargs passthrough) as clean upstream/main -- no regressions. - tox -e format, tox -e lint-src, tox -e lint-tests (10.00/10): all clean.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
|
No linked issues found. Please link an issue in your pull request description or title. Per our Contributing Guidelines, all PRs must:
You can also use cross-repo references like |
|
Note on the linked-issue guideline: #508 is a newly-filed issue, so it obviously doesn't yet have the usual 5+ 👍 community-reaction threshold the contribution guidelines ask for. I want to flag that directly rather than let the automated check quietly fail. The reason I'm asking for a look despite that: this isn't a speculative report. It's the direct, explicit follow-up to #491's own text — "The Gemini realtime path has the same class of gap on a safety-blocked |
Fixes #508.
Description
GeminiLanguageModel._process_single_promptreturnedScoredOutput(score=1.0, output=response.text)unconditionally.google.genai'sresponse.textreturnsNone(not an exception) when:prompt_feedback.block_reasonset, no candidates),STOPfinish_reason(SAFETY,RECITATION,PROHIBITED_CONTENT, etc.) with no content, orEvery one of these was silently reported as a successful empty extraction, discarding the actual block/refusal reason.
This is the same class of bug already fixed for the OpenAI realtime provider in #491 / PR #496. That issue explicitly noted:
PR #496 only touched
openai.py, leaving the Gemini side unaddressed. I verified theNone-on-block/refusal behavior directly against the installedgoogle-genai2.11.0 SDK source (GenerateContentResponse._get_text) rather than assuming it from the docstring.Fix
After reading
response.text, check forNoneand raiseInferenceRuntimeErrorwith the best available diagnostic:prompt_feedback.block_reason(+block_reason_messageif present) for prompt-level blocks,candidates[0].finish_reasonfor candidate-level non-STOPstops,Also re-raises
InferenceRuntimeErrorbefore the generic retry handler, so a safety-blocked prompt isn't retried with identical content (mirrors the intent of the OpenAI fix).How Has This Been Tested?
Added
GeminiRefusalHandlingTesttotests/provider_schema_test.pywith 5 tests: successful text passthrough, prompt-level block, candidate-level non-STOPfinish reason, no-diagnostic-available fallback, and a check that a refusal is not retried. The 4 error-path tests were confirmed to fail against the pre-fix code (git stashisolating justgemini.py).Checklist