Skip to content

fix(gemini): raise on refusal/safety-blocked responses instead of reporting success - #507

Open
mittalpk wants to merge 1 commit into
google:mainfrom
mittalpk:fix/gemini-realtime-refusal-handling
Open

fix(gemini): raise on refusal/safety-blocked responses instead of reporting success#507
mittalpk wants to merge 1 commit into
google:mainfrom
mittalpk:fix/gemini-realtime-refusal-handling

Conversation

@mittalpk

@mittalpk mittalpk commented Jul 27, 2026

Copy link
Copy Markdown

Fixes #508.

Description

GeminiLanguageModel._process_single_prompt returned ScoredOutput(score=1.0, output=response.text) unconditionally. google.genai's response.text returns None (not an exception) when:

  • the prompt was blocked before generation (prompt_feedback.block_reason set, no candidates),
  • a candidate stopped for a non-STOP finish_reason (SAFETY, RECITATION, PROHIBITED_CONTENT, etc.) with no content, or
  • the response contains only non-text parts.

Every one of these was silently reported as a successful empty extraction, discarding the actual block/refusal reason.

This is the same class of bug already fixed for the OpenAI realtime provider in #491 / PR #496. That issue explicitly noted:

The Gemini realtime path has the same class of gap on a safety-blocked response.text (version-dependent); worth folding the same guard in there too.

PR #496 only touched openai.py, leaving the Gemini side unaddressed. I verified the None-on-block/refusal behavior directly against the installed google-genai 2.11.0 SDK source (GenerateContentResponse._get_text) rather than assuming it from the docstring.

Fix

After reading response.text, check for None and raise InferenceRuntimeError with the best available diagnostic:

  • prompt_feedback.block_reason (+ block_reason_message if present) for prompt-level blocks,
  • candidates[0].finish_reason for candidate-level non-STOP stops,
  • a generic message otherwise.

Also re-raises InferenceRuntimeError before the generic retry handler, so a safety-blocked prompt isn't retried with identical content (mirrors the intent of the OpenAI fix).

How Has This Been Tested?

Added GeminiRefusalHandlingTest to tests/provider_schema_test.py with 5 tests: successful text passthrough, prompt-level block, candidate-level non-STOP finish reason, no-diagnostic-available fallback, and a check that a refusal is not retried. The 4 error-path tests were confirmed to fail against the pre-fix code (git stash isolating just gemini.py).

.venv/bin/python -m pytest tests/provider_schema_test.py -q
# 32 passed (27 baseline + 5 new), same 1 pre-existing unrelated failure as clean main

.venv/bin/python -m pytest tests -k gemini -q
# 137 passed, 7 skipped (live API creds unavailable), 0 failed

.venv/bin/python -m pytest tests -q -m "not live_api"
# 673 passed (668 baseline + 5 new), same 39 pre-existing unrelated failures
# (OpenAI kwargs passthrough) as clean upstream/main -- no regressions

.venv/bin/python -m tox -e format      # clean
.venv/bin/python -m tox -e lint-src    # clean
.venv/bin/python -m tox -e lint-tests  # 10.00/10

Checklist

  • I have read the contribution guidelines.
  • I have added or updated tests where appropriate.
  • I have verified that all tests pass locally.
  • My changes are limited to the scope of this fix.

…orting success

_process_single_prompt returned ScoredOutput(score=1.0, output=response.text)
unconditionally. google.genai's response.text returns None (not an exception)
when: the prompt was blocked before generation (prompt_feedback.block_reason
set, no candidates), a candidate stopped for a non-STOP finish_reason (SAFETY,
RECITATION, PROHIBITED_CONTENT, etc.) with no content, or the response
contains only non-text parts. Every one of these was silently reported as a
successful empty extraction, discarding the actual block/refusal reason.

This is the same class of bug already fixed for the OpenAI realtime provider
in google#491/PR google#496 -- that issue explicitly noted "The Gemini realtime path has
the same class of gap... worth folding the same guard in there too," but
PR google#496 only touched openai.py, leaving this side unaddressed. Verified
directly against the installed google-genai 2.11.0 SDK source
(GenerateContentResponse._get_text) that None is returned, not raised, in
all three cases.

Fix: check output_text is None after reading response.text, and raise
InferenceRuntimeError with a diagnostic built from whichever information is
available -- prompt_feedback.block_reason(+message) for prompt-level blocks,
candidates[0].finish_reason for candidate-level stops, or a generic message
otherwise. Also re-raises InferenceRuntimeError before the generic retry
handler, matching the OpenAI fix's intent: retrying a safety-blocked prompt
with identical content won't produce a different result.

Added 5 tests to GeminiRefusalHandlingTest in tests/provider_schema_test.py:
successful text, prompt-level block, candidate-level non-STOP finish reason,
no-diagnostic-available fallback, and a check that a refusal is not retried.
All 4 error-path tests confirmed to fail against the pre-fix code (git stash
isolation of just gemini.py). Full verification:
- tests/provider_schema_test.py: 32 passed (up from 27 baseline, exactly the
  5 new tests), same 1 pre-existing unrelated failure as clean upstream/main.
- tests -k gemini: 137 passed, 7 skipped (live API creds unavailable), 0
  failed.
- Full suite (-m "not live_api"): 673 passed (up from 668 baseline), same 39
  pre-existing unrelated failures (OpenAI kwargs passthrough) as clean
  upstream/main -- no regressions.
- tox -e format, tox -e lint-src, tox -e lint-tests (10.00/10): all clean.
@google-cla

google-cla Bot commented Jul 27, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@github-actions

Copy link
Copy Markdown

No linked issues found. Please link an issue in your pull request description or title.

Per our Contributing Guidelines, all PRs must:

  • Reference an issue with one of:
    • Closing keywords: Fixes #123, Closes #123, Resolves #123 (auto-closes on merge in the same repository)
    • Reference keywords: Related to #123, Refs #123, Part of #123, See #123 (links without closing)
  • The linked issue should have 5+ 👍 reactions from unique users (excluding bots and the PR author)
  • Include discussion demonstrating the importance of the change

You can also use cross-repo references like owner/repo#123 or full URLs.

@github-actions github-actions Bot added the size/S Pull request with 50-150 lines changed label Jul 27, 2026
@mittalpk

Copy link
Copy Markdown
Author

Note on the linked-issue guideline: #508 is a newly-filed issue, so it obviously doesn't yet have the usual 5+ 👍 community-reaction threshold the contribution guidelines ask for. I want to flag that directly rather than let the automated check quietly fail.

The reason I'm asking for a look despite that: this isn't a speculative report. It's the direct, explicit follow-up to #491's own text — "The Gemini realtime path has the same class of gap on a safety-blocked response.text... worth folding the same guard in there too" — which #491's own fix (#496) didn't cover. I verified the exact None-on-block/refusal behavior against the installed google-genai 2.11.0 SDK source rather than assuming it, and the fix mirrors the pattern already accepted for the OpenAI provider in #496, just extended to the second provider the original report called out. Happy to wait for organic engagement on #508 if that's preferred instead — just didn't want to submit this fully silent on the gap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/S Pull request with 50-150 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Gemini realtime path returns empty success (score=1.0, output=None) on a safety-blocked or refused response

1 participant