during run evaluation of a test case inferences could be NoneType which has no len attribute and error is not caught

[ShaharKatz]

Describe the bug

When running eval via the adk-web UI i recieve the following error within the eval module of adk-python.
While debugging, it's clearly shown that the error catching code assumes that inferences is not None Type and a seperate error should be raised in that case.

if len(inference_result.inferences) != len(eval_case.conversation):
  raise ValueError(
      'Inferences should match conversations in eval case. Found'
      f'{len(inference_result.inferences)} inferences '
      f'{len(eval_case.conversation)} conversations in eval cases.'
  )

17/run_eval HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/applications.py", line 112, in call
await self.middleware_stack(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
await self.app(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 714, in call
await self.middleware_stack(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/services/market-scout-poc/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/cli/adk_web_server.py", line 742, in run_eval
eval_case_results = await _collect_eval_results(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/cli/cli_eval.py", line 188, in _collect_eval_results
async for eval_result in agen:
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/evaluation/local_eval_service.py", line 162, in evaluate
inference_result, eval_case_result = await evaluation_task
^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/.local/share/uv/python/cpython-3.12.9-macos-aarch64-none/lib/python3.12/asyncio/tasks.py", line 631, in _wait_for_one
return f.result() # May raise f.exception().
^^^^^^^^^^
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/evaluation/local_eval_service.py", line 151, in run_evaluation
return await self._evaluate_single_inference_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/shaharkatz/ai/libs/adk-python/src/google/adk/evaluation/local_eval_service.py", line 208, in _evaluate_single_inference_result
if len(inference_result.inferences) != len(eval_case.conversation):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: object of type 'NoneType' has no len()

To Reproduce
Steps to reproduce the behavior:

1. Install '...'
2. Run '....'
3. Open '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
After defining the test case and pressing the Run Evaluation button the error happens.

Desktop (please complete the following information):

• OS: macOS Sequia 15.3.1
• Python version(python -V): Python 3.12.9
• ADK version(pip show google-adk): 1.12

Model Information:
model agnostic issue.

Additional context
Add any other context about the problem here.

[ShaharKatz]

The underlying error that caused the bug is: User-provided Session id is not supported for VertexAISessionService.
This is caught in the inference_result in the _evaluate_single_inference_result function as seen by the error_message attribute that it has:

inference_result.error_message
'User-provided Session id is not supported for VertexAISessionService.'

[ShaharKatz]

I've opened an issue for the underlying issue here #2731.
It's still valuable for this issue to be addressed separately.

[dorz-saymine]

What is the status of this bug?
I'm using adk 1.17.0 and getting this error without using VertexAISessionService just the default MemorySessionService

[ShaharKatz]

@dorz-saymine Nearly complete, couple more days.

[surajksharma07]

@ShaharKatz This issue is still present in ADK 1.17.0.

The root cause is at local_eval_service.py:229 where the code calls len(inference_result.inferences) without checking if inferences is None.

This happens when inference fails (e.g., session service errors, network failures) - the InferenceResult has status=FAILURE and error_message populated, but inferences remains None.

Fix: Add None check before len() call and return NOT_EVALUATED status for failed inferences.

Defense-in-depth approach:
(1) filter failed inferences in evaluate() method,
(2) add safety check in _evaluate_single_inference_result().

Thanks!

[surajksharma07]

@ankursharmas Please have a look into it.