Skip to content

Eval fails for non-English languages #3111

@lifnaja

Description

@lifnaja

Describe the bug
When evaluating text in other languages (e.g., Thai, etc.), the eval logic incorrectly returns mismatches (Match score: 0)— even when the evaluated expression should be valid or identical in meaning.

To Reproduce

root_agent = Agent(
    name="sample_agent",
    model="gemini-2.5-flash",
    description="Greeting agent",
    instruction="""
    You are an assistant that will reply to the user with only the word 'สวัสดี'
    """,
)

Expected behavior
Eval should correctly handle input in other languages.

Screenshots

Image

Desktop (please complete the following information):

  • OS: macOS
  • Python : 3.12.8
  • ADK version: 1.15.1

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used : gemini-2.5-flash

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

Labels

eval[Component] This issue is related to evaluation

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions