Skip to content

STT Evaluation: Debug STT errors in ATREE evaluation run #672

@AkhileshNegi

Description

@AkhileshNegi

Is your feature request related to a problem?
We encountered issues during STT Evaluation in staging with ATREE, where several samples returned an error stating {'message': 'Cannot fetch content from the provided URL.', 'code': 3}. This inconsistency affected the reliability of the evaluation process.

These evaluation runs by ATREE are in staging (sample language: Kannada):

Evaluation Run Run ID Samples Failed
STT Evals 6 260 49 20
STT Evals 5 259 49 16
STT Evals 5 252 49 11
STT Evals 4 252 49 13

Describe the solution you'd like

  • Debug the current STT evaluation process to identify the root cause of the errors.
  • Consider altering our approach for multi-modal batch operations using Gemini.

Screenshot

Image

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status

To Do

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions