Skip to content

Refactor model config#109

Merged
JosephMarinier merged 4 commits into
mainfrom
joseph/refactor-model-config
May 14, 2026
Merged

Refactor model config#109
JosephMarinier merged 4 commits into
mainfrom
joseph/refactor-model-config

Conversation

@JosephMarinier
Copy link
Copy Markdown
Collaborator

@JosephMarinier JosephMarinier commented May 7, 2026

  • Yield multiple validation errors at once from _check_companion_services(), not just the first one.
    For example, you could see
    2 validation errors for RunConfig
    model.pipeline.stt_params
      "api_key" required in EVA_MODEL__STT_PARAMS for deepgram STT
    model.pipeline.tts_params
      "model" required in EVA_MODEL__TTS_PARAMS for cartesia TTS
    
  • Refactor model config
    • Get rid of _strip_other_mode_fields() by merging the 3 model classes into a single ModelConfig that always accepts extra EVA_MODEL__*_PARAMS environment variables, even if not used (so no need to strip anything).
    • Skip validating EVA_MODEL__* environment variables when EVA_MAX_RERUN_ATTEMPTS=0. That way, if you specify EVA_MAX_RERUN_ATTEMPTS=0, you no longer need to remove these EVA_MODEL__* environment variables from your .env file. Somewhat-breaking change: we used to skip validating EVA_MODEL__* environment variables when EVA_FORCE_RERUN_METRICS=True, but that could be set while there are still failed conversations to rerun, so now you have to specify EVA_MAX_RERUN_ATTEMPTS=0.
    • This removed 82 lines of code in src/eva/models/config.py.

from `_check_companion_services()`, not just the first one.
- Get rid of `_strip_other_mode_fields()` by merging the 3 model classes into a single `ModelConfig` that always accepts extra `EVA_MODEL__*_PARAMS` environment variables, even if not used.
- Skip validating `EVA_MODEL__*` environment variables when `EVA_MAX_RERUN_ATTEMPTS=0` (and not when `EVA_FORCE_RERUN_METRICS=True`, since it could be set while there are still conversations to rerun). That way, if you specify `EVA_MAX_RERUN_ATTEMPTS=0`, you no longer need to remove these `EVA_MODEL__*` environment variables from your `.env` file.
@JosephMarinier JosephMarinier force-pushed the joseph/refactor-model-config branch from e94cfec to 6886ad6 Compare May 11, 2026 16:12
@gabegma
Copy link
Copy Markdown
Collaborator

gabegma commented May 13, 2026

Awesome! I tried it on an existing run where I had some errors, and I had an annoying error, unrelated to your changes. II will push it here - you can revert it if you want. :)

Comment thread src/eva/metrics/runner.py
Copy link
Copy Markdown
Collaborator

@raghavm243512 raghavm243512 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great.

Definitely something I should have just done when I added the ALM instead of making this mess

@JosephMarinier JosephMarinier added this pull request to the merge queue May 14, 2026
Merged via the queue into main with commit 6515e1e May 14, 2026
1 check passed
@JosephMarinier JosephMarinier deleted the joseph/refactor-model-config branch May 14, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants