Skip to content

LiteLLM Integeration & Batch Call Interface #70

@simonguozirui

Description

@simonguozirui

I wrote a very manual LLM inference engine system for the initial version of KernelBench.

Two things we should do to better integrate

  • Use litellm and .env so we can support a variety of future models and not writing a new backend for each
  • For pass@k and test-time compute settings, we should use the batch call API to simultaneously call rather than a thread of each.

Will implement with team @pythonomar22 @nathanjpaek @AffectionateCurry .

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions