[Feature] Asynchronous Batch API support

[rayanehmi]

What feature would you like to see?

My use-case involves using a GEPA-optimized DSPy module to do tabular prediction. Since I'm independent (and not rich), I'd like to be able to use Asynchronous Batch endpoints (ex. OpenAI Batch API) natively from DSPy to reduce costs on large datasets.

LiteLLM supports using Batch API endpoints (link to docs). I think it would be nice to have something like a module.abatch() method directly from DSPy to be able to use those Batch endpoints.

Pros:

• Reduce costs for various tasks like pure inference, evaluation and optimization.
• Bypass rate limits for some providers.

Cons:

• Variable inference time (15s to a few minutes on my tests).
• Tricky to do for complex / multi-call modules...

Would you like to contribute?

• Yes, I'd like to help implement this.
• No, I just want to request it.

Additional Context

I do have a bloated-but-working implementation (from 5.1-Codex) on my fork along with a demo notebook. It only works for single-call modules as of writing this.

Would absolutely love to contribute to a proper implementation with some guidance!

[TomeHirata]

Hi, @rayanehmi. Is my understanding correct that the OpenAI batch API only guarantees maximum 24-hours turn around time? If so, how do you want to run the entire DSPy program with this API? Do we also want to implement abatch as "ack" + "status check", or does the method wait until the entire program execution is completed?