Massive Asynchronous Parallel Processor with Instructor

[Chasearmer]

Question

Has anyone built the equivalent of the api_request_parallel_processor.py from the OpenAI Cookbook, but instead using Instructor asynchronously (with apatch)?

api_request_parallel_processor.py

https://github.com/openai/openai-cookbook/blob/4b352176be8e06f122fa9e3b4e1e0cfa6e7b5320/examples/api_request_parallel_processor.py#L375

Context

I'm planning to build that now, and am happy to share it in the examples or wherever it would be useful, but would appreciate not starting from scratch if it has already been built before.

It seems like the main challenge would be that max_retries must be set to zero, so that when a ValidationError occurs the retry attempt can be added to the queue_of_requests_to_retry to ensure the token cost of retrying is accounted for. It seems like the retry prompting would need to be manually performed (reimplemented outside the Instructor library) in order to accomplish this, but the challenge here is that I don't see how to get the model's original response that caused the ValidationError. The only thing that appears to be accessible in the case of a ValidationError is the validation error itself. Therefore, I assume that our only choice is to re-prompt the model without the invalid response -- perhaps appending the validation error to the original message in the form of "Avoid making this error: {e}".

Also I plan to have the input to the parallel processing function be an arbitrary iterator rather than a file.

If there are any high-level thoughts on how to implement this best, that would be appreciated. Otherwise, let me know if it would be valuable to add somewhere once I've finished (assuming everything works).

[jxnl]

https://jxnl.github.io/instructor/blog/2023/11/13/learn-async/

with some caching layer

but nothing around counting tokens for retries. as it can get nasty for streaming applications.