Thoughts on concurrent/async requests?

[defunctl]

Hello!

I was attempting to replace some of the underlying concrete implementations of this project in order to send concurrent API requests to OpenAI to generate multiple completions at once, but due to the architecture of the Resources, they will always make a Request and generate a Response.

For example, 10 synchronous requests to the /completions endpoint with this library can take up to 50 seconds, depending on what's being generated.

I did a basic implementation using Laravel's Http client utilizing pooling (basically Guzzle Async), and I can generate the same 10 completions in ~4-5 seconds.

Any thoughts on adding concurrent/async support in the future, or at least some way of collecting a pool of Requests, so developers could process them on their own?

Pay as you go users can use up to 3000 requests /minute after 48 hours.

Thanks!

[gehrisandro]

Hi @defunctl

Personally I've never worked with async requests in Guzzle, but it doesn't look too complicated.

And for me there are more important improvements to do before, from which all (or at least more) users can benefit.

But I agree, it would be very nice to have async requests, even if I'm not sure, how many users would going to use it. Nevertheless I dug a bit into it and came up with that idea:

$results = $client->completions()
    ->parallel()
    ->create(['model' => 'davinci', 'prompt' => 'PHP is'])
    ->create(['model' => 'davinci', 'prompt' => 'Javascript is'])
    ->create(['model' => 'davinci', 'prompt' => 'CSS is'])
    ->run();

$results then would be an array of CreateResponse objects. When parallel() hasn't been called, it would run synchronously and return the response directly.
What do you think?

And feel free to contribute 🚀

[defunctl]

Thanks for the response, @gehrisandro!

The fluent API you proposed seems pretty good to me, but maybe a different method other than create() so we can avoid if statements in that method, e.g. createParallel() or createAsync() or something like that.

I've been playing around with this for bit, I don't see a way around having to add a method to the \OpenAI\Contracts\Transporter contract, but given it's marked @internal I hope this would be acceptable.

    /**
     * Sends a pool of requests to the server.
     *
     * @param  Payload[]  $payloads
     *
     * @return array<array<array-key, mixed>>
     *
     * @throws ErrorException|UnserializableResponse|TransporterException
     */
    public function requestObjects( array $payloads ): array;

Also, each Resource returns its own implementation of the \OpenAI\Contracts\Response contract, so I guess each resource will have to implement their own run method, but we can at least create a new Parallel contract for that method.

[defunctl]

I'll submit a PR though with an initial rough draft with some tests and we can go from there :)

[nunomaduro]

I would prefer the API to be discussed here - before we making any decisions on this.

[defunctl]

Hey @nunomaduro,

I personally find it a lot easier to discuss with a bit of code in place so I know what you like and don't like. Here's a proof of concept I put together today, let me know if it's heading in the right direction or not, or if there's anything you don't like?

defunctl#1

I really wish there were PSR Async/Promise interfaces, so this is currently pretty coupled to the Guzzle Interfaces at the moment, but I just wanted to provide a potential starting point.

Let me know

[dillingham]

If the client was the laravel http facade, it would have pool built in and allow for fake / testing

[joshua-bn]

Is the API affected by there being parallel requests? Seems as though unless it has an effect on the results by sending them all at once, it is massive bloat for that to be part of this API. Use a different library to wrap those requests so they are async.

[debuggerone]

Maybe you can use https://reactphp.org/ if you want to stick with php here (we are using golang for production).

[nunomaduro]

No plans for this.

[vesper8]

For anyone wishing to achieve this. I came across this python script. Sure it would be great to do it all in php but this won't be the first time I have my Laravel project call python scripts and then parse the results back into it.

https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py