JS/TS version

[clharman]

Would love to use this in my product, but can't until it has a JS/TS implementation (and async). Interestingly, neither MSFT Guidance nor LMQL offer it either, so it could be a huge differentiator for adoption. Tons of LLM-based systems are now being built in JS/TS since they don't need the AI/ML capabilities that pre-LLM projects needed from Python.

Barring a JS/TS version, if it had async and retry/timeout logic, I'd consider using an interface in a Lambda function or something.

[zaqqwerty]

Hi @clharman, that is an interesting proposition - I don't think a JS/TS implementation is in scope for us, but could you say more about what "async and retry/timeout logic" might look like?

[fire]

I think running python on the web may work.

https://github.com/pyodide/pyodide

Any approaches?

[fire]

A Lambda function means it starts a vm-like that returns the result of the outlines computation.

[clharman]

@zaqqwerty sure here's what I meant by asnyc and retry/timeout. For context, I use Azure OpenAI service in production with customers and these issues are really important to most production use cases.

Async: OpenAI calls are slow and IO intensive. Many use cases need to make calls simultaneously, for example any branching logic (like tree of thought, I don't use that but I do similar things). I often make 10-20 calls at the same, you can imagine how slow that would be if it all happened sequentially. So I had to build my own async API wrapper so that those calls wouldn't block while waiting for responses.

Retry/timeout: There's a small percentage of OpenAI calls that simply hang until they time out, which takes a really long time. Obviously, that provides a horrid user experience, so I configure a timeout that kills the request after a certain amount of time (usually by 4 seconds it will have succeded, or else you know it's either going to hang). Then, if the timeout was hit, the request gets retried. Strangely, the failure behavior is only semi-random: certain words/contexts seem to raise the likelihood of failure, like "viscosity". As you can imagine, pretty important to have a good mitigation pattern for production usage.

A big challenge I think you folks will deal with on this project is supporting all the "OpenAI wrapper API" features that there are 1000 repos for, figuring out which to integrate into your system. I can confirm from my background as an enterprise implementer that these features would be very helpful, but I'm sure you're already getting spammed with all the other ones... good luck, great project!