Thin wrapper over openAI api, and potentially other AI provider alternatives. It helps handle out of the box caching of request.
It also provide some very minor QOL enhancments to the API
- Local jsonl cache dir
- MongoDB connection
- openAI
- claude ?
- forefront (embedding not supported)
In general running LLM, is an expensive process. Caching however helps offset the cost involved for frequent and common query. The downside is, this is not appropriate for all use cases.