opengrall

this is a light openai api wrapper for grall.

setup

1. clone the repo
2. start your grall models, keep all the sockets in the same directory

lets say your models are in ./models/

grall ipc ./models/model1.gril ./sockets/model1.sock
grall ipc ./models/model2.gril ./sockets/model2.sock

2. setup environment variables, heres the most basic ones for the above example

SOCKS_DIR=sockets

see configuration for more env variables.

3. start

go run cmd/opengrall/main.go

supported endpoints

1. GET /v1/models 1. POST /v1/chat/completions (stream and non-streaming)

configuration

ADDR=:8080 # note the : in the beginning
SOCKS_DIR=./sockets
LIMIT=1024 # limit each generation to these many bytes
DELAY=8 # ms delay for each generation
ID_MAX=256 # increase if you have many concurrent generations
OWNED_BY=user # model owner info on /v1/models

todo

• authenticaiton
• better config