[User Story] How I am using tingly box as a self-hosted LLM gateway

[MingxiYe]

Hey, I wanted to share how I have been running tingly box on my server. Maybe this could help others

What this thing actually is

Here is my situation. I got a bunch of servers and devices all running different AI workloads, and I was tired of managing API keys everywhere. With tingly box, I can manage all the keys in one server, and then everything else just talks to tingly box instead of the providers directly.

---

How I run it on my server

With Node.js installed, I just simply run npx tingly-box@latest start.

Or use npx tingly-box@latest start --host 0.0.0.0 to make service publich (must confirm firewall and user token is changed)

---

The SSh trick for setup

And here is one trick for configuring tingly box, since the web UI lives on the server but I would like to configure it from my laptop. Here is hot it works.

# From my laptop
ssh -L 12580:localhost:12580 user@your-server.com

Now localhost:12580 on my laptop = localhost:12580 on the server.

Then open http://localhost:12580 in your browser. You can access to the web UI of tingly box now!

---

How to use

from openai import OpenAI

client = OpenAI(
    api_key="sk-tingly-token",
    base_url="http://SERVER_IP:12580/openai/v1"
)

[richardxx]

Thanks for the sharing. Another typical use is to set up a proxy for team to share models: Buying one key for each model provider and easily share to all members.