Skip to content

litellm: the catalog ships no database-backed build — virtual keys, teams and spend are undeployable #16

Description

@amiralis1365

The catalog ships litellm as a single stateless container, and its own listing
says what that costs:

This deployment runs stateless — no database and no persistent volume — so it
suits the basic proxy mode; durable virtual-key and spend state needs the
documented database setup, which this listing does not configure.

That sentence is accurate and it describes the wrong default. Virtual keys,
teams, budgets, rate limits and the per-key spend ledger are all Prisma rows.
Without DATABASE_URL the proxy never creates the tables, so the only
credential a deployment has is LITELLM_MASTER_KEY — one shared secret, no
per-client attribution, nothing to invoice from. The feature that makes LiteLLM
worth putting in front of a provider account is the one the catalog cannot
deploy.

What is missing

A composed item: LiteLLM with its own Postgres, and Redis beside it for the
state a second replica needs (shared response cache, cross-replica tpm/rpm
counters, router health).

db.dsn   -> DATABASE_URL  (POSTGRES)
cache.url -> REDIS_URL    (REDIS)

Both producers already exist in the corpus. The two outputs above do not:
items/postgres and items/redis each publish only address, a
mesh-internal host:port. LiteLLM reads one DSN and one URL, so the wire needs
the composed form — the same shape blueprint §4.2's own worked example wires
(fromOutput: connectionString).

Secondary: nothing in the corpus exercises a graph

All 13 items are single-node with connections: {}. The multi-component half of
the blueprint contract — fromRole/fromOutput resolution, ERR_UNKNOWN_INPUT,
type and semanticType compatibility, ERR_UNWIRED_REQUIRED_INPUT — is covered
by tests/rules.test.ts against synthetic items and by nothing that ships. A
real wired item makes the corpus the worked example the README claims it is.

Known constraint, so it is not discovered in review

Postgres wires end to end with no manual step. Redis does not, and cannot in
this platform version.
redis_usage_cache — the client the proxy shares
counters, spend and router state through — is assigned in exactly one place,
ProxyConfig._init_cache, reachable only from the litellm_settings.cache is True branch of load_config. No environment variable reaches that flag in
v1.83.14-stable.patch.3 or in v1.99.1, the current release.

A config.yaml cannot be shipped either. The published image's ENTRYPOINT is
docker/prod_entrypoint.sh, whose last line is exec litellm "$@", and per
spec ADR-0010 a Musher command overrides the image's CMD while inheriting
its ENTRYPOINT — so every word a component can pass the container arrives as a
litellm CLI argument, and there is nowhere to write a file for --config to
name.

What closes it is the other wire. With STORE_MODEL_IN_DB set,
ProxyConfig.save_config writes litellm_settings to the config table in
Postgres and _update_config_from_db merges it back at every start, so enabling
the cache is a one-time action that is durable and applies to every replica.
REDIS_URL is already read from the environment by
_redis_kwargs_from_environment(), and RedisCache drops a null host, port and
password before connecting — so the URL alone is the whole connection once the
flag is on.

The item should wire Redis, say this plainly in the listing, and carry the
reasoning in the component so the next person does not re-derive it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions