Panel TokenGate

Panel TokenGate

TokenGate lets administrators import a gate checkpoint, train a local gate, and compare served inference with and without a token keep budget. Open TokenGate at /rust/tokengate to choose the workflow. All submissions require an authenticated administrator. A job is considered successful only when its recorded process exit status says it succeeded.

Import checkpoint

1. Open Import checkpoint (/rust/tokengate/import).
2. Enter absolute paths for Gate checkpoint .pt, LoRA checkpoint .pt, Base GGUF and a new Output GGUF. You may also provide Parameters JSON and Mapping TSV together.
3. Select Import checkpoint. The Panel invokes lexi-engine tokengate-import GATE.pt LORA.pt params.json mapping.tsv base.gguf output.gguf as a tracked job.
4. Open Jobs and status. Confirm the import succeeded before using the output adapter.
5. Enter Instance identifier and Existing adapter GGUF. Tick Confirm adapter binding, then select Use in instance. Before writing the native instance profile, the Panel checks that the adapter exists, has a readable GGUF header and includes TokenGate layer metadata. It saves TOKENGATE_ADAPTER only after confirmation. Restart the instance separately to activate the adapter; the save response is not proof of an active gate. An incompatible profile or unsupported runtime is rejected.

When both Parameters JSON and Mapping TSV are empty, the importer derives the fields and exact mapping from recognizable checkpoint tensor names, metadata and base geometry. Unsupported inventories are rejected, not guessed. If either is supplied, both must be supplied. When alpha is absent from checkpoint metadata, the generated LoRA scale defaults to the derived rank. Files must already exist on the Panel host; the native form does not upload file bytes.

Train a gate

1. Open Train a gate (/rust/tokengate/train).
2. Supply Base GGUF, Training rows JSONL and a new Output GGUF.
3. Optionally set Steps (default 1), Gate layer (default 0), Seed (default 42) and Resume checkpoint (optional).
4. Select Train a gate, then inspect the job record on Jobs and status.

The Panel launches the existing lexi-train tokengate --base BASE.gguf --data rows.jsonl --out ADAPTER.gguf command, adding supplied numeric options. Successful process completion does not independently certify model quality.

Evaluate gate

1. Start the instance with a validated gate adapter.
2. Open Evaluate gate (/rust/tokengate/evaluate).
3. Enter Loopback completion URL and Served model identifier. The URL must use an explicit loopback host and a numeric nonzero port; user information, fragments and remote hosts are refused.
4. Submit Evaluate gate and inspect Jobs and status (/rust/tokengate/status).

The evaluation uses lexi-engine tokengate-eval --mode served --url URL --model NAME. It compares the ungated baseline against gated keep budgets. It refuses to claim latency, pruning or memory savings unless the runtime returns verifiable per-request evidence. Some server versions cannot produce that evidence and their evaluations fail rather than presenting made-up measurements.

Controls and API

Control	Tooltip	API
Import checkpoint	Convert a gate checkpoint and LoRA into a validated adapter; this creates a new output file.	POST /api/tokengate/import
Use in instance	Show the adapter environment binding for the selected instance; restart the instance to apply it.	POST /api/tokengate/bind
Confirm adapter binding	Saving an adapter changes the instance profile; restart separately to activate it.	POST /api/tokengate/bind
Train a gate	Train a gate from local data; steps, gate layer and seed control reproducibility and duration.	POST /api/tokengate/train
Evaluate gate	Compare a running instance with and without a keep budget; missing runtime evidence fails the evaluation.	POST /api/tokengate/evaluate
Jobs and status	Inspect recent job progress, exit status and output files without exposing checkpoint contents.	GET /api/tokengate/jobs, GET /api/tokengate/job?id=JOB, GET /api/tokengate/status

Field help

Input	Tooltip	API
Gate checkpoint .pt (tokengate.import.gate)	Select an existing gate checkpoint on the Panel host.	POST /api/tokengate/import
LoRA checkpoint .pt (tokengate.import.lora)	Select the LoRA checkpoint paired with the gate.	POST /api/tokengate/import
Parameters JSON (tokengate.import.params)	Supply explicit parameters, or leave empty with mapping for automatic derivation.	POST /api/tokengate/import
Mapping TSV (tokengate.import.mapping)	Supply explicit tensor mapping, or leave empty with params for automatic derivation.	POST /api/tokengate/import
Base GGUF (tokengate.import.base)	Select the exact base model whose tensor geometry will be checked.	POST /api/tokengate/import
Output GGUF (tokengate.import.output)	Choose a new output path; existing files are never selected by the form.	POST /api/tokengate/import
Instance identifier (tokengate.bind.instance)	Choose which existing native instance will receive the adapter setting.	POST /api/tokengate/bind
Existing adapter GGUF (tokengate.bind.adapter)	Choose an imported GGUF adapter to save in the instance configuration.	POST /api/tokengate/bind
Confirm adapter binding (tokengate.bind-confirm)	Saving an adapter changes the instance profile; restart separately to activate it.	POST /api/tokengate/bind
Base GGUF (tokengate.train.base)	Choose the base model for local gate training.	POST /api/tokengate/train
Training rows JSONL (tokengate.train.data)	Choose the local training dataset; the trainer reads this file.	POST /api/tokengate/train
Output GGUF (tokengate.train.out)	Choose a new GGUF adapter output path for training.	POST /api/tokengate/train
Steps (default 1) (tokengate.train.steps)	Set the number of local training updates; higher values take longer.	POST /api/tokengate/train
Gate layer (default 0) (tokengate.train.gate-layer)	Select the base model layer whose activations feed the gate.	POST /api/tokengate/train
Seed (default 42) (tokengate.train.seed)	Set the deterministic training seed for repeatable data ordering.	POST /api/tokengate/train
Resume checkpoint (optional) (tokengate.train.resume)	Resume from a compatible existing checkpoint instead of starting over.	POST /api/tokengate/train
Loopback completion URL (tokengate.evaluate.url)	Enter the local completion endpoint; remote evaluation is refused.	POST /api/tokengate/evaluate
Served model identifier (tokengate.evaluate.model)	Choose the model name reported by the running instance.	POST /api/tokengate/evaluate

The work area uses the same Panel navigation, accessibility controls and security boundary as other native pages. Read-only roles never receive a TokenGate job-submission or binding form. Administrator permissions and token route scopes are checked again by the owner when submitting a job or changing a binding. Form submissions are size-limited; malformed fields produce an error rather than a success page.

These APIs accept JSON input for the POST actions. The job submission response includes a job identifier. The status API returns stored job details and, after completion, a result object with the exit status. Output and log paths are local to the Panel host. The status page also displays adapter bindings from existing native instance profiles; unverified gate layers and savings remain unset. The status page displays the most recent records available in the local job store, not a guarantee that an active process survived a Panel restart.