Skip to content

Add Lynkr router submission#167

Merged
yl231 merged 3 commits into
RouteWorks:mainfrom
vishalveerareddy123:add-lynkr
Jul 23, 2026
Merged

Add Lynkr router submission#167
yl231 merged 3 commits into
RouteWorks:mainfrom
vishalveerareddy123:add-lynkr

Conversation

@vishalveerareddy123

@vishalveerareddy123 vishalveerareddy123 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Router: Lynkr

Lynkr is an open-source (Apache-2.0), self-hosted LLM gateway that routes requests across local and cloud models. Routing here uses its embedding-anchor tier classifier (no LLM classifier call — routing decisions are local and deterministic).

Files

  • router_inference/config/lynkr.json — router configuration
  • router_inference/predictions/lynkr.json — full-split predictions with generated_result populated
  • router_inference/predictions/lynkr-robustness.json — robustness predictions
  • router_inference/router/lynkr_router.py + registration in __init__.py — adapter for reproducibility
  • Additive entries in model_cost/model_cost.json (z-ai/glm-4.7 pricing) and universal_model_names.py for the pool models

Local evaluation results (for reference; please re-run officially)

Arena 67.65 · Accuracy 68.41% · $0.29/1K queries · Robustness 92.38 · Opt.Sel 11.13 / Opt.Cost 17.00 / Opt.Acc 85.37

Disclosures

  • Optimality metrics were computed on 629/809 of the optimality queries locally — our OpenRouter credits ran out on the GLM tail. The prediction files are complete; your official evaluation should produce full coverage.
  • Routing is heavily concentrated on gpt-oss-120b (~96% of queries) — a consequence of the tier boundaries, not tuning on RouterArena data.
  • Per the evaluation-only policy: no Lynkr component was trained, fitted, or tuned on RouterArena data or labels. The anchor classifier was built entirely from Lynkr's own telemetry before this evaluation; RouterArena was run once as a held-out test.

Happy to adjust anything to match submission conventions.


Leaderboard row details (for the manifest/README, to save you a lookup)

  • Router name: Lynkr
  • Affiliation: 👤 @vishalveerareddy123
  • Links: Code · npm
  • Open-source: yes (self-hosted gateway; routing decision is local, no LLM call)
  • Prediction files: lynkr / lynkr-robustness (this PR)

Lynkr is an open-source (Apache-2.0), self-hosted LLM gateway that routes
requests via embedding-anchor tiering. Pool: openai/gpt-oss-120b,
qwen3-235b-a22b-2507, z-ai/glm-4.7 (all via OpenRouter).

Includes router config, full-split predictions with generated_result,
robustness predictions, the router adapter class, and additive entries to
model_cost/universal_model_names for the pool models.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vishalveerareddy123

Copy link
Copy Markdown
Contributor Author

/evaluate

…t router

13 entries where the model responded but no answer text was extractable now
carry generated_answer="" and success=false with an explanatory error field
(previously null answer with success=true). Validator passes locally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vishalveerareddy123

Copy link
Copy Markdown
Contributor Author

/evaluate

@github-actions

Copy link
Copy Markdown

Router Evaluation Results

Router: lynkr
Dataset Split: full

RouterArena Metrics

Metric Value
RouterArena Score 0.6761
Accuracy 68.35%
Total Cost $2.424762
Avg Cost per Query $0.000289
Avg Cost per 1K Queries $0.2887
Number of Queries 8400
Abnormal Entries 6
Robustness Score 0.9238

⚠️ 6 of 8400 queries (0.1%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

Optimality Metrics

Metric Value
Opt.Sel (Optimal Selection) 0.1116
Opt.Cost (Cost Efficiency) 0.1700
Opt.Acc (Accuracy vs Optimal) 0.8565

Evaluation completed by RouterArena automated workflow

All previously-failed entries (credit exhaustion during original run) now have
real generated results — 176/176 successful. Validator passes locally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vishalveerareddy123

Copy link
Copy Markdown
Contributor Author

/evaluate

1 similar comment
@vishalveerareddy123

Copy link
Copy Markdown
Contributor Author

/evaluate

@github-actions

Copy link
Copy Markdown

Router Evaluation Results

Router: lynkr
Dataset Split: full

RouterArena Metrics

Metric Value
RouterArena Score 0.6765
Accuracy 68.41%
Total Cost $2.456236
Avg Cost per Query $0.000292
Avg Cost per 1K Queries $0.2924
Number of Queries 8400
Abnormal Entries 0
Robustness Score 0.9238

Optimality Metrics

Metric Value
Opt.Sel (Optimal Selection) 0.1097
Opt.Cost (Cost Efficiency) 0.1608
Opt.Acc (Accuracy vs Optimal) 0.8448

Evaluation completed by RouterArena automated workflow

@vishalveerareddy123

Copy link
Copy Markdown
Contributor Author

@yl231 The full-split evaluation is now clean — 0 abnormal entries (down from 6 on the previous run; root cause was the truncated entries in the regenerated prediction file). Final numbers from the run above: Arena 0.6765, Accuracy 68.41%, $0.2924/1K queries, Robustness 0.9238, 8,400/8,400 scored, all CI checks green.

The submitted config is Lynkr's untuned production default, per the evaluation-only policy. Ready for review whenever you have a moment — happy to adjust anything needed for the leaderboard entry.

@yl231 yl231 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer review — approving. Structural audit clean (8,400 base + 420 robustness + 1,618 optimality rows; genuine per-row generations; costs reconcile against the registered per-token prices). The added z-ai/glm-4.7 pricing matches the existing underscore variant exactly. lynkr_router.py is a thin gateway adapter — no RouterArena data access. Official /evaluate: Accuracy 68.41% · Arena 0.6765 · $0.2924 per 1K · Robustness 0.9238. All checks green. Thanks @vishalveerareddy123 — merging; leaderboard row to follow.

@yl231
yl231 merged commit c90b9d0 into RouteWorks:main Jul 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants