add rerank and tokenization endpoints

[robmsmt]

No description provided.

[Copilot]

Pull request overview

Adds additional LLM proxy endpoints to the FastAPI backend to support reranking, scoring, and (de)tokenization operations, wired into the app and covered by basic auth/route-registration tests.

Changes:

• Introduces new /v1/rerank, /v1/score, /v1/tokenize, and /v1/detokenize routes (auth-protected) and registers them in the application.
• Extends llm_service with new proxy helpers (llm_proxy_rerank/score/tokenize/detokenize) that forward requests to the upstream LLM service.
• Updates backend tests to assert the new routes are present and reject unauthenticated requests.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
backend/tests/test_app.py	Extends route-registration assertions and adds auth rejection tests for new endpoints.
backend/services/llm_service.py	Adds proxy functions for rerank/score/tokenize/detokenize that delegate to the shared proxy handler with backoff.
backend/routers/tokenization.py	New router implementing /v1/tokenize and /v1/detokenize endpoints.
backend/routers/rerank.py	New router implementing /v1/rerank and /v1/score endpoints.
backend/main.py	Registers the new routers so endpoints appear in OpenAPI and are served by the app.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.