Is the Claude behind your Kilo custom provider real? A 3-request check (thinking signature)

[mupromax]

If you run Kilo with your own key through a Custom provider (a reseller, relay or gateway instead of Anthropic directly), here's a 3-request check that the Claude behind it is actually Claude. Disclosure: I run one of those gateways (APIVAI), so I have an interest; the check works on any endpoint, mine included, and I'd like people to run it on everything.

Why not just ask the model? "Who are you" and "what's your knowledge cutoff" are useless: a relay can inject a system prompt, and genuine Claude gives inconsistent answers anyway. The model field in the response is written by the relay.

What does work: the extended-thinking signature. With thinking on, every thinking block carries an opaque signature. When you send the assistant turn back (which Kilo does on every tool call), Anthropic verifies it. So:

1. Ask something that triggers a tool call, thinking enabled → you get a signed thinking block plus tool_use.
2. Send the turn back unchanged with a tool result → a genuine backend returns 200.
3. Send it back with the last 8 characters of the signature changed → a genuine backend returns 400 Invalid signature in thinking block.

A relay serving another model can invent a signature-looking string, but it can't make Anthropic's verifier reject the tampered one unless requests really reach Anthropic.

Kilo-specific note: this needs the native format, i.e. Provider API = Anthropic Messages in the custom provider. With OpenAI Compatible, signatures are dropped in conversion, so you can't run the test there, and multi-turn thinking + tools may break with "Invalid signature" errors if the relay mangles blocks. Kilo's field takes the base URL with /v1; the script below wants it without.

import os, requests

BASE = os.environ["RELAY_BASE_URL"]      # without /v1
KEY = os.environ["RELAY_API_KEY"]
MODEL = os.environ.get("MODEL", "claude-sonnet-4-6")

H = {"x-api-key": KEY, "anthropic-version": "2023-06-01", "content-type": "application/json"}
TOOLS = [{"name": "get_weather", "description": "Get the weather for a city",
          "input_schema": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}]
THINK = {"type": "enabled", "budget_tokens": 1024}
Q = {"role": "user", "content": "What is the weather in Paris? Use the tool."}

def call(messages):
    r = requests.post(f"{BASE}/v1/messages", headers=H, timeout=120, json={
        "model": MODEL, "max_tokens": 1600, "thinking": THINK, "tools": TOOLS, "messages": messages})
    return r.status_code, r.json()

s, a = call([Q])
blocks = a.get("content", [])
print("1st:", s, a.get("id"), [b["type"] for b in blocks])
tool = next((b for b in blocks if b["type"] == "tool_use"), None)
think = next((b for b in blocks if b["type"] == "thinking"), None)
if not (tool and think and think.get("signature")):
    raise SystemExit("No signed thinking block: can't run the test")

def second(content):
    return call([Q, {"role": "assistant", "content": content},
                 {"role": "user", "content": [{"type": "tool_result", "tool_use_id": tool["id"], "content": "Sunny, 18C"}]}])

print("unchanged signature:", second(blocks)[0], "(expect 200)")
bad = [dict(b, signature=b["signature"][:-8] + "AAAAAAAA") if b["type"] == "thinking" else b for b in blocks]
s3, b3 = second(bad)
print("tampered signature:", s3, b3.get("error", {}).get("message", ""), "(expect 400)")

Costs well under a cent on Haiku, about a cent on Sonnet.

unchanged	tampered	meaning
200	400	reaches Anthropic's Claude; thinking blocks pass through intact
200	200	nobody checks signatures: not a genuine Anthropic backend
400	400	the relay breaks thinking blocks; expect multi-turn tool sessions in Kilo to fail too

Limits, so nobody over-reads it: it proves "Anthropic's Claude", not which Claude (Opus quietly served as Sonnet passes; compare cost and latency against the official API for that). It's a snapshot, so rerun it now and then. Passing means "no evidence of substitution", not proof.

My own run, for transparency: Haiku 4.5 and Sonnet 4.6 on my gateway gave 200 on the unchanged signature and 400 Invalid signature on the tampered one. Results from other endpoints are welcome below, failures especially.