docs: sync with sherlock backend changes (status, reason codes, is_banned, dashboard-session list)#4
Merged
Conversation
The Mintlify autogenerator keeps producing skill.md content that
contradicts the source docs — RESTful priors over source content. After
two rounds of strengthening the source pages (adding explicit Notes,
restructuring sections), the same inaccuracies still appear in the
generated /skill.md:
- Revoke documented as POST .../{key_id}/revoke (actual:
DELETE .../{key_id})
- Webhook rotate documented without the /org/ segment
- "Same as API key" storage analogy applied to the webhook secret
(which is stored plaintext, not hashed)
- completed_with_partial surfaced as if active
- Revoke described as admin-only
- Confidence triggers condensed inaccurately
- Usage endpoints omitted
Per Mintlify's spec, a skill.md at the project root overrides
auto-generation. This commit ships that override, written to the
agentskills.io v0 spec (frontmatter + body). Content is grounded in
verified facts from sherlock/src:
- api/routes_auth.py:254 → /api/v2/org/webhook-secret/rotate
- api/routes_auth.py:314 → DELETE /api/v2/org/api-keys/{key_id}
- api/routes_auth.py:318-321,337-338 → revoke role rules
- db/mongodb.py:513 → webhook_secret stored plaintext
- services/aggregator.py:499-530 → confidence triggers
- services/aggregator.py:354 → score band thresholds
Also adds a brief "Organization identifier" section to
authentication.mdx noting that org_id is opaque (no fixed alphabet) —
fixes the org_<hex> hallucination in the previous skill.md.
The override file is 306 lines (well under the 500-line guidance from
the agentskills.io spec).
Usage / analytics endpoints are dashboard-only — they are not part of the public API contract. Removing them from the published docs so external customers don't build integrations against them. - Deleted the four api/usage/*.mdx pages (totals, scans, priority, warmer-health) and the empty parent directory. - Removed the Usage group from docs.json navigation. - skill.md: dropped the Usage table; added an explicit note that usage data is dashboard-only so agents stop hallucinating GET paths. - dashboard-overview.mdx: the Usage sidebar row now reads "Dashboard-only" instead of linking to API pages. Removed the "Get usage totals" link from the Home row and reworded the role-gating table. - api/errors.mdx: dropped "or usage endpoints" from the Org-settings-not-found row. Cross-repo follow-up: the corresponding endpoints in sherlock/src/api/routes_auth.py (get_usage, get_usage_scans, get_usage_priority, warmer-health) should be gated to dashboard sessions only — API-key auth on these paths should return 403, to match the new docs.
User intent: nothing in customer-facing docs should help a reader
reverse-engineer Tumban's detection pipeline. The previous pass
exposed too much — strategy names, judge model, bump_up/bump_down
adjudication, internal score arithmetic.
Scoring (concepts/recommendations.mdx)
- Removed the "three independent detection strategies" paragraph and
the bump_up/bump_down adjudication detail.
- Replaced with a contract-only explanation: what the score means
("confidence a violation is present"), what fields are part of the
public contract, and that other fields are opaque and may change.
Confidence (concepts/confidence.mdx)
- Removed the exact trigger table that listed blocklist thresholds,
multi-strategy agreement, contextual-model confidence pass-through,
and judge invocation.
- Replaced with a semantic table (what high/medium/low mean to a
reviewer). Kept the "never low when no_flags" gotcha — it's a
contract-level invariant, not mechanism.
Field surface (api/scans/get.mdx)
- Dropped the ResponseField blocks for strategy_scores and
judge_model_invoked, removed them from the example JSON, and
removed the "Strategy scores" bullet from the dashboard section.
Folded them into the existing "internal, opaque, subject to change"
disclaimer (which I also generalised — no longer enumerates the
specific internal fields).
Reason codes (reference/reason-codes.mdx)
- Reworded descriptions that explicitly named the judge model or the
"contextual model" / classifier internals. Codes themselves
(JUDGE_BUMP_UP, LLM_API_ERROR, etc.) are left literal because the
API emits them verbatim — cross-repo follow-up below.
- Stripped the source-trail MDX comment that cited
aggregator.py:_aggregate_reason_codes and strategies/llm_decision.py.
Cross-doc cleanup
- evidence-index, scans/get, webhooks/payload: replaced
"the contextual model cited" wording with "Tumban cited".
- index.mdx: removed "multi-strategy detection pipeline".
- skill.md: rewrote Confidence and Score-band sections to match the
scrubbed concepts pages; emphasises that score internals are not
part of the public contract.
Cross-repo follow-ups for sherlock
- API still returns strategy_scores and judge_model_invoked on
GET /api/v2/scans/{scan_id}. Remove from the response model so
customers can't inspect detection internals via the field surface.
- Reason codes JUDGE_BUMP_UP, JUDGE_BUMP_DOWN, LLM_API_ERROR,
LLM_PARSE_ERROR are emitted with internal terminology baked into
the enum strings. Consider renaming (e.g. SCORE_ADJUSTED_UP,
ANALYSIS_ERROR) so the public reason_codes list doesn't leak the
same mechanics this PR scrubbed from prose.
The pipeline no longer carries completed_with_partial as a reserved status — ScanStatusT is now just processing | completed | failed. Scans that experience step failures land in `completed` with the `coverage` object recording what was skipped. The partial_reason webhook field went away with it. Reason-code renames to match aggregator.py and llm_decision.py: CONTENT_SAFETY_TEXT_FLAGGED -> TEXT_FLAGGED CONTENT_SAFETY_IMAGE_FLAGGED -> IMAGE_FLAGGED CONTENT_FILTER_TRIGGERED -> UNSAFE_CONTENT_BLOCKED JUDGE_BUMP_UP -> SECONDARY_REVIEW_CONFIRMED JUDGE_BUMP_DOWN -> SECONDARY_REVIEW_DOWNGRADED LLM_API_ERROR -> ANALYSIS_ERROR LLM_PARSE_ERROR -> PARSE_ERROR Header note added that codes are an unordered set AND open-ended so integrators code defensively against unknown codes.
Adds the recently-shipped public fields and surface changes from the
sherlock backend:
- Get scan: document `is_banned` (bool | null) at the top level of the
scan record. Null until the ban-checker evaluates, then true/false.
- Create batch: tighten the partial-acceptance worked example so the
ordering ("first N accepted, last `profiles_skipped` dropped") is
unambiguous; use the 50-URL / 30-quota numbers from the worked
example brief.
- Authentication + skill.md: dashboard-session-only list expands from
three to five endpoints — POST /org/api-keys and GET /org/api-keys
also reject API keys now (prevents API-key → API-key chaining).
- Errors: document the new 403 detail string for those two endpoints:
"This endpoint requires a dashboard session. API keys are not
permitted."
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings the public docs back in sync with ~14 days of sherlock API-surface changes ahead of cutover. Two waves of edits, two commits.
Wave 1 — drop
completed_with_partialand rename 7 reason codesScanStatusTin sherlock is now justprocessing | completed | failed. Thepartial_reasonwebhook field went away with it. Scans that experience step failures land incompletedwith thecoverageobject recording what was skipped.Reason-code renames (per
aggregator.pyandllm_decision.py):CONTENT_SAFETY_TEXT_FLAGGEDTEXT_FLAGGEDCONTENT_SAFETY_IMAGE_FLAGGEDIMAGE_FLAGGEDCONTENT_FILTER_TRIGGEREDUNSAFE_CONTENT_BLOCKEDJUDGE_BUMP_UPSECONDARY_REVIEW_CONFIRMEDJUDGE_BUMP_DOWNSECONDARY_REVIEW_DOWNGRADEDLLM_API_ERRORANALYSIS_ERRORLLM_PARSE_ERRORPARSE_ERRORHeader note added that codes are an unordered set AND open-ended, so integrators code defensively against unknown codes.
Wave 2 —
is_banned, dashboard-session list, 403 wordingis_banned(bool | null) at the top level of the scan record.nulluntil evaluated; thentrueif the upstream platform has banned the creator (404/410/redirect),falseif still live.triage_report.coverage. Webhook payload carries the same object at the top level.profiles_skippeddropped") is explicit. Worked numbers: 50 URLs submitted with 30 remaining → 30 queued, 20 dropped.POST /org/api-keysandGET /org/api-keysalso reject API keys to prevent API-key → API-key chaining.This endpoint requires a dashboard session. API keys are not permitted.Wave 3 — policy decisions (out of scope for this PR)
The four advanced
PATCH /org/settingsfields (llm_decision_chain,judge_chain,llm_max_tokens,judge_max_tokens) and the consolidated daily-quota concept page were considered and intentionally not added — kept the public surface minimal pre-cutover.Base-branch note
Branched off
docs/skill-override(PR #3, still open). Until #3 merges, this PR will show all 5 commits; once #3 lands, only the 2 new commits remain. The base commits in this stack:8400b4adocs: scrub internal mechanics from scoring and confidence explanations33cc736docs: remove usage endpoints from the public surface18d30c9docs: ship custom skill.md override to fix autogenerator inaccuraciesVerification
Returns zero hits.
Test plan
JUDGE_BUMP_*/LLM_*_ERRORrows, header note about open-ended setcompleted_with_partial)is_bannedResponseField + JSON example; Coverage section says coverage is attriage_report.coverageon pollsCreate API key/List API keyscompleted_with_partialorpartial_reasonis_bannedblurb