LLM analyzers run with no sampling controls, so the published recommendation is not reproducible

[shi-eric]

A scan's published recommendation can change between runs on byte-identical input. Two consecutive scans of the same unmodified skill returned DO_NOT_INSTALL and then CAUTION.

Both runs completed cleanly with full coverage, so a dropped batch or a provider hiccup is ruled out. The LLM-backed analyzers are simply called without any sampling parameters, so each run draws a fresh sample, and the report does not record which model produced it.

What was observed

The same skill directory, unmodified between runs, scanned four times:

Run	Mode	Score	Severity	Recommendation
1	default	66/100	HIGH	DO_NOT_INSTALL
2	default	47/100	MEDIUM	CAUTION
3	--no-llm	48/100	MEDIUM	CAUTION
4	--no-llm	48/100	MEDIUM	CAUTION

The static rules fired identically in all four runs, and the two --no-llm runs returned the same nine rule IDs. All of the variance came from LLM-judged SDI-1 findings, which appeared three times in run 1 and once in run 2. SDI-1 is emitted by semantic_developer_intent.py, where the rule is defined inside the prompt rather than in code.

Nothing was dropped. Both default runs reported Execution: successful, Coverage: 98.4%, 61 files fully inspected, 1 partial, 0 uninspected, and every analyzer completed.

recommendation is derived from the overall severity in nodes/report.py, where HIGH and CRITICAL both map to DO_NOT_INSTALL. So a couple of extra LLM findings are enough to move the severity band, and the published verdict moves with it.

Why it varies

No sampling controls are set on any provider path. A search for temperature, seed, and top_p across the source tree returns nothing.

create_openai_compatible_chat_model constructs ChatOpenAI with model, base_url, api_key, max_completion_tokens, timeout, default_headers, and an optional reasoning_effort. The ChatAnthropic and ChatBedrockConverse paths set no sampling parameters either. The only knob exposed anywhere is the SKILLSPECTOR_REASONING_EFFORT environment variable, which selects reasoning depth rather than sampling behavior.

So every LLM analyzer call runs at whatever the provider defaults to. OpenAI-compatible endpoints accept seed and Anthropic accepts temperature, and neither is passed.

The report also does not say what produced the result. nodes/report.py records skillspector_version and scanned_at. It records no provider and no model, so a reader holding a report cannot tell which model judged the semantic findings, and two reports of the same skill cannot be told apart except by their timestamps.

Why this matters left alone

A workflow that gates on the recommendation cannot be satisfied by improving a skill, because the thing being checked is not a property of the skill. DO_NOT_INSTALL and CAUTION lead to different decisions, and which one a skill receives depends on the draw.

Confirming a result means running it again and getting a different one, with no way to attribute the difference to the skill, the model, the provider, or chance, and no provider or model recorded in either report to compare against. That also makes re-running a workable way to get a preferred answer. A DO_NOT_INSTALL may clear on the second attempt, a clean skill may be blocked by an unlucky draw, and nothing in the record tells those two situations apart from an honest first run.

Baselines do not survive it either. --baseline matches by fingerprint, so an LLM finding that was reviewed and accepted can simply fail to appear next time, while a new one shows up that no baseline covers.

The two halves also reach the reader as one number. The static analyzers are deterministic and the LLM analyzers are not, but the score, severity, and recommendation merge them with no indication of which findings anyone should expect to see again. Today --no-llm is the only way to get a result that can be reproduced, and it drops the semantic analysis entirely to do it.

Environment

• SkillSpector 2.5.1
• Scans run as skillspector scan <dir> --format json, and --no-llm for the static-only control
• Source inspected at a818f50
• Linux, Python 3.12