Stop verb-chain extraction at non-verb-like tokens

[Aaronontheweb]

Problem

Clause.Verb currently relies on BashArity table lookups (default 2-token, with hand-curated 3-token entries for docker compose and bun run). This breaks down in two ways:

1. Multi-token CLI subcommands without table entries are truncated. freshdesk ticket list --status open, git worktree list, kubectl get pods, aws s3 cp, dotnet ef migrations add all extract a too-short verb chain because their root verbs aren't in the table at the appropriate arity.

2. The table approach can't scale. Custom/private CLIs (freshdesk and any internal company tool) and the long tail of cloud-CLI subcommand groups will never appear in any curated table. Maintenance cost is unbounded.

3. Even with a complete table, the git X Y ambiguity is structurally undecidable. git push origin main (3-token chain git push origin?) vs git worktree list (3-token chain git worktree list) look identical to the parser without per-CLI semantic knowledge. No syntactic rule disambiguates branch names from subcommand names.

Proposed change

Parser side: improve the heuristic

Replace BashArity-table lookup with a "stop at non-verb-like token" heuristic. A token is "verb-like" if it's a bare identifier (lowercase letters + hyphens + dots, no special chars, not too long) and not a flag, not a path-shape, not a value-like token (numeric, URL, env-var ref).

The walk consumes consecutive verb-like tokens from the start of the clause, treating known flag-with-value pairs (existing FlagsWithValue table) as transparent — -C /repo is consumed without breaking the verb-chain walk. Stop at the first non-verb-like token.

Effect on the same examples:

Command	Old extraction (BashArity-based)	New extraction (stop-at-non-verb-like)
freshdesk ticket list --status open	[freshdesk] (1-token default)	[freshdesk, ticket, list] (stops at --status)
git -C /repo worktree list --porcelain	[git, worktree]	[git, worktree, list] (stops at --porcelain)
kubectl get pods my-pod	[kubectl, get]	[kubectl, get, pods, my-pod] (over-extracts; see below)
git push origin main	[git, push]	[git, push, origin, main] (over-extracts; see below)
cat /etc/foo	[cat]	[cat] (stops at /etc/foo path)
chmod 755 file	[chmod]	[chmod] (stops at 755 numeric)

The over-extraction cases (bare-word args like origin, main, my-pod) are unfixable at the parser layer — they're indistinguishable from subcommand verbs without per-CLI semantic knowledge. The new heuristic is strictly better than today's BashArity-based extraction, just not perfect on bare-word args.

Documentation side: scope Clause.Verb correctly

Update SPEC.md to make explicit:

• Clause.Verb is a convenience hint, not a security contract. It's a best-effort identification of the canonical verb chain. Consumers using it for display, audit dedup, or other non-load-bearing purposes can rely on it.
• Consumers needing security-grade verb identification walk the token stream directly. The pattern-matching algorithm should be: "command matches pattern iff first N command tokens equal pattern's verb prefix, where N = pattern's verb-prefix length." This punts the depth choice to the user (via the pattern they author) and eliminates the parser's responsibility to guess.
• The BashArity table will not grow to enumerate CLI subcommand structures. Existing entries (docker compose, bun run) stay for the convenience hint; new ones are not added. The table is a small set of well-known multi-word verb idioms, not an exhaustive registry.

Why over-extraction is acceptable for security

When the parser over-extracts (e.g., [git, push, origin, main] instead of [git, push]), pattern-depth-driven matching handles it correctly:

• User's pattern git push * has verb-prefix length 2.
• Command's first 2 tokens are [git, push].
• Match check: do the first 2 tokens equal the pattern's verb prefix? ✓ MATCH.

Conversely, when a prompt auto-proposes a pattern, greedy over-extraction is the security-correct default. Auto-proposed pattern for git push origin main is git push origin main * (over-specific). This is better than git push * because:

• A subsequent git push wrongremote wrongbranch doesn't auto-grant.
• Re-prompts on variation are audit checkpoints, not friction.
• Operators wanting broader grants opt in explicitly via CLI (netclaw approvals trust-verb 'git push *').

False-negative (re-prompt) is recoverable; false-positive (silent destructive grant) is not. Narrow-by-default favors the recoverable failure mode.

Suggested test cases for the corpus

input:    "freshdesk ticket list --status open"
expected: verb=[freshdesk, ticket, list]

input:    "git -C /repo worktree list --porcelain"
expected: verb=[git, worktree, list]

input:    "kubectl get pods"
expected: verb=[kubectl, get, pods]

input:    "kubectl get pods my-pod"
expected: verb=[kubectl, get, pods, my-pod]   # over-extracts; consumer handles via pattern depth

input:    "aws s3 cp src dst"
expected: verb=[aws, s3, cp, src, dst]   # over-extracts on bare-word path-like args

input:    "git push origin main"
expected: verb=[git, push, origin, main]   # over-extracts on bare-word branch/remote names

input:    "cat /etc/passwd"
expected: verb=[cat]                       # stops at path

input:    "chmod 755 file"
expected: verb=[chmod]                     # stops at numeric mode

input:    "echo --version"
expected: verb=[echo]                      # stops at flag

input:    "ls -la /tmp"
expected: verb=[ls]                        # stops at flag

Non-goals

• Per-CLI semantic knowledge baked into the parser (no git-specific or kubectl-specific subcommand tables).
• Disambiguating bare-word args from subcommand verbs.
• Any UI/UX choices about how consumers display or match these verb chains. Those are consumer concerns.

Severity

Medium. Today's behavior under-extracts for any CLI not in the table, which causes pattern-matching false negatives for consumers (a saved approval pattern doesn't match the command the user thought it would). The proposed fix is strictly better — moves from "always 2-token unless in table" to "stop at non-verb-like" which handles flags and paths correctly without any per-CLI knowledge. Bare-word over-extraction remains; consumers handle that via pattern-depth-driven matching.

Prior discussion

See comments below for the path that led here — earlier proposals to extend the BashArity table with curated entries, plus a multi-option specificity-picker prompt UX, were both rejected. The table approach can't scale to unknown CLIs; multi-option pickers don't survive translation to text-only channel adapters. The current proposal punts depth choice to consumers and keeps the parser stateless about CLI semantics.

[Aaronontheweb]

Update: this is a general multi-token-subcommand problem, not git-specific

Reporter follow-up: this same misextraction has been observed on several non-git CLI families. The pattern is "well-known CLI tool with 3+ token natural verb chains" — wherever the BashArity table maxes out at 2 for the family, the 3rd+ tokens fall into Args and downstream consumers see truncated verb chains.

Affected CLI families (non-exhaustive)

Tool family	Example natural verb	Tokens	Currently extracts as
dotnet ef	dotnet ef migrations add	4	[dotnet, ef]
dotnet ef	dotnet ef database update	4	[dotnet, ef]
dotnet workload	dotnet workload install	3	[dotnet, workload]
kubectl	kubectl get pods	3	[kubectl, get]
kubectl	kubectl apply -f	3	[kubectl, apply]
kubectl	kubectl delete deployment	3	[kubectl, delete]
aws	aws s3 cp	3	[aws, s3]
aws	aws ec2 describe-instances	3	[aws, ec2]
gcloud	gcloud compute instances list	4	[gcloud, compute]
az	az vm list	3	[az, vm]
helm	helm install	2	OK
helm	helm repo add	3	[helm, repo]
terraform	terraform workspace select	3	[terraform, workspace]
apt / apt-get	apt install <pkg>	2	OK (pkg is arg)
pip	pip install <pkg>	2	OK

The shape is consistent: <root-tool> <category> <action> where the category and action together identify the operation. The user's mental model and any approval pattern they author will be the full chain.

Implication for the fix

Option A from the original report (extend the BashArity table) is still the right approach but the table needs a broader sweep, not just git entries. Suggested additions:

// dotnet
["dotnet ef"] = 3,
["dotnet ef migrations"] = 4,    // git-style nested subcommand
["dotnet ef database"] = 4,
["dotnet workload"] = 3,
["dotnet tool"] = 3,
["dotnet nuget"] = 3,

// kubectl / k8s
["kubectl get"] = 3,
["kubectl describe"] = 3,
["kubectl apply"] = 3,
["kubectl delete"] = 3,
["kubectl create"] = 3,
["kubectl edit"] = 3,
["kubectl rollout"] = 3,
["kubectl scale"] = 3,
["kubectl exec"] = 3,
["kubectl logs"] = 3,

// AWS
["aws s3"] = 3,
["aws ec2"] = 3,
["aws iam"] = 3,
["aws lambda"] = 3,
["aws rds"] = 3,
["aws cloudformation"] = 3,
["aws sts"] = 3,

// GCP
["gcloud compute"] = 3,
["gcloud compute instances"] = 4,    // 4-token for nested
["gcloud auth"] = 3,
["gcloud projects"] = 3,
["gcloud iam"] = 3,
["gcloud container"] = 3,
["gcloud functions"] = 3,

// Azure
["az vm"] = 3,
["az storage"] = 3,
["az network"] = 3,
["az aks"] = 3,
["az group"] = 3,

// Helm
["helm repo"] = 3,
["helm release"] = 3,

// Terraform
["terraform workspace"] = 3,
["terraform state"] = 3,

// git additions per original report
["git worktree"] = 3,
["git submodule"] = 3,
["git stash"] = 3,
["git remote"] = 3,
["git config"] = 3,
["git bisect"] = 3,
["git notes"] = 3,
["git tag"] = 3,

// docker (some already in spec at arity 3 for `docker compose`)
["docker compose"] = 3,    // already noted in SPEC.md
["docker container"] = 3,
["docker image"] = 3,
["docker network"] = 3,
["docker volume"] = 3,

This is still finite and maintainable, but the table grows a lot. Option B from the original report (consume consecutive non-flag tokens up to a bounded depth) starts looking more attractive for the long tail.

A hybrid might be best:

• Table for known-stable verb chains (longest-prefix-match, as current)
• Heuristic fallback: if first token is in a recognized "subcommand-heavy" family list (git, dotnet, kubectl, aws, gcloud, az, helm, terraform, docker, npm, yarn, pnpm, bun, cargo, go...), consume consecutive non-flag, non-path-shaped tokens up to depth 4

That gives correct extraction for unknown subcommands of known tools without requiring the table to enumerate every command.

Severity

Re-emphasizing: this isn't a "fix git and move on" — every multi-cloud CLI, every modern .NET tool, kubectl, etc. is affected. For a security-gate consumer, this means approval patterns are systematically too broad (kubectl get matches kubectl get pods AND kubectl get secrets AND kubectl get nodes — which are very different authorization grants). Approvers think they're approving one thing and they're approving a whole category.

[Aaronontheweb]

Architectural redirect: don't pick verb-chain depth at all

After more analysis with another non-git example surfacing the same shape (freshdesk ticket list --status open — a custom CLI no BashArity table will ever know about), and considering the implications for text-only channel adapters (Slack letter-mode, IRC, SignalR text fallback) where multi-option specificity pickers don't survive, the right architectural answer is:

Stop having the parser pick a verb-chain depth. Expose token stream. Let consumers do depth-by-pattern matching.

Why "extend the BashArity table" is fundamentally limited

• Custom/private CLIs (freshdesk, internal company tools) will never appear in any curated table.
• Cloud CLI families have hundreds of subcommand groups (gcloud compute instances ..., aws s3api ...); enumerating exhaustively is a maintenance treadmill.
• Even with a complete table, the git X Y ambiguity remains structurally undecidable: git push origin main vs git worktree list look identical to the parser without per-CLI knowledge.

Why the previous "specificity picker" idea also fails

The earlier comment proposed a multi-option prompt where the user picks the granularity (Allow git, Allow git worktree, Allow git worktree list, Allow exact command). This breaks for text-only channels — multi-option pickers can't be cleanly mapped to A/B/C/D keystroke responses, and stage-based pickers (pick specificity, then pick scope) require state that some channels can't carry.

The correct direction

Parser side:

• Expose Clause.Tokens (or equivalent) as an ordered list of non-flag tokens from the start of the clause through to the first flag or path operand.
• Clause.Verb (the existing convenience field) can stay, but document it as a best-effort hint, not a security contract. Consumers needing security-grade matching read the token stream directly.
• Drop or deprioritize the BashArity table. Keep entries that exist (docker compose, bun run) for back-compat, but stop adding new ones.

Consumer side (e.g. Netclaw approval gate):

• For pattern matching: walk N command tokens where N = pattern's verb-prefix length. No depth choice on the parser side. Pattern git push * matches if first 2 tokens are git push. Pattern git worktree list * matches if first 3 tokens are git worktree list. Pattern length is the user's decision.
• For prompt rendering: auto-propose <all-non-flag-tokens-from-clause-start> * as the pattern shown on the Always button. This is greedy by default — narrow patterns are the security-correct choice. git push origin main * is better than git push * because it doesn't auto-grant git push wrongremote wrongbranch. Re-prompts on variation are audit checkpoints, not friction.
• Operators who explicitly want broader grants opt in via CLI (netclaw approvals trust-verb 'git *'). This is a deliberate widening decision, not the default.

Net effect on the prompt UX

Same 4-button row. Same A/B/C/D text-mode mapping. No specificity picker. The default Always button proposes a narrow pattern; widening is explicit and out-of-band.

Why this is the right answer

• Handles unknown CLIs (custom tools, private internal CLIs) without ever needing to know about them.
• Security-default-narrow: false-negative is "user re-prompted" (recoverable); false-positive is "silent destructive grant" (not recoverable). Narrow patterns favor the recoverable failure mode.
• No arity-table maintenance burden: ShellSyntaxTree doesn't have to track every CLI subcommand structure in the world.
• Text-mode-friendly: zero additional UI options at prompt time.
• git worktree list and git worktree remove get distinct patterns under greedy extraction — approving one doesn't auto-allow the other. Consumer-side authorization-confusion bug from the original report is closed.

Suggested action on this issue

Close as "won't fix — handled at consumer layer."

Recommend a SPEC.md update clarifying:

• Clause.Verb is a convenience hint, not a security contract.
• Consumers needing security-grade verb identification should walk the clause's non-flag tokens directly with depth determined by their own context (e.g. user-supplied pattern length, audit policy, etc.).
• The BashArity table will not grow to enumerate CLI subcommand structures.

The existing 2-token entries (and the docker compose / bun run 3-token entries) stay for the convenience hint to be informative for common cases; they're not load-bearing.

cc original consumer (Netclaw): the trust-zones rewrite's verb-pattern matcher should implement pattern-depth-driven matching as described above.